A crowd counting method, device, electronic equipment and storage medium

Abstract

The present disclosure relates to the field of computer vision, and provides a crowd counting method and device, electronic equipment and storage medium. The method comprises: acquiring a crowd image; extracting a multi-scale fusion feature of the crowd image; encoding and decoding the multi-scale fusion feature to obtain a plurality of feature-level head scale density features; determining a predicted density map corresponding to each feature-level head scale density feature; dividing each predicted density map into a plurality of image slices, calculating a confidence label of each image slice at each feature level, and determining an optimal feature level for predicting the number of people in the image slice according to the confidence label; and predicting the number of people according to the optimal feature level corresponding to each image slice of each predicted density map to obtain the crowd count of the crowd image. The present disclosure can reduce the undercounting of the head dense area and the overcounting of the head sparse area, and has high counting accuracy for dense crowds.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computer vision, and particularly relates to a crowd counting method and device, an electronic device and a storage medium. BACKGROUND

[0002] In public places, crowded crowd flow statistics and order evacuation have always been a topic of concern. If the dense crowd count in the captured picture and the crowd density in different directions can be accurately identified, it will play a key role in on-site evacuation and order maintenance.

[0003] There are two major calculation methods for current crowd counting. One is to count the crowd by detecting the head and shoulder or half body in the picture. However, in a dense scene, due to high crowd density, the head and shoulder and other body parts are very easy to be occluded, resulting in very few effective head and shoulder and other body information being detected, which directly affects the accuracy of counting. The other is a regression-based method, which learns the mapping from low-dimensional features of images to crowd counting through data. However, this method is difficult to solve the problem of continuous change of head size and density, resulting in overcounting or undercounting in local areas, and thus the accuracy of counting is not high. SUMMARY

[0004] Therefore, the embodiments of the present disclosure provide a crowd counting method, device, electronic device and storage medium to solve the problem that the existing method for counting dense crowds is difficult to solve the problem of continuous change of head size and density, resulting in overcounting or undercounting in local areas, and thus the accuracy of counting is not high.

[0005] In a first aspect, the embodiments of the present disclosure provide a crowd counting method, comprising:

[0006] obtaining a crowd image;

[0007] extracting a multi-scale fusion feature of the crowd image;

[0008] encoding and decoding the multi-scale fusion feature to obtain a plurality of feature-level head size and density features;

[0009] determining a predicted density map corresponding to each feature-level head size and density feature;

[0010] dividing each predicted density map into a plurality of image slices, calculating a confidence label of each image slice at each feature level, and determining an optimal feature level for predicting the number of people in the image slice according to the confidence label;

[0011] The counting module is configured to perform crowd number prediction according to the optimal feature level corresponding to each image slice of each prediction density map, to obtain crowd counting of the crowd image.

[0012] In a second aspect, the present disclosure provides a crowd counting device, comprising:

[0013] The acquisition module is configured to acquire a crowd image.

[0014] The extraction module is configured to extract multi-scale fusion features of the crowd image.

[0015] The coding and decoding module is configured to perform coding and decoding processing on the multi-scale fusion features, to obtain head scale density features of multiple feature levels.

[0016] The determination module is configured to determine a prediction density map corresponding to each feature level of the head scale density features.

[0017] The calculation module is configured to divide each prediction density map into multiple image slices, to calculate confidence labels of each image slice at each feature level, and to determine an optimal feature level used for crowd number prediction of the image slice according to the confidence labels.

[0018] The counting module is configured to perform crowd number prediction according to the optimal feature level corresponding to each image slice of each prediction density map, to obtain crowd counting of the crowd image.

[0019] In a third aspect, the present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0020] In a fourth aspect, the present disclosure provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of the above method.

[0021] Compared with the prior art, the embodiments of the present disclosure have at least the following beneficial effects: by first acquiring a crowd image, extracting multi-scale fusion features of the crowd image, without increasing additional parameters, the continuously changing head scale and head density features in a dense crowd scene can be described in detail, and the head positioning is more accurate; then the multi-scale fusion features are encoded and decoded to obtain head scale and density features of multiple feature levels; then the prediction density map corresponding to each feature level head scale and density feature is determined; each prediction density map is divided into multiple image slices, the confidence labels of each image slice at each feature level are calculated, and the optimal feature level used for predicting the number of image slices is determined according to the confidence labels; the number of people is predicted according to the optimal feature level corresponding to each image slice of each prediction density map, and the crowd counting of the crowd image is obtained, which can realize adaptive selection of the optimal feature level corresponding to each image slice, more perfect and accurate feature detail description, effectively reduce the undercounting of the head dense area and the overcounting of the head sparse area, and improve the crowd counting accuracy in the dense crowd scene. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 is a flowchart of a crowd counting method provided by an embodiment of the present disclosure;

[0024] Figure 2 is an original input crowd image provided by an embodiment of the present disclosure;

[0025] Figure 3 is a network structure diagram of a multi-scale feature fusion network in the crowd counting method provided by an embodiment of the present disclosure;

[0026] Figure 4 is a multi-scale fusion feature extraction process diagram in the crowd counting method provided by an embodiment of the present disclosure;

[0027] Figure 5 is a structure diagram of a pyramid structure encoding-decoding network in the crowd counting method provided by an embodiment of the present disclosure;

[0028] Figure 6 is a structure diagram of a decoding layer in the crowd counting method provided by an embodiment of the present disclosure;

[0029] Figure 7 is a human head density distribution map provided by an embodiment of the present disclosure;

[0030] Figure 8 is a structure diagram of a density branch provided by an embodiment of the present disclosure;

[0031] Figure 9 is a whole network structure diagram of a crowd counting network provided by an embodiment of the present disclosure;

[0032] Figure 10 is a generation process diagram of a confidence label provided by an embodiment of the present disclosure;

[0033] Figure 11 is a structure diagram of a crowd counting device provided by an embodiment of the present disclosure;

[0034] Figure 12 is a structure diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In the following description, specific details are set forth in order to provide a thorough understanding of embodiments of the present disclosure. However, persons skilled in the art will understand that embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known systems, structures, circuits, and methods have not been described in detail in order to avoid obscuring the description of the present disclosure.

[0036] A crowd counting method and device according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0037] Figure 1 is a flow diagram of a crowd counting method provided by an embodiment of the present disclosure. Figure 1 The crowd counting method can be performed by a server. As shown in Figure 1 , the crowd counting method includes:

[0038] Step S101, obtaining a crowd image.

[0039] The crowd image can be an image containing multiple people captured by a monitoring camera device (such as a monitoring camera, etc.) installed in a public place (for example, a street, a square, etc.). For example, as shown in Figure 2 , a crowd image at a certain angle on a certain road. Of course, it can also be a crowd image captured by a device with a shooting function such as a mobile phone, a computer, etc.

[0040] Step S102, extracting multi-scale fusion features of the crowd image.

[0041] In the annotation of a dense crowd base, a point of a human head is usually annotated to generate a density map as annotation data. However, the distribution of human heads is very uneven. The closer to the camera, the larger the scale of the human head, and the sparser the relative point density. Conversely, the farther to the camera, the smaller the scale of the human head, and the denser the relative point density. Using the same scale of convolution to extract features in the same image of a crowd will cause a large error. In addition, the annotation of an image itself does not contain information about density and scale distinction. If some regions are manually divided into different density regions, the cost is very huge.

[0042] In order to extract more scale feature information in a crowd image, the multi-scale feature fusion network is used to extract multi-scale fusion features of the crowd image in the embodiment of the present disclosure. The network structure of the multi-scale feature fusion network is shown in Figure 3 As shown in Figure 3 The multi-scale feature fusion network 300 includes a first convolutional layer 301, a second convolutional layer 302, a third convolutional layer 303, and a fourth convolutional layer 304. The first convolutional layer 301 is a dilated convolutional layer with a kernel size of 3x3 and a ratio of 1 (conv3, DF=1). The second convolutional layer 302 includes a first convolutional block 3021, a second convolutional block 3022, a third convolutional block 3023, and a fourth convolutional block 3024. The first convolutional block 3021 is a dilated convolutional block with a kernel size of 3x3 and a ratio of 1 (conv3, DF=1), the second convolutional block 3022 is a dilated convolutional block with a kernel size of 3x3 and a ratio of 2 (conv3, DF=2), the third convolutional block 3023 is a dilated convolutional block with a kernel size of 3x3 and a ratio of 3 (conv3, DF=3), and the fourth convolutional block 3024 is a dilated convolutional block with a kernel size of 3x3 and a ratio of 4 (conv3, DF=2). The third convolutional layer 303 includes a fifth convolutional block 3031 and a sixth convolutional block 3032, both of which are dilated convolutional blocks with a kernel size of 3x3 and a ratio of 1 (conv3, DF=1). The fourth convolutional layer 304 is a dilated convolutional layer with a kernel size of 3x3 and a ratio of 1 (conv3, DF=1).

[0043] In some embodiments, the step S202 can specifically include the following steps:

[0044] performing a convolutional processing on the crowd image to obtain a first scale convolutional feature;

[0045] performing a second, third, fourth, and fifth convolutional processing on the first scale convolutional feature to obtain a second scale convolutional feature, a third scale convolutional feature, a fourth scale convolutional feature, and a fifth scale convolutional feature.

[0046] fusing the second scale convolution feature and the third scale convolution feature to obtain a first fused feature;

[0047] fusing the fourth scale convolution feature and the fifth scale convolution feature to obtain a second fused feature;

[0048] fusing the first fused feature and the second fused feature to obtain a multi-scale fused feature.

[0049] As an example, in combination with Figure 3 and Figure 4 , first, the crowd image F can be input into the first convolution layer 301, so that the crowd image F is subjected to a convolution kernel size of 3x3 and a rate of 1 to obtain the first scale convolution feature F1 through a dilated convolution; then, the first scale convolution feature F1 is input into the first convolution block 3021, the second convolution block 3022, the third convolution block 3023 and the fourth convolution block 3024 of the second convolution layer 302 respectively, and the second scale convolution feature F2 is obtained through the first convolution block 3021 to perform a conv3, DF=1 dilated convolution, the third scale convolution feature F3 is obtained through the second convolution block 3022 to perform a conv3, DF=2 dilated convolution, the fourth scale convolution feature F4 is obtained through the third convolution block 3023 to perform a conv3, DF=3 dilated convolution, and the fifth scale convolution feature F5 is obtained through the fourth convolution block 3024 to perform a conv3, DF=4 dilated convolution; then, the second scale convolution feature and the third scale convolution feature are input into the fifth convolution block 3031 of the third convolution layer 303 to perform a conv3, DF=1 dilated convolution respectively, and the first fused feature F6 is obtained by adding the features after the convolution; the fourth scale convolution feature and the fifth scale convolution feature are input into the sixth convolution block 3032 of the third convolution layer 303 to perform a conv3, DF=1 dilated convolution respectively, and the second fused feature F7 is obtained by adding the features after the convolution; finally, the first fused feature F6 and the second fused feature F7 are input into the fourth convolution layer 304 to perform a conv3, DF=1 dilated convolution respectively, and the multi-scale fused feature F8 is obtained by adding the features after the convolution.

[0050] Through the multi-scale feature fusion network provided by the embodiments of the present disclosure, the human head features with continuously changing scales can be extracted using multiple scales of receptive fields without changing the resolution, and without increasing additional parameters. In addition, the obtained multi-scale features are subjected to two-stage feature fusion, which can describe the continuously changing scale and density features in the dense crowd scene in detail, the positioning of the human head is more accurate, and the expression ability of the feature information is better.

[0051] Step S103, the multi-scale fusion features are encoded and decoded to obtain human head scale density features of multiple feature levels.

[0052] As an example, a pyramid structure encoding-decoding network as shown in FIG. 5 can be used to encode and decode the multi-scale fusion features. As shown in FIG. 5, the pyramid structure encoding-decoding network 500 includes an encoding network layer 501 in the encoding stage and a decoding network layer 502 in the decoding stage. The encoding network layer 501 includes a first encoding layer 5011, a second encoding layer 5012, a third encoding layer 5013, a fourth encoding layer 5014, and a fifth encoding layer 5015 connected in sequence. The decoding network layer 502 includes a first decoding layer 5021, a second decoding layer 5022, a third decoding layer 5023, a fourth decoding layer 5024, and a fifth decoding layer 5025 connected in sequence. The first encoding layer 5011 is connected to the first decoding layer 5021, the second encoding layer 5012 is connected to the second decoding layer 5022, the third encoding layer 5013 is connected to the third decoding layer 5023, the fourth encoding layer 5014 is connected to the fourth decoding layer 5024, and the fifth encoding layer 5015 is connected to the fifth decoding layer 5025. Figure 5 Figure 5 As an example, a pyramid structure encoding-decoding network as shown in FIG. 5 can be used to encode and decode the multi-scale fusion features. As shown in FIG. 5, the pyramid structure encoding-decoding network 500 includes an encoding network layer 501 in the encoding stage and a decoding network layer 502 in the decoding stage. The encoding network layer 501 includes a first encoding layer 5011, a second encoding layer 5012, a third encoding layer 5013, a fourth encoding layer 5014, and a fifth encoding layer 5015 connected in sequence. The decoding network layer 502 includes a first decoding layer 5021, a second decoding layer 5022, a third decoding layer 5023, a fourth decoding layer 5024, and a fifth decoding layer 5025 connected in sequence. The first encoding layer 5011 is connected to the first decoding layer 5021, the second encoding layer 5012 is connected to the second decoding layer 5022, the third encoding layer 5013 is connected to the third decoding layer 5023, the fourth encoding layer 5014 is connected to the fourth decoding layer 5024, and the fifth encoding layer 5015 is connected to the fifth decoding layer 5025.

[0053] In some embodiments, the human head scale density features of multiple feature levels include i human head scale density features of feature levels, i is a positive integer greater than or equal to 3;

[0054] The multi-scale fusion features are encoded and decoded to obtain human head scale density features of multiple feature levels, including:

[0055] In the encoding stage, the multi-scale fusion features are processed by step-by-step down-sampling to obtain down-sampled features of the first to jth feature levels, j is a positive integer greater than or equal to 3;

[0056] In the decoding stage, the down-sampled features of the jth feature level are determined as the human head scale density features of the ith feature level, and the human head scale density features of the i-1th to first feature levels are determined according to the human head scale density features of the ith feature level and the down-sampled features of the j-pth feature level, p = 1, 2,..., (j-1).

[0057] ​In an example embodiment, assuming i = 5, j = 5, in the encoding stage, the multi-scale fusion feature F8 can be sequentially input into the first encoding layer 5011, the second encoding layer 5012, the third encoding layer 5013, the fourth encoding layer 5014 and the fifth encoding layer 5015 of the encoding network layer 501 for step-by-step down-sampling processing, respectively outputting the down-sampling features E1 (corresponding to the output of the first encoding layer 5011), E2 (corresponding to the output of the second encoding layer 5012), E3 (corresponding to the output of the third encoding layer 5013), E4 (corresponding to the output of the fourth encoding layer 5014) and E5 (corresponding to the output of the fifth encoding layer 5015) of the first to fifth (j = 5) feature levels. Next, in the decoding stage, the down-sampling feature E5 of the fifth feature level (determined as the head scale density feature D5 of the fifth feature level) can be input into the fourth decoding layer 5024. According to the head scale density feature of the fifth feature level and the down-sampling features of the fifth to first (p = 1, 2, 3, 4) feature levels, the head scale density features of the fourth to first feature levels are determined.

[0058] In some embodiments, according to the head scale density feature of the i-th feature level and the down-sampling features of the j-p-th feature level, the head scale density features of the i-th to first feature levels are determined, which can be specifically determined by up-sampling the head scale density feature of the i-th feature level to obtain an up-sampling feature; and aggregating the up-sampling feature with the down-sampling features of the j-p-th feature level to obtain the head scale density feature of the i-q-th feature level, q = 1, 2,... (i-1); p = q.

[0059] In combination with the above examples, in combination with Figure 5 and Figure 6 When i = 1, Figure 6 D i+1 is D5, D i is D4, E i is E4. First, the head scale density feature D5 output from the fifth decoding layer 5025 is input into the fourth decoding layer 5024 for up-sampling processing to obtain an up-sampling feature with the same resolution size as the output of the fourth decoding layer 5024, and then the up-sampling feature is aggregated with the down-sampling feature E4 output by the fourth encoding layer 5014 with the same resolution size, and then a convolution kernel of 3x3 is used for convolution processing (conv3) to obtain the head scale density feature D4 of the fourth feature level.

[0060] Similarly, the human head scale density feature D4 output from the fourth decoding layer 5024 is input into the third decoding layer 5023 for upsampling processing to obtain an upsampling feature with the same resolution size as the output of the third decoding layer 5023, and then the upsampling feature is aggregated with the down-sampling feature E3 output by the third encoding layer 5013 with the same resolution size, and then a convolution processing (conv3) with a 3x3 convolution kernel is performed to obtain the human head scale density feature D3 of the third feature level. The human head scale density feature D3 output from the third decoding layer 5023 is input into the second decoding layer 5022 for upsampling processing to obtain an upsampling feature with the same resolution size as the output of the second decoding layer 5022, and then the upsampling feature is aggregated with the down-sampling feature E2 output by the second encoding layer 5012 with the same resolution size, and then a convolution processing (conv3) with a 3x3 convolution kernel is performed to obtain the human head scale density feature D2 of the second feature level. The human head scale density feature D2 output from the second decoding layer 5022 is input into the first decoding layer 5021 for upsampling processing to obtain an upsampling feature with the same resolution size as the output of the first decoding layer 5021, and then the upsampling feature is aggregated with the down-sampling feature E1 output by the first encoding layer 5011 with the same resolution size, and then a convolution processing (conv3) with a 3x3 convolution kernel is performed to obtain the human head scale density feature D1 of the first feature level.

[0061] In the encoding stage, the feature scale of the down-sampling feature of the multi-scale fusion feature F8 is gradually reduced from E1 to E5, and in the decoding stage, the feature scale of the human head scale density feature is gradually increased from D5 to D1, which can enable the multi-scale fusion feature to be gradually transmitted in the encoding-decoding network of the pyramid structure, so as to facilitate subsequent adaptive learning of the optimal learning region and the human head scale of the region in the feature space with different resolutions. Among them, the rich detail information in the high-resolution feature layer (such as the first decoding layer 5021) is conducive to the prediction of small-scale heads, and the low-resolution feature layer (such as the fifth decoding layer 5025) with rich context information is better for the prediction of large-scale heads.

[0062] Through the above steps, the original input crowd image as shown in FIG. 1A can be processed into a human head density distribution map as shown in FIG. 1B. Figure 2 Figure 7 As shown in FIG. 1C, from bottom to top, the human head density distribution area corresponding to the P1 feature level is shown as the purple point distribution area in FIG. 1C, the human head density distribution area corresponding to the P2 feature level is shown as the yellow point distribution area in FIG. 1C, the human head density distribution area corresponding to the P3 feature level is shown as the green point distribution area in FIG. 1C, the human head density distribution area corresponding to the P4 feature level is shown as the cyan point distribution area in FIG. 1C, and the human head density distribution area corresponding to the P5 feature level is shown as the blue point distribution area in FIG. 1C. Figure 7 Figure 7 As shown in FIG. 1C, from bottom to top, the human head density distribution area corresponding to the P1 feature level is shown as the purple point distribution area in FIG. 1C, the human head density distribution area corresponding to the P2 feature level is shown as the yellow point distribution area in FIG. 1C, the human head density distribution area corresponding to the P3 feature level is shown as the green point distribution area in FIG. 1C, the human head density distribution area corresponding to the P4 feature level is shown as the cyan point distribution area in FIG. 1C, and the human head density distribution area corresponding to the P5 feature level is shown as the blue point distribution area in FIG. 1C. Figure 7 ​​The blue dot distribution area is shown in the image. The head density distribution area corresponding to the P3 feature level is as follows: Figure 7 As shown in the green dot distribution area, the head density distribution area corresponding to the P4 feature level is as follows: Figure 7 As shown in the red dot distribution area, the head density distribution area corresponding to the P5 feature level is as follows: Figure 7 The yellow dot distribution area is shown. The five feature levels {P1, P2, P3, P4, P5} represent the cases where the head size gradually decreases and the head density gradually increases, respectively.

[0063] In practical applications, the number of feature levels (feature density levels) can be appropriately reduced according to the difference in the head density distribution to be fitted. For example, the number of feature levels for the final output head-scale density feature can be determined to be 3 or 4 based on the difference in the head density distribution to be fitted.

[0064] In practical applications, the preferred encoding-decoding network with a pyramid structure is as follows: Figure 5 The network structure shown can be customized by selecting any number of layers to form three or four levels of head-scale density features, depending on the difference in head density distribution to be fitted.

[0065] Step S104: Determine the predicted density map corresponding to the head-scale density feature at each feature level.

[0066] Figure 8 This is a schematic diagram of a density branch structure provided in an embodiment of this disclosure. For example... Figure 8 As shown, the density branch includes five density heads: a first density head 801, a second density head 802, a third density head 803, a fourth density head 804, and a fifth density head 805. The first density head 801 is connected to the first decoding layer 5021, the second density head 802 is connected to the second decoding layer 5022, the third density head 803 is connected to the third decoding layer 5023, the fourth density head 804 is connected to the fourth decoding layer 5024, and the fifth density head 805 is connected to the fifth decoding layer 5025. Each density head includes three convolutional branches and one skip connection branch.

[0067] In some embodiments, step S204 may specifically include the following steps:

[0068] For each feature level of head-scale density feature, the head-scale density feature is subjected to the first convolution process to obtain the first convolution feature.

[0069] The first convolution feature is respectively subjected to second convolution processing, third convolution processing and fourth convolution processing to obtain second convolution feature, third convolution feature and fourth convolution feature, and the convolution kernel sizes of the second convolution processing, the third convolution processing and the fourth convolution processing are different.

[0070] The second convolution feature, the third convolution feature and the fourth convolution feature are concatenated to obtain a concatenated feature.

[0071] The concatenated feature is subjected to fifth convolution processing and up-sampling processing to obtain a prediction density map corresponding to each feature level.

[0072] In combination Figure 8 In combination with the above example, for the head scale density feature D1 corresponding to the P1 feature level, first, the head scale density feature D1 is input into the first density head 801, and is subjected to channel reduction through convolution operation with a convolution kernel of 1*1 to obtain a first convolution feature; then the first convolution feature is respectively input into three convolution branches with different convolution kernel sizes for second convolution processing, third convolution processing and fourth convolution processing to obtain context information of different receptive fields, and to obtain second convolution feature, third convolution feature and fourth convolution feature output by each convolution branch; then, the second convolution feature, the third convolution feature and the fourth convolution feature are input into a skip connection branch to concatenate them along the channel dimension to obtain a concatenated feature; finally, after convolution processing with a convolution kernel size of 1*1 and up-sampling processing, a prediction density map DM1 with the same size as the original input crowd image is obtained.

[0073] Similarly, referring to the method for obtaining the above prediction density map DM1, a prediction density map DM2 corresponding to the head scale density feature D2 of the P2 feature level, a prediction density map DM3 corresponding to the head scale density feature D3 of the P3 feature level, a prediction density map DM4 corresponding to the head scale density feature D4 of the P4 feature level, and a prediction density map DM5 corresponding to the head scale density feature D5 of the P5 feature level can be obtained.

[0074] In the embodiments of the present disclosure, each density head can use a real density map DM gt Supervised training is performed, and in the training process, the Euclidean distance between the prediction density map DM i and the real density map DM gt is used as a loss function to guide the optimization training of the density head. Finally, the losses of different feature level layers are summed to obtain a density loss function L density , as shown in the following formula (1):

[0075]

[0076] In formula (1), DM idenotes the predicted density map corresponding to the i-th feature level, i = 1 ~ 5; DM gt denotes the real density map of the original input crowd image.

[0077] In step S105, each predicted density map is divided into a plurality of image slices, the confidence label of each image slice at each feature level is calculated, and the optimal feature level for predicting the number of image slices is determined according to the confidence label.

[0078] In some embodiments, dividing each predicted density map into a plurality of image slices and calculating the confidence label of each image slice at each feature level includes:

[0079] Each predicted density map is down-sampled to obtain a plurality of image slices corresponding to each predicted density map.

[0080] For each image slice of each predicted density map, the image slice is convoluted to obtain image slice convolution features, and the confidence label of the image slice convolution features at the 1st to i-th feature levels is calculated.

[0081] Taking the predicted density map DM1 as an example, in the algorithm model training stage, the predicted density map DM1 can be down-sampled to the size of the original input crowd image , so that the predicted density map DM1 can be divided into k image slices with a size of k x k. Then, each image slice is convoluted with a 3 x 3 convolution kernel to obtain image slice convolution features.

[0082] As an example, assuming that the predicted density map DM1 is down-sampled to 1 / 4 of the size of the original input crowd image, so that the predicted density map DM1 can be divided into 4 image slices with a size of 4 x 4. Next, each image slice is convoluted with a 3 x 3 convolution kernel to obtain image slice convolution features corresponding to the 4 image slices. Then, the confidence label of each image slice convolution feature at the 1st to i-th feature levels is calculated. Assuming that there are three feature levels, p1 feature level, p2 feature level and p3 feature level, the confidence label of each image slice convolution feature at the p1 feature level, the p2 feature level and the p3 feature level is calculated.

[0083] In some embodiments, calculating the confidence label of the image slice convolution features at the 1st to i-th feature levels includes:

[0084] Obtaining a real density map corresponding to the crowd image, the real density map carrying a real count label;

[0085] Extract the true density image slices from the true density map that correspond to the convolutional features of the image slices;

[0086] Determine the slice count value corresponding to the true density image slice based on the true count label;

[0087] Determine the predicted count values ​​of the image slice convolutional features at the 1st to ith feature levels;

[0088] Based on the predicted count and the slice count, the confidence labels of the image slice convolutional features at the 1st to 1st feature levels are calculated.

[0089] A true density map is a point map showing the distribution of head density points corresponding to the original input image of a crowd. This true density map is marked with head points and actual number of people labels (i.e., actual count labels), typically with one head point representing one person.

[0090] As an example, following the same image slicing rules as described above for the predicted density map DM1, the true density map is also divided into four 4×4 true density image slices. Then, the slice count value (i.e., the actual number of people in the true density image slice region) corresponding to the image slice position corresponding to the image slice convolution feature is extracted. Next, the predicted number of people (i.e., the predicted count value) at the p1, p2, and p3 feature levels are obtained by the image slice convolution features respectively. Then, the counting error between the predicted number of people at the p1, p2, and p3 feature levels obtained by the image slice convolution features and the slice count value of the true density image slice corresponding to the image slice convolution feature position is calculated. The positive label of 1 is assigned to the feature level whose estimated number of people is closest to the true value, and the negative label of 0 is assigned to the feature level with the largest prediction error, using the following formula (2). The label of other feature layers is -1, which is ignored during the training of the algorithm model.

[0091]

[0092] In equation (2), This represents the confidence label of the image slice located at position (m,n) in the predicted density image at the i-th feature level, where i is a positive integer ≥ 3, and N is a positive integer. l,m,n This represents the predicted number of people for the image slice located at position (m,n) in the predicted density image at the l-th feature level. Let l represent the number of real people in the real density image slice located at position (m,n) of the real density image at the l-th feature level, where l is a positive integer ≥ 3 and l = i.

[0093] Figure 9 This is a schematic diagram of the overall network structure of a crowd counting network provided in an embodiment of this disclosure.Figure 9 As shown, the crowd counting network comprises an input end, a multi-scale feature fusion network 300 connected with the input end, a pyramid structure encoding-decoding network 500 connected with the multi-scale feature fusion network 300, a first density head 801, a second density head 802, a third density head 803, a fourth density head 804 and a fifth density head 805 connected with a first decoding layer 5021, a second decoding layer 5022, a third decoding layer 5023, a fourth decoding layer 5024 and a fifth decoding layer 5025 in the pyramid structure encoding-decoding network 500 respectively, a first confidence branch 901, a second confidence branch 902, a third confidence branch 903, a fourth confidence branch 904 and a fifth confidence branch 905 connected with the first decoding layer 5021, the second decoding layer 5022, the third decoding layer 5023, the fourth decoding layer 5024 and the fifth decoding layer 5025 in the pyramid structure encoding-decoding network 500 respectively; a first fusion branch 906 connected with the first density head 801, the second density head 802, the third density head 803, the fourth density head 804 and the fifth density head 805; a second fusion branch 907 connected with the first confidence branch 901, the second confidence branch 902, the third confidence branch 903, the fourth confidence branch 904 and the fifth confidence branch 905; a third fusion branch 908 connected with the first fusion branch 906 and the second fusion branch 907; and an output end connected with the third fusion branch 908.

[0094] In combination Figure 9 and Figure 10For example, the predicted density map DM1 is input into the first confidence branch 901, and the predicted density map DM1 is divided into four image slices of the same size. Similarly, the true density map is also divided into four true density image slices of the same size. For the image slice at the upper left corner of the predicted density map DM1, the predicted number of people at the p1 feature level, the p2 feature level and the p3 feature level is respectively estimated, and the results are 16.53, 15.15 and 12.89 respectively. The true number of people of the true density image slice at the upper left corner of the true density map is 24.76, and the counting error between the predicted number of people at the p1 feature level, the p2 feature level and the p3 feature level and the true number of people of the true density image slice at the upper left corner of the true density map is calculated respectively. According to the calculation result, the counting error between the predicted number of people at the p1 feature level and the true number of people of the true density image slice at the upper left corner of the true density map is the smallest, and the counting error between the predicted number of people at the p2 feature level and the true number of people of the true density image slice at the upper left corner of the true density map is the largest. Therefore, the positive label of 1 is assigned to the p1 feature level by the above formula (1), the negative label of 0 is assigned to the p2 feature level, and the label of -1 is assigned to the p3 feature level. The p1 feature level is determined as the optimal feature level for estimating the number of people of the image slice at the upper left corner of the predicted density map DM1.

[0095] Similarly, the determination method of the confidence of the image slices at the upper right corner, the lower left corner and the lower right corner of the predicted density map DM1 can be determined one by one with reference to the determination method of the confidence of the image slice at the upper left corner of the predicted density map DM1, which will not be described here. According to the above steps, it can be finally determined that the feature levels corresponding to the confidence labels of 1 of the image slices at the upper left corner, the upper right corner, the lower left corner and the lower right corner of the predicted density map DM1 are the p1 feature level, the p2 feature level, the p2 feature level and the p1 feature level respectively.

[0096] The determination method of the confidence label of each image slice of the predicted density map DM2, the predicted density map DM3, the predicted density map DM4 and the predicted density map DM5 at each feature level can be determined one by one with reference to the determination method of the confidence label of each image slice of the predicted density map DM1 at each feature level, which will not be described here.

[0097] In the model inference process, different from the model training process described above, for each image slice of each prediction density map, after the image slice is convoluted to obtain the image slice convolution feature, the confidence score of the image slice convolution feature at the first to the i-th feature level is obtained through the Sigmoid activation function, and the feature level with the highest confidence is determined as the optimal feature level of the image slice convolution feature.

[0098] Since the receptive field of the specific feature level P i is limited, it is only applicable to predicting the head in a relatively narrow scale range. In order to make full use of the advantages of the extracted multi-scale fusion feature representation, the embodiments of the present disclosure first use the density branch to independently use P i for prediction in a scale-independent manner to obtain the prediction density map corresponding to each feature level; then, with the assistance of the confidence branch, the optimal feature level of each image slice in the prediction density map corresponding to each feature level is determined, and the final prediction result is obtained according to the optimal feature level. Since the heads in a specific image slice share approximately the same scale information, selecting the optimal feature level for estimating the number of people in the specific image slice through the confidence branch can achieve a lower counting error, which is conducive to reducing the undercounting in the head dense area and the overcounting in the head sparse area, and improving the accuracy of the number of people prediction.

[0099] Step S106, according to the optimal feature level corresponding to each image slice of each prediction density map, the number of people is predicted to obtain the crowd count of the crowd image.

[0100] In combination with the above examples, the optimal feature level corresponding to each image slice of the prediction density map DM1, the prediction density map DM2, the prediction density map DM3, the prediction density map DM4, and the prediction density map DM5 is used for number of people prediction to obtain the prediction number of people corresponding to each image slice of each prediction density map, and then all the prediction numbers of people are superimposed to obtain the crowd count of the crowd image.

[0101] The technical scheme provided by the embodiments of the present disclosure can first acquire a crowd image, extract multi-scale fusion features of the crowd image, and can accurately depict the continuously changing head scale and head density features in a dense crowd scene without increasing additional parameters, and the head positioning is also more accurate. Then, the multi-scale fusion features are encoded and decoded to obtain head scale and density features of multiple feature levels. Then, the prediction density map corresponding to each feature level head scale and density feature is determined according to the density head corresponding to each feature level. Each prediction density map is divided into multiple image slices through the confidence branch corresponding to each feature level, the confidence label of each image slice on each feature level is calculated, and the optimal feature level used for predicting the number of image slices is determined according to the confidence label. The number of people is predicted according to the optimal feature level corresponding to each image slice of each prediction density map, and the crowd counting of the crowd image is obtained. The optimal feature level corresponding to each image slice can be adaptively selected, the feature details are more perfect and accurate, the undercounting of the head dense area and the overcounting of the head sparse area can be effectively reduced, and the crowd counting accuracy in the dense crowd scene can be improved.

[0102] In some embodiments, after obtaining the final prediction density map corresponding to each feature level using the adaptive selection strategy described above, the pixel area with difficult counting in the density map can be further selected by PRA (propagation heuristic graph search algorithm) loss, and fine-grained optimization is performed thereon to improve the accuracy of the number prediction.

[0103] All the optional technical solutions described above can be combined to form optional embodiments of the present application, which will not be described one by one here.

[0104] The following is an embodiment of the device of the present disclosure, which can be used to execute the method embodiments of the present disclosure. For details not disclosed in the device embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

[0105] Figure 11 is a schematic diagram of a crowd counting device provided by an embodiment of the present disclosure. As shown in Figure 10 , the crowd counting device comprises:

[0106] The acquisition module 1101 is configured to acquire a crowd image.

[0107] The extraction module 1102 is configured to extract multi-scale fusion features of the crowd image.

[0108] The encoding and decoding module 1103 is configured to encode and decode the multi-scale fusion features to obtain head scale and density features of multiple feature levels.

[0109] The determining module 1104 is configured to determine a prediction density map corresponding to each head scale density feature at each feature level;

[0110] The calculating module 1105 is configured to divide each prediction density map into a plurality of image slices, calculate a confidence label of each image slice at each feature level, and determine an optimal feature level for predicting the number of people in the image slice according to the confidence label;

[0111] The counting module 1106 is configured to predict the number of people according to the optimal feature level corresponding to each image slice of each prediction density map to obtain the crowd count of the crowd image.

[0112] In some embodiments, the extracting module 1102 can specifically include:

[0113] A first convolution unit configured to perform one convolution processing on the crowd image to obtain first scale convolution features;

[0114] A second convolution unit configured to perform second, third, fourth and fifth convolution processing on the first scale convolution features to obtain second scale convolution features, third scale convolution features, fourth scale convolution features and fifth scale convolution features;

[0115] A first fusion unit configured to fuse the second scale convolution features and the third scale convolution features to obtain first fusion features;

[0116] A second fusion unit configured to fuse the fourth scale convolution features and the fifth scale convolution features to obtain second fusion features;

[0117] A third fusion unit configured to fuse the first fusion features and the second fusion features to obtain multi-scale fusion features.

[0118] In some embodiments, the plurality of head scale density features at each feature level includes i head scale density features at each feature level, and i is a positive integer greater than or equal to 3. The encoding and decoding module 1103 can specifically include:

[0119] An encoding unit configured to perform step-by-step down-sampling processing on the multi-scale fusion features in the encoding stage to obtain down-sampling features at the first to jth feature levels, and j is a positive integer greater than or equal to 3;

[0120] A decoding unit configured to determine the down-sampling features at the jth feature level as the head scale density features at the ith feature level in the decoding stage, and determine the head scale density features at the i-1th to first feature levels according to the head scale density features at the ith feature level and the down-sampling features at the j-pth feature level, and p = 1, 2,..., (j-1).

[0121] In some embodiments, the decoding unit includes:

[0122] an up-sampling component configured to up-sample the human head scale density feature of the i-th feature level to obtain an up-sampled feature;

[0123] an aggregating component configured to aggregate the up-sampled feature and the down-sampled feature of the j-p-th feature level to obtain a human head scale density feature of an i-q-th feature level, q = 1, 2, … (i-1); p = q.

[0124] In some embodiments, the determining module 1104 specifically includes:

[0125] a first convolution unit configured to, for each human head scale density feature of each feature level, perform first convolution processing on the human head scale density feature to obtain a first convolution feature;

[0126] a second convolution unit configured to perform second convolution processing, third convolution processing and fourth convolution processing on the first convolution feature respectively to obtain a second convolution feature, a third convolution feature and a fourth convolution feature, the convolution kernel sizes of the second convolution processing, the third convolution processing and the fourth convolution processing being different;

[0127] a concatenating unit configured to concatenate the second convolution feature, the third convolution feature and the fourth convolution feature to obtain a concatenated feature;

[0128] a third convolution unit configured to perform fifth convolution processing and up-sampling processing on the concatenated feature to obtain a prediction density map corresponding to each feature level.

[0129] In some embodiments, the calculating module 1105 includes:

[0130] a slicing unit configured to perform down-sampling processing on each prediction density map to obtain a plurality of image slices corresponding to each prediction density map;

[0131] a calculating unit configured to, for each image slice of each prediction density map, perform convolution processing on the image slice to obtain an image slice convolution feature, and calculate a confidence label of the image slice convolution feature on the 1st to i-th feature level.

[0132] In some embodiments, the calculating unit includes:

[0133] an obtaining component configured to obtain a real density map corresponding to the crowd image, the real density map carrying a real counting label;

[0134] a clipping component configured to clip out a real density image slice corresponding to the image slice convolution feature in the real density map.

[0135] The first determining component is configured to determine a slice count value corresponding to the real density image slice according to the real count label;

[0136] The second determining component is configured to determine a predicted count value of the image slice convolution feature at the first to i-th feature level.

[0137] The calculating component is configured to calculate a confidence label of the image slice convolution feature at the first to i-th feature level according to the predicted count value and the slice count value.

[0138] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0139] Figure 12 is a schematic diagram of an electronic device 12 provided by the embodiments of the present disclosure. As shown in the figure, the electronic device 12 of the embodiment includes a processor 1201, a memory 1202, and a computer program 1203 stored in the memory 1202 and executable on the processor 1201. The processor 1201 implements the steps in each of the above method embodiments when executing the computer program 1203. Alternatively, the processor 1201 implements the functions of each module / unit in each of the above device embodiments when executing the computer program 1203. Figure 12

[0140] The electronic device 12 can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The electronic device 12 can include but is not limited to the processor 1201 and the memory 1202. Those skilled in the art can understand that the electronic device 12 is only an example of the electronic device 12 and does not constitute a limitation on the electronic device 12, and can include more or fewer components than the diagram or different components. Figure 12 The processor 1201 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.

[0141] The processor 1201 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.

[0142] ​The memory 1202 can be an internal storage unit of the electronic device 12, for example, a hard disk or a memory of the electronic device 12. The memory 1202 can also be an external storage device of the electronic device 12, for example, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, or the like equipped on the electronic device 12. The memory 1202 can also include both an internal storage unit and an external storage device of the electronic device 12. The memory 1202 is used to store computer programs and other programs and data required by the electronic device.

[0143] It should be clearly understood by those skilled in the art that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0144] The integrated module / unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can be executed by a processor to implement the steps of each method embodiment described above. The computer program can include computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the contents included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0145] The above examples are only used to illustrate the technical solutions of the present disclosure, rather than limit the same; although the present disclosure has been described in detail with reference to the foregoing examples, it should be understood by those of ordinary skill in the art that the technical solutions recorded in the foregoing examples can still be modified, or some technical features thereof can be replaced by equivalent replacements; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure.

Claims

1. A method for counting people, characterized in that, include: Acquire crowd images; Extract multi-scale fusion features from the crowd images; The multi-scale fusion features are encoded and decoded to obtain head-scale density features at multiple feature levels; Determine the predicted density map corresponding to the head-scale density feature at each feature level; Each of the predicted density maps is divided into multiple image slices, and the confidence label of each image slice at each feature level is calculated. Based on the confidence label, the optimal feature level for predicting the number of people in the image slice is determined. The number of people is predicted based on the optimal feature level corresponding to each image slice of the predicted density map, and the crowd count of the crowd image is obtained.

2. The method according to claim 1, characterized in that, Extracting multi-scale fusion features from the crowd image includes: Perform a convolution process on the crowd image to obtain the first-scale convolutional features; By performing second, third, fourth, and fifth convolution processes on the first-scale convolution features respectively, convolution features at the second, third, fourth, and fifth scales are obtained. The second-scale convolutional features and the third-scale convolutional features are fused to obtain the first fused feature; The fourth-scale convolutional features and the fifth-scale convolutional features are fused to obtain the second fused feature; The first fusion feature and the second fusion feature are fused to obtain a multi-scale fusion feature.

3. The method according to claim 1, characterized in that, The multiple feature levels of head-scale density features include i feature levels of head-scale density features, where i is a positive integer ≥ 3; The multi-scale fusion features are encoded and decoded to obtain head-scale density features at multiple feature levels, including: During the encoding stage, the multi-scale fusion features are downsampled step by step to obtain downsampled features at the 1st to jth feature levels, where j is a positive integer ≥3; During the decoding stage, the downsampled feature of the j-th feature level is determined as the head-scale density feature of the i-th feature level. Based on the head-scale density feature of the i-th feature level and the downsampled feature of the jp-th feature level, the head-scale density features of the (i-1)-1 to 1-1 feature levels are determined, where p = 1, 2, ... (j-1).

4. The method according to claim 3, characterized in that, Based on the head-scale density features at the i-th feature level and the downsampling features at the jp-th feature level, the head-scale density features at the (i-1)-1 to 1-th feature levels are determined, including: Upsample the head-scale density feature at the i-th feature level to obtain the upsampled feature; The upsampled features are aggregated with the downsampled features at the jp-th feature level to obtain the head-scale density features at the iq-th feature level, where q = 1, 2, ... (i-1); p = q.

5. The method according to claim 1, characterized in that, Determine the predicted density map corresponding to the head-scale density feature at each feature level, including: For each feature level of human head scale density feature, the human head scale density feature is subjected to a first convolution process to obtain the first convolution feature; The first convolutional feature is subjected to a second convolutional process, a third convolutional process, and a fourth convolutional process respectively to obtain a second convolutional feature, a third convolutional feature, and a fourth convolutional feature. The convolutional kernel sizes of the second convolutional process, the third convolutional process, and the fourth convolutional process are different. The second, third, and fourth convolutional features are concatenated to obtain concatenated features; The cascaded features are subjected to a fifth convolution and upsampling process to obtain the prediction density map corresponding to each feature level.

6. The method according to claim 3, characterized in that, Each of the predicted density maps is divided into multiple image slices, and the confidence label of each image slice at each feature level is calculated, including: Each of the predicted density maps is downsampled to obtain multiple image slices corresponding to each of the predicted density maps; For each image slice of each predicted density map, the image slice is convolved to obtain the image slice convolutional features, and the confidence labels of the image slice convolutional features at the 1st to ith feature levels are calculated.

7. The method according to claim 6, characterized in that, Calculating the confidence labels of the image slice convolutional features at the 1st to ith feature levels includes: Obtain the true density map corresponding to the crowd image, wherein the true density map carries a true count label; Extract the real density image slices from the real density map that correspond to the convolutional features of the image slices; The slice count value corresponding to the real density image slice is determined based on the real count label; Determine the predicted count values ​​of the image slice convolutional features at the 1st to 1st feature levels; Based on the predicted count and the slice count, the confidence labels of the image slice convolutional features at the 1st to 1st feature levels are calculated.

8. A crowd counting device, characterized in that, include: The acquisition module is configured to acquire images of crowds; The extraction module is configured to extract multi-scale fusion features from the crowd image; The encoding / decoding module is configured to encode and decode the multi-scale fusion features to obtain head-scale density features at multiple feature levels. The determination module is configured to determine the predicted density map corresponding to the head-scale density feature at each feature level. The calculation module is configured to divide each of the predicted density maps into multiple image slices, calculate the confidence label of each image slice at each feature level, and determine the optimal feature level for predicting the number of people in the image slice based on the confidence label. The counting module is configured to predict the number of people based on the optimal feature level corresponding to each image slice of the predicted density map, and obtain the crowd count of the crowd image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images