Crowd counting method based on scale attention

Abstract

The invention relates to the technical field of crowd counting, and discloses a scale attention-based crowd counting method, which comprises the following steps of: performing feature extraction on an input image; the multi-scale-level features pass through a scale attention module and a feature fusion module, and information from all low-resolution layers is aggregated and inherited at the highest-scale-level resolution layer; on each resolution level, inputting the fusion features into a density map generation module to obtain predicted density maps under different scale levels; designing a loss function based on a multi-expert thought, and training the crowd counting model; and preprocessing a to-be-predicted crowd image, inputting the preprocessed to-be-predicted crowd image into the trained crowd counting model, and outputting a prediction density map at the highest scale level resolution layer to realize crowd number prediction. According to the method, the internal relation between different feature levels and different pedestrian head scales is explored, so that scale sensing capabilities under different feature levels are fully utilized and integrated, and accurate crowd counting is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of crowd counting, and in particular to a crowd counting method based on scale attention. Background Art

[0002] Crowd counting refers to the algorithmic analysis of images or video streams acquired in a specific area, and the total number of pedestrians in the area is calculated. Early traditional crowd counting methods mainly relied on manual feature extraction, while the current popular crowd counting methods are mainly based on deep learning, usually using convolutional neural networks (CNN) or Transformer architectures. Among the crowd counting algorithms based on deep learning, using density maps to calculate the total number of people is the mainstream method. The overall process of using density maps to achieve crowd counting is as follows: first, mark a point at each pedestrian position in the image, and then use a two-dimensional Gaussian kernel function to convert these discrete marked points into a true density map at the pixel level, which is used as a supervisory signal for network training. Each pixel value in the density map reflects the crowd density at that location. By integrating the entire density map, the total number of pedestrians in the image can be obtained.

[0003] Considering the frequent occurrence of large-scale crowd gatherings, it is necessary to count the number of people and estimate the density in public places prone to crowds. However, the application of crowd counting is still greatly limited at this stage. The problem of inconsistent head size in images has attracted the attention of most researchers. Since the distance between people and cameras is changing, the size of heads in the distance is small, and the size of heads in the near distance is large. This change is called scale change, which leads to perspective distortion in the captured images. In addition, the different positions and heights of the cameras cause differences in shooting angles. The perspective distortion and the difference in shooting angles cause the target scales in different areas of the image to differ greatly. The heads in an image show multi-scale changes, which makes crowd counting difficult.

[0004] In order to solve the problem of drastic changes in crowd size in dense scenes, current researchers have improved crowd counting algorithms and network structures, mainly making the following explorations and attempts:

[0005] (1) Multi-column network structure:

[0006] The earliest crowd counting model based on CNN adopted a single branch structure and used a fixed convolution kernel to extract features. However, this fixed-size convolution kernel cannot capture multi-scale features due to its fixed receptive field, which inevitably leads to serious counting errors. In order to solve the scale change problem, some researchers use multi-column CNN models (such as MCNN networks) or use hierarchical structures to learn multi-scale representations from different layers of the backbone network. Each column of MCNN is composed of convolution layers with different receptive fields. The features learned by multi-column CNN are helpful for dealing with the problem of uneven head sizes in images. In the multi-column CNN model, larger convolution kernels can extract larger scale information, and smaller convolution kernels extract smaller scale information.

[0007] However, this multi-column network has obvious defects: (I) The performance of this type of method depends on the number of network branches. Although more columns or branches can be used to enrich the receptive field, the number of network parameters and the amount of calculation will increase greatly with each additional column or branch, and real-time counting is difficult. (II) The size of each column of convolution kernel can only be set manually based on experience, and the size of the selected convolution kernel, such as 3×3, 5×5, etc., is a discrete value, which does not take into account the continuity of scale changes, and also leads to feature redundancy between column branches, and ultimately generates unclear density maps.

[0008] (2) A single network integrates multi-scale features:

[0009] Given the limitations of multi-column convolutional networks, subsequent researchers have mainly tried to fuse multi-scale features with a single network, such as using a hierarchical structure to learn multi-scale representations from different layers of the backbone network. However, these methods simply add or connect the extracted features of different scales. This fusion method is not actually effective in adapting to multi-scale situations, because the importance of features of different scales in the image is likely to be different, and contain noise and redundant information, which needs to be screened or assigned different weights during the fusion process. For this reason, some subsequent methods use visual attention mechanisms to fuse multi-scale features.

[0010] Despite the significant progress made by these multi-scale methods, most existing methods aggregate features from multiple levels in a scale-independent manner by utilizing all feature levels to generate the final prediction, ignoring the intrinsic correspondence between feature levels and head scales. This feature aggregation may lead to mutual suppression of features at different scales, as it is extremely challenging to accurately compute the same object at all scales.

[0011] In summary, existing crowd counting methods are still insufficient in dealing with changes in crowd size. Therefore, further optimizing the solution to the problem of crowd size changes is crucial to improving the performance of crowd counting algorithms. Summary of the invention

[0012] The significant change in crowd size is the most important factor that restricts the performance of crowd counting algorithms. In different scenes, whether due to different camera positions or natural changes in camera angles in the same environment, the distance between the target crowd in the image and the camera will be different, which will cause the perspective effect in the image. Specifically, pedestrians close to the camera will appear larger and contain more crowd details; while pedestrians far away from the camera will appear smaller and their information richness will be greatly reduced. This imbalance in the amount of information makes it extremely difficult to accurately identify people of different sizes in the same image.

[0013] In view of the shortcomings of existing crowd counting methods in dealing with changes in crowd scale, the present invention provides a crowd counting method based on scale attention, which improves the structure of the traditional multi-branch convolutional neural network, explores the intrinsic connection between different feature levels and different pedestrian head scales, and fully utilizes and integrates the scale perception capabilities at different feature levels, so as to finally adapt to the complex and changeable changes in head scale in dense crowd scenes, generate high-quality crowd density maps, and achieve accurate crowd counting.

[0014] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0015] A crowd counting method based on scale attention, wherein the crowd counting model adopted includes a feature extraction module, a scale attention module, a feature fusion module and a density map generation module; the crowd counting method includes the following steps:

[0016] Step 1: Use the HRNet network as a feature extraction module to extract features from the input image, thereby obtaining features at multi-scale resolution layers, i.e., multi-scale level features;

[0017] Step 2: Pass the multi-scale level features through the scale attention module and the feature fusion module: from the low scale level to the high scale level, the first fused feature of the current scale level is input into the scale attention module, and the first fused feature is decoupled into a scale-strongly correlated feature whose correlation with the current scale level is higher than the set condition, and a scale-weakly correlated feature whose correlation with the current scale level is weaker than the set condition; then the scale-strongly correlated feature corresponding to the current scale level and the feature under the resolution layer of the previous scale level are input into the feature fusion module for feature fusion, and the second fused feature Out of the previous scale level is obtained. i-1 ; Input the scale-strong correlation features and scale-weak correlation features corresponding to the current scale level, as well as the features under the resolution layer of the previous scale level, into the feature fusion module for feature fusion to obtain the first fusion feature of the previous scale level Finally, the highest-scale resolution layer aggregates and inherits information from each lower-scale resolution layer to adapt to the changes in the scale of the human head in the image;

[0018] Step 3: At each scale level, the second fused feature is input into the density map generation module to obtain the predicted density map at different scale levels.

[0019] Step 4: Design a loss function based on the multi-expert idea and train the crowd counting model: Adopt an expert selection strategy based on image blocks to adaptively select and learn different density areas at each resolution layer; Calculate the loss between the predicted density and the true value density in the corresponding areas of different resolution layers, and finally obtain the final loss function of the crowd counting model by weighted summing the losses corresponding to each resolution layer, train the crowd counting model, and obtain a trained crowd counting model;

[0020] Step 5: In the inference stage, the crowd image to be predicted is preprocessed and input into the trained crowd counting model; the crowd counting model will output the predicted density map at the highest scale resolution layer to predict the number of people.

[0021] Furthermore, the scale attention module includes a strong correlation feature extraction module, a weak correlation feature extraction module and a scale feature decoupler;

[0022] Multi-scale features are recorded as N is the total number of scale levels of the resolution layer, F i is the feature at the resolution layer of the i-th scale level, from F0 to F N-1 The number of channels doubles and the spatial resolution halves successively. The spatial resolution of the feature at the i-th scale level resolution layer is expressed as (h i ,w i )=(H / 2 i+2 ,W / 2 i+2 ); H and W represent the height and width of the input image respectively;

[0023] The scale feature decoupler consists of three convolutional layers. The first two convolutional layers are followed by a batch normalization layer and a ReLU activation function layer. The last convolutional layer is used to reduce the number of channels and generate two attention maps in the channel dimension through the Softmax function, which are recorded as the first attention map A and A respectively. s and the second attention map A w ;

[0024] Both the strong correlation feature extraction module and the weak correlation feature extraction module include two convolutional layers and corresponding batch normalization layers and ReLU activation function layers. The first fusion feature of the i-th scale level After being processed by the strong correlation feature extraction module and the weak correlation feature extraction module respectively, they will be decoupled into a strong correlation feature with a correlation with the i-th scale level higher than the set condition and a scale weak correlation feature with a correlation with the specific scale level weaker than the set condition; the two attention maps are matrix-multiplied with the scale strong correlation feature and the scale weak correlation feature respectively to obtain the final scale strong correlation feature S of the i-th scale level i and scale-weakly correlated feature W i :

[0025]

[0026] in, represents the strong correlation feature extraction module, represents the weak correlation feature extraction module, θ s and θ w are the learnable parameters of the strong correlation feature extraction module and the weak correlation feature extraction module respectively.

[0027] Furthermore, when the feature fusion module performs feature fusion, it specifically includes:

[0028] The feature F at the i-th scale level resolution layer i Decoupled scale-dependent features S i , and the feature F at the i-1th scale resolution layer i-1 , input to the feature fusion module EFF for feature fusion, and obtain the second fusion feature Out at the i-1th scale level i-1 , input to the density map generation module:

[0029] Out i-1 =EFF{F i-1 , S i}

[0030] The feature F at the i-th scale level resolution layer i Decoupled scale-dependent features S i and scale-weakly correlated feature W i , and the feature F at the i-1th scale resolution layer i-1 Input to the feature fusion module for feature fusion, the first fusion feature of the i-1th scale level

[0031]

[0032] Among them, S i +W i Indicates S i and W i Perform element-wise addition.

[0033] Furthermore, step three specifically includes:

[0034] The density map generation module includes two DySample upsamplers, a context-aware module, and a convolution operation;

[0035] The second fusion feature Out i-1 After passing through the DySample upsampler, two convolution operations, the DySample upsampler, the context-aware module and two convolution operations, the predicted density map is obtained;

[0036] Among them, the context-aware module captures the association between different regions in the feature map through multi-scale observation and weighted fusion mechanism, which can enhance the spatial context information of the feature map;

[0037] Density map generation module Only the highest scale level resolution layer is trained, and the other scale levels resolution layers have copies, density map generation module The parameter θ c The resolution layers at other scale levels will remain frozen.

[0038] Furthermore, step four specifically includes:

[0039] Define the binary mask M i , binary mask M i The pixel value can only be 0 or 1, and the binary mask M i Indicates the area that is most suitable for prediction at the current resolution layer; introduces the sum of binary masks With M ii The following relationship exists:

[0040]

[0041] After passing through the density map generation module, the predicted density maps on branches with different resolutions will be output True value density map The point annotation is generated by mapping the point annotation by a Gaussian kernel function of a fixed size; the point annotation is a manual annotation of the center point of the head on the image; that is, the point annotation is a point annotation, not a box representation in the detection field.

[0042] Next, the predicted density maps and true density maps in the resolution layers of different scale levels are divided into X×Y image blocks, where X and Y represent the number of divisions in the height and width directions respectively:

[0043]

[0044] p h ,p w All are fixed values ​​256. Represents the height and width of the image block at the resolution layer of the i-th scale level;

[0045] In order to determine the optimal resolution of a given image block, the cost calculation formula is defined to obtain the most suitable resolution layer for the image block:

[0046]

[0047] Label MoE = arg min(Cost x，y )

[0048]

[0049] Among them, Cost x,y (i) Records the cost of different scale-level resolution layers on X×Y image blocks. The first term is the true value density of a certain scale-level resolution layer at (x, y) of the image block. With predicted density The average mean square error between the two, the second term is the instance mean square error, e is a constant; Label MoE It indicates the label that records the best resolution of each of the X×Y image blocks; scatter(·) indicates the scatter function for Label MoE Transform to generate binary masks M of different scale levels of resolution i , and further calculated

[0050] During the training process, the loss function l is defined for the resolution layers of different scale levels. i ,

[0051]

[0052] Among them, L ED Indicates the calculation of Euclidean distance, is the output of the density map generation module, i.e., the predicted density map By weighting the loss functions of resolution layers at different scale levels, the final loss function L is obtained:

[0053]

[0054] N is the total number of scale levels of the resolution layer, λ i is the weight of the loss function for each resolution layer.

[0055] Compared with the prior art, the beneficial technical effects of the present invention are:

[0056] (1) The continuity of the change in head size is taken into account.

[0057] Compared with the earliest multi-column CNN models (such as MCNN networks, etc.), the present invention does not use multi-column and fixed-size convolution kernels to extract multi-scale features. Instead, it uses HRNet-W48 as the backbone network to progressively obtain multi-scale features from multiple resolution layers, fully considering the continuity of scale changes, which is helpful for further refinement of the extracted multi-scale features.

[0058] (2) The quality of the generated crowd density map is improved to achieve more accurate crowd counting.

[0059] Compared with other networks that use hierarchical structures to learn multi-scale representations, this paper explores the intrinsic connection between different feature levels and different pedestrian head scales, and decouples the features of each resolution layer so that specific feature levels can focus more on predictions within their appropriate scale range. In addition, this method also fully utilizes and integrates the scale perception capabilities of different feature levels, so that it can ultimately adapt to the changes in head scale in dense crowd scenes and achieve accurate crowd counting.

[0060] (3) The training process of the crowd counting model was optimized, and the robustness and generalization ability of the model were enhanced.

[0061] Compared with many crowd counting models that use a single loss function, this method introduces a hybrid loss function based on the idea of ​​multiple experts to constrain the learning process of density maps at each level, adaptively select appropriate prediction areas for each resolution layer, and continuously optimize during the training process to ensure the improvement of model performance to adapt to the complex and changeable changes in head scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a flow chart of the method of the present invention;

[0063] Figure 2 It is the principle diagram of the present invention;

[0064] Figure 3 It is a structural diagram of the crowd counting model of the present invention;

[0065] Figure 4 This is a sample visualization diagram of the prediction results of some images on the ShanghaiTech dataset using the method of the present invention;

[0066] Figure 5 is a structural schematic diagram of a feature extraction module according to an embodiment of the present invention;

[0067] Figure 6 Schematic diagram of the structure of the scale attention module of an embodiment of the present invention;

[0068] Figure 7 It is a structural schematic diagram of a feature fusion module according to an embodiment of the present invention;

[0069] Figure 8 It is a schematic diagram of the structure of a density map generating module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0071] Figure 1 The method flow of the present invention is shown. Specifically, it may include the following five steps:

[0072] Step 1: Use the HRNet-W48 network as the feature extraction module (backbone network) to extract features from the input image, thereby obtaining different features of multiple resolution layers, namely multi-scale level features.

[0073] Step 2: For the multi-scale features extracted by the backbone network, these features are successively passed through the scale attention module and the feature fusion module. Specifically, at each resolution layer from low-scale level to high-scale level, the extracted features are first decoupled into scale-strongly correlated features and scale-weakly correlated features so that they can be processed independently. Then, the high-resolution feature representation is continuously inherited from the useful feature representation from the low-resolution to achieve further feature fusion. Finally, the effective information from each low-resolution layer is aggregated and inherited at the highest scale level resolution layer to better adapt to the drastic changes in the scale of the human head.

[0074] Step 3: At each scale level, one of the output branches after feature fusion will be input into the density map generation module to obtain the predicted density maps at different scale levels.

[0075] Step 4: Design a loss function based on the idea of ​​multiple experts, train the crowd counting model, and obtain a trained crowd counting model.

[0076] Specifically, during the training phase, the crowd counting model adopts an expert selection strategy based on image blocks to adaptively select and learn different density areas at resolution layers of different scale levels; in the corresponding areas of resolution layers of different scale levels, the loss between the predicted density and the true density is calculated, and finally, the final loss function of the crowd counting model is obtained by adding the losses of each layer according to a certain weight, thereby imposing effective constraints on the crowd counting model.

[0077] Step 5: Prepare the crowd image to be predicted, and after certain preprocessing operations, input it into the trained crowd counting model; in the inference stage, the crowd counting model will output a high-quality predicted density map at the highest scale resolution layer and achieve accurate prediction of the number of people.

[0078] like Figure 3 As shown, the overall process of the crowd counting model is as follows:

[0079] The crowd counting model proposed in the present invention is mainly composed of a feature extraction module, a scale attention module, a feature fusion module and a density map generation module. Specifically, the crowd counting model uses the HRNet-W48 network as a feature extraction module to process the input image to obtain multi-scale level features. The multi-scale level features are then further processed by the scale attention module and the feature fusion module, and are sent to the density map generation module to obtain crowd density maps of different sizes. During the training stage, the crowd counting model adopts an expert selection strategy based on image blocks to adaptively select and learn different density areas at the resolution layers of each scale level, and imposes effective constraints on the crowd counting model by weighting the losses of each resolution layer. Figure 3 As shown in the figure, the dark gray parts in the density map indicate that these image areas will not participate in the calculation of the loss function, and the other unblocked parts reflect the best prediction areas at each resolution level. Finally, in the inference stage, the model will complete the accurate prediction of crowd density through the highest scale resolution layer.

[0080] The present invention conducts relevant experiments on the classic crowd counting dataset, ShanghaiTech dataset, which consists of two parts, Part A and Part B. The crowd size in Part A changes dramatically, while the crowd distribution in Part B is relatively sparse. The relevant experimental results are shown in Table 1. Figure 4 shown.

[0081] (1) Comparison of commonly used evaluation indicators of crowd counting algorithms

[0082] Table 1 shows the experimental results of the crowd counting model (Scale Attention Network, SANet) used by the crowd counting method based on scale attention proposed in this invention and the models used by other ten excellent crowd counting algorithms on the ShanghaiTech dataset. Among them, MAE and MSE are the most commonly used evaluation indicators in the field of crowd counting, which respectively represent the mean absolute error and mean square error between the predicted number of people and the actual number of people on the test set. MAE and MSE are used to evaluate the accuracy and robustness of the crowd counting method, respectively. The lower their values, the higher the counting accuracy of the crowd counting method.

[0083] Table 1 Comparison of experimental results of various crowd counting methods on the ShanghaiTech dataset:

[0084]

[0085] The experimental results show that the SANet model proposed in the present invention has the best MAE index performance on the PartA dataset with dense crowds. Compared with the suboptimal ASNet model, its MAE value is reduced by 5.1%, and the MSE index reaches the suboptimal level, slightly better than the MBTTBF model. On the PartB dataset with sparse crowds, the MAE and MSE of the SANet model proposed in the present invention are ahead of other models, and are reduced by 10.0% and 20.9% respectively compared with the suboptimal RANet model. This shows that in scenes with dense and sparse crowds, the method of the present invention can better adapt to changes in head size and improve the accuracy and robustness of crowd counting.

[0086] (2) Visualization examples of the model predicting crowd density and crowd size

[0087] Figure 4 The visualization example of the prediction results of some images in the test set of the ShanghaiTech dataset by the method of the present invention is shown. Each row from left to right is the original image, the true value density map, and the estimated density map. GT in the figure represents the true number of people, and Pre represents the predicted number of people. By comparing the two columns of density maps with the actual number of people, it can be seen that the method of the present invention can accurately predict the total number of people and the spatial distribution of the crowd.

[0088] Example

[0089] This embodiment proposes a crowd counting model based on scale attention (Scale Attention Network, SANet). The crowd counting model mainly consists of a feature extraction module, a scale attention module, a feature fusion module and a density map generation module. The overall structure of the model is as follows: Figure 3 As shown. The overall process of the crowd counting model is as follows:

[0090] First, HRNet-W48 is used as the backbone network to perform preliminary feature extraction on the input image, and feature representations at various resolutions are obtained, while ensuring that the generated high-resolution representations also have rich semantic information. Afterwards, by using the scale attention module, branches at different feature levels can be more focused on predictions within their appropriate scale ranges, thereby making full use of the scale perception capabilities at each resolution scale level. In addition, since the high resolution of the feature map is very important for the crowd counting model, the present invention hopes to accurately predict the crowd density at the resolution layer of the highest scale level. Therefore, it is necessary to further receive good semantics from the low-resolution layer, and this process is achieved through an efficient feature fusion module. Finally, the density map generation module generates predicted density maps for branches at different feature levels; at the same time, during the entire training process of the model, the loss function based on the multi-expert idea will constrain the density map learning process at each level, adaptively select appropriate prediction areas for each resolution layer, and continuously optimize during training, so that the model can eventually adapt to complex and changeable head scale changes.

[0091] The important components of the method of the present invention are described in detail as follows:

[0092] (1) Feature extraction module

[0093] The HRNet (the High-Resolution Network) network architecture was originally designed and proposed by previous researchers in 2019 for the task of human pose estimation. The network maintains high-resolution representations throughout the process and uses low-resolution representations to enhance high-resolution representations in parallel. Specifically, the high-resolution network maintains high-resolution representations by connecting high-resolution convolutions and low-resolution convolutions in parallel and repeatedly performing multi-scale fusion between parallel convolutions. The resulting high-resolution representation is not only strong, but also spatially accurate.

[0094] Figure 5 The network structure of HRNet is shown. Specifically, the network consists of four stages and four parallel sub-networks. The resolution of each stage is gradually halved, and the number of channels of the feature map is correspondingly increased to twice the original. The input image first passes through a stem network, which consists of two 3×3 convolutions with a stride of 2, reducing the resolution to 1 / 4 of the original image size, and then enters the first stage. The first stage contains 4 residual units, each of which is consistent with the settings in ResNet-50 and consists of a bottleneck structure with a width of 64. At the end of the first stage, the number of channels of the feature map is reduced to C through a 3×3 convolution. Starting from the second stage, each subnet will continue to exchange information with other subnets, such as Figure 5As shown by the light blue dotted line in the figure, this structure is defined as an exchange unit. In the exchange unit, the information transfer from low resolution to high resolution is achieved through the upsampling operation after 1×1 convolution; and the conversion from high resolution to low resolution is completed by 3×3 convolution with a step size of 2 in order to reduce the information loss that may occur during the downsampling process. The subsequent two stages are similar. The second, third, and fourth stages contain 1, 4, and 3 exchange units, respectively. Finally, the number of feature map channels output by the four subnets representing the four resolutions is C, 2C, 4C, and 8C, respectively.

[0095] The backbone network of the method of the present invention specifically adopts HRNet-W48, where "W48" means that the number of channels of the highest resolution feature map is 48, that is, C = 48. After the backbone network, the output is four feature representations of different resolutions. The sizes of these feature maps are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image size from high to low, and the corresponding channel numbers are 48, 96, 192 and 384 respectively.

[0096] (2) Scale Attention Module

[0097] The method of the present invention needs to ensure that before generating the density map, the different scale-level features extracted from the backbone network Backbone are first decoupled into features and other features suitable for processing at the current scale level, and they are defined as scale-strongly correlated features and scale-weakly correlated features, respectively, to ensure that they can be further processed independently. This approach can effectively avoid introducing too much noise and help generate high-quality density maps. Therefore, the present invention proposes a novel scale attention module, which adopts a divide-and-conquer strategy, aiming to make full use of the respective advantages of different feature levels, so that a specific feature level is more focused on predictions within its suitable scale range.

[0098] Specifically, the scale attention module consists of a strong correlation feature extraction module, a weak correlation feature extraction module and a scale feature decoupler, and its structure is as follows: Figure 6 As shown. The input image is denoted as I, H and W represent the height and width of the input image respectively. The features of different scale levels extracted by the backbone network can be expressed as Since the backbone network uses HRNet-W48, it will output four levels of features, that is, N = 3. From F0 to F3, the number of channels doubles and the spatial resolution halves successively. The spatial resolution of each layer can be expressed as (h i ,w i )=(H / 2 i+2 ,W / 2 i+2 ). The scale feature decoupler, the strong correlation feature extraction module and the weak correlation feature extraction module are respectively denoted as and These three modules are all learnable, where θ d ,θ s and θ w Represents their respective parameters. The scale feature decoupler is mainly composed of three convolutional layers. The first two are 3×3 convolutions. Each convolution is followed by a BN layer and a ReLU layer. Then, a 1×1 convolution is used to reduce the number of channels to 2, and two attention maps are generated in the channel dimension through the Softmax function. These two attention maps (A s and A w ) will then perform matrix dot multiplication operations with the feature maps processed by the strong correlation feature extraction module and the weak correlation feature extraction module. Both the strong correlation feature extraction module and the weak correlation feature extraction module contain two 3×3 convolutions and corresponding BN and ReLU layers. After being processed by these two branches, the features of a specific level will be decoupled into scale-strong correlation features and scale-weak correlation features (respectively denoted as S i and W i ), these features will be further used to achieve feature fusion.

[0099] (3) Efficient feature fusion module

[0100] Figure 7 The specific structure of the efficient feature fusion module is shown. The entire feature fusion process follows a progressive strategy from low-scale resolution layers to high-scale resolution layers, aiming to enable high-resolution representation to absorb deep information from low-resolution representation. i , the scale-dependent feature S obtained by decoupling i Combined with the feature F of the previous scale resolution layer i-1 The feature fusion is performed, and the fusion result will be used as the output of the resolution layer of the i-1th scale level, and then enter the subsequent density map generation module. At the same time, the decoupled scale weak correlation feature W i Still need to work with S i Passed together to the i-1th scale level resolution layer, and the feature F i-1 Fusion is performed to support feature decoupling of the i-1th scale resolution layer. The rationality of this design is that features that are not suitable for prediction at the current resolution layer may have new prediction potential in a higher resolution layer, so it is necessary to continue to pass them to the previous layer to participate in the subsequent feature decoupling process. The specific formula is as follows:

[0101]

[0102] Out i-1 =EFF{F i-1 , S i}

[0103]

[0104] In the initial stage, is set to F3 as the starting input of the scale attention module, and the generated S i and W i It will enter the feature fusion module together with F2 as its initial input, and then produce two output results: Out2 and It is worth noting that in the highest-scale resolution layer, only weakly correlated feature extraction modules are used, because there is no need to pass features upward to higher levels at this level.

[0105] (4) Density map generation module

[0106] At the resolution layer of each scale level, the fused feature Out after feature fusion will be input into the density map generation module to obtain the predicted density maps at different scale levels.

[0107] The density map generation module is denoted as Its parameters are defined as θ c , whose structure is as follows Figure 8 As shown. During the whole process, the input features will undergo two upsampling operations, expanding the size to four times the previous one, in order to enhance the expressiveness of the feature map, so as to better adapt to the distribution of people at different scales and improve the quality of density map generation. It is worth noting that the upsampler used in the density map generation module will specifically adopt the DySample upsampler, which is a simple and efficient lightweight dynamic upsampler that can generate more refined and data distribution-compliant upsampling results without introducing a high computational burden. After each upsampling, the feature map will pass through two 3×3 convolutional layers in sequence, gradually reducing the number of channels to extract key features. After the second upsampling, a context-aware module is also introduced. This module does not directly fuse multiple channel information, but captures the correlation between different regions in the feature map through multi-scale observation and weighted fusion mechanism, which can enhance the spatial context information of the feature map. This design helps the model to more deeply understand and represent the complex situation of dense and uneven distribution of people at the current resolution level. It is worth noting that since the resolution layer at the highest scale level ultimately aggregates information from each low-resolution layer, the density map generation module Only the highest-scale resolution layer is trained, and other resolution layers have copies of it. The parameter θ c These layers will remain frozen. By using the same parameters, the characteristic patterns of the estimated ground truth Gaussian kernels at each resolution are ensured to be consistent.

[0108] (5) Hybrid loss function based on multi-expert thinking

[0109] In the method of the present invention, in order to constrain the density map learning process of each resolution level, a mixed loss function based on the mixture of experts (MoE) is proposed to adaptively select a suitable prediction area for each resolution layer to ensure the optimization and improvement of the model performance, so as to ultimately enable the model to adapt to the complex and changeable changes in human head scale.

[0110] The core idea of ​​multi-expert is to decompose a complex task into multiple subtasks, each of which is handled by a specific "expert" model. In the method of the present invention, features of different resolution layers have been extracted before, and these different resolution layers can be analogized to different experts. By calculating and comparing the scores of experts from different resolutions on a certain image area, each resolution layer can select its appropriate area to predict the crowd density.

[0111] Specifically, we will define an ideal binary mask (pixel values ​​can only be 0 or 1) M i , which indicates the most suitable area for prediction at the current resolution layer. At the same time, another binary mask is introduced It is related to M i The following relationship exists:

[0112]

[0113] It actually reflects the inheritance of low-resolution representation by high-resolution representation.

[0114] After passing through the density map generation module, the predicted density maps on branches with different resolutions will be output True value density map It is generated by mapping the point annotations by a Gaussian kernel function of a fixed size. Next, the two density maps in different resolution layers are divided into X×Y image patches, where X and Y represent the division in the height h and width w directions respectively. The specific division method is as follows:

[0115]

[0116] Where (ph,p w ) is set to (256,256) by default. Represents the patch size at different resolutions. For a given patch, in order to determine the corresponding optimal resolution, a cost calculation formula is defined, as shown in Formula 4, whose goal is to find which resolution layer has the lowest cost for a given patch, that is, to find the resolution layer that best suits the patch:

[0117]

[0118] Label MoE = arg min(Cost x，y )

[0119]

[0120] Among them, Cost x,y The cost of four resolution layers on X×Y patches is recorded. The first item is the true value density of a certain resolution layer on patch (x, y) With predicted density The first term is the average mean square error (AMSE) between the two patches, and the second term is the instance mean square error (IMSE). AMSE focuses on measuring the overall difference and is still effective when there is no head in the patch; while IMSE emphasizes the ability to predict the head of a crowd. e is a very small number to avoid the impact caused by the patch being full of background. Label MoE The labels of the best resolutions of each of the X×Y patches are recorded, for example, 0 and 3 represent the highest resolution and the lowest resolution, respectively. Finally, the built-in scatter function of Pytorch is used to scatter the Label MoE Convert and generate binary masks M of different resolution layers i , and can be further calculated

[0121] Many existing density map estimation methods generally train the model by calculating the Euclidean distance loss between the true value density map and the predicted density map. The loss function L den It can be expressed as:

[0122] L den =L ED (D pre , D gt )

[0123] Where L ED Indicates the calculation of the Euclidean distance between the two. On this basis, the method of the present invention defines the loss function l for different resolution layers during the training process i , which is used to measure the difference between the regional features that are ideally suitable for processing at the current resolution layer and the regional features that are actually learned by the model, as shown in the following formula:

[0124]

[0125] in It is the output of the density map generation module, that is, the predicted density map D Pre After that, the final loss function can be obtained by further weighting the losses of each layer:

[0126]

[0127] In formula 7, λ i is the loss weight of each resolution layer. Since the loss is halved from high to low resolution, it can be set

[0128] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0129] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A crowd counting method based on scale attention, characterized in that: The crowd counting model used includes a feature extraction module, a scale attention module, a feature fusion module and a density map generation module; the crowd counting method includes the following steps: Step 1: Use the HRNet network as a feature extraction module to extract features from the input image, thereby obtaining features at multi-scale resolution layers, i.e., multi-scale level features; Step 2: Pass the multi-scale level features through the scale attention module and the feature fusion module: from the low scale level to the high scale level, the first fused feature of the current scale level is input into the scale attention module, and the first fused feature is decoupled into a scale-strongly correlated feature whose correlation with the current scale level is higher than the set condition, and a scale-weakly correlated feature whose correlation with the current scale level is weaker than the set condition; then the scale-strongly correlated feature corresponding to the current scale level and the feature under the resolution layer of the previous scale level are input into the feature fusion module for feature fusion, and the second fused feature Out of the previous scale level is obtained. i-1 ; Input the scale-strong correlation features and scale-weak correlation features corresponding to the current scale level, as well as the features under the resolution layer of the previous scale level, into the feature fusion module for feature fusion to obtain the first fusion feature of the previous scale level Finally, the highest-scale resolution layer aggregates and inherits information from each lower-scale resolution layer to adapt to the changes in the scale of the human head in the image; Step 3: At each scale level, the second fused feature is input into the density map generation module to obtain the predicted density map at different scale levels. Step 4: Design a loss function based on the multi-expert idea and train the crowd counting model: Adopt an expert selection strategy based on image blocks to adaptively select and learn different density areas at each resolution layer; Calculate the loss between the predicted density and the true value density in the corresponding areas of different resolution layers, and finally obtain the final loss function of the crowd counting model by weighted summing the losses corresponding to each resolution layer, train the crowd counting model, and obtain a trained crowd counting model; Step 5: In the inference stage, the crowd image to be predicted is preprocessed and input into the trained crowd counting model; the crowd counting model will output the predicted density map at the highest scale resolution layer to predict the number of people.

2. A crowd counting method based on scale attention according to claim 1, characterized in that: The scale attention module includes a strong correlation feature extraction module, a weak correlation feature extraction module and a scale feature decoupler; Multi-scale features are recorded as N is the total number of scale levels of the resolution layer, F i is the feature at the resolution layer of the i-th scale level, from F0 to F N-1 The number of channels doubles and the spatial resolution halves successively. The spatial resolution of the feature at the i-th scale level resolution layer is expressed as (h i ,w i )=(H / 2 i+2 ,W / 2 i+2 ), H and W represent the height and width of the input image respectively; The scale feature decoupler consists of three convolutional layers. The first two convolutional layers are followed by a batch normalization layer and a ReLU activation function layer. The last convolutional layer is used to reduce the number of channels and generate two attention maps in the channel dimension through the Softmax function, which are recorded as the first attention map A and A respectively. s and the second attention map A w ; Both the strong correlation feature extraction module and the weak correlation feature extraction module include two convolutional layers and corresponding batch normalization layers and ReLU activation function layers. The first fusion feature of the i-th scale level After being processed by the strong correlation feature extraction module and the weak correlation feature extraction module respectively, they will be decoupled into a strong correlation feature whose correlation with the i-th scale level is higher than the set condition and a weak correlation feature whose correlation with the specific scale level is weaker than the set condition; The two attention maps are matrix-multiplied with the scale-strong correlation feature and the scale-weak correlation feature to obtain the final scale-strong correlation feature S at the i-th scale level. i and scale-weakly correlated feature W i : in, represents the strong correlation feature extraction module, represents the weak correlation feature extraction module, θ s and θ w are the learnable parameters of the strong correlation feature extraction module and the weak correlation feature extraction module respectively.

3. A crowd counting method based on scale attention according to claim 2, characterized in that: When the feature fusion module performs feature fusion, it specifically includes: The feature F at the i-th scale level resolution layer i Decoupled scale-dependent features S i , and the feature F at the i-1th scale resolution layer i-1 , input to the feature fusion module EFF for feature fusion, and obtain the second fusion feature Out at the i-1th scale level i-1 , input to the density map generation module: Out i-1 =EFF{F i-1 ,S i } The feature F at the i-th scale level resolution layer i Decoupled scale-dependent features S i and scale-weakly correlated feature W i , and the feature F at the i-1th scale resolution layer i-1 Input to the feature fusion module for feature fusion, the first fusion feature of the i-1th scale level Among them, S i +W i Indicates S i and W i Perform element-wise addition.

4. The crowd counting method based on scale attention according to claim 1, characterized in that: Step three specifically includes: The density map generation module includes two DySample upsamplers, a context-aware module, and a convolution operation; The second fusion feature Out i-1 After passing through the DySample upsampler, two convolution operations, the DySample upsampler, the context-aware module and two convolution operations, the predicted density map is obtained; Among them, the context-aware module captures the association between different regions in the feature map through multi-scale observation and weighted fusion mechanism, which can enhance the spatial context information of the feature map; Density map generation module Only the highest scale level resolution layer is trained, and the other scale levels resolution layers have copies, density map generation module The parameter θ c The resolution layers at other scale levels will remain frozen.

5. The crowd counting method based on scale attention according to claim 1, characterized in that: Step 4 specifically includes: Define the binary mask M i , binary mask M i The pixel value can only be 0 or 1, and the binary mask M i Indicates the area that is most suitable for prediction at the current resolution layer; introduces the sum of binary masks With M ii The following relationship exists: After passing through the density map generation module, the predicted density maps on branches with different resolutions will be output True value density map The point annotation is generated by mapping the point annotation by a Gaussian kernel function of a fixed size; the point annotation is a manual marking of the center point of the head on the image; Next, the predicted density maps and true density maps in the resolution layers of different scale levels are divided into X×Y image blocks, where X and Y represent the number of divisions in the height and width directions respectively: p h ,p w All are fixed values ​​256. Represents the height and width of the image block at the resolution layer of the i-th scale level; In order to determine the optimal resolution of a given image block, the cost calculation formula is defined to obtain the most suitable resolution layer for the image block: Label MoE =argmin(Cost x，y ) Among them, Cost x,y (i) Records the cost of different scale-level resolution layers on X×Y image blocks. The first term is the true value density of a certain scale-level resolution layer at (x, y) of the image block. With predicted density The average mean square error between the two, the second term is the instance mean square error, e is a constant; Label MoE It indicates the label that records the best resolution of each of the X×Y image blocks; scatter(·) indicates the scatter function for Label MoE Transform to generate binary masks M of different scale levels of resolution i , and further calculated During the training process, the loss function l is defined for the resolution layers of different scale levels. i , Among them, L ED Indicates the calculation of Euclidean distance, is the output of the density map generation module, i.e., the predicted density map By weighting the loss functions of resolution layers at different scale levels, the final loss function L is obtained: N is the total number of scale levels of the resolution layer, λ i is the weight of the loss function for each resolution layer.

Images