A crowd counting method based on scaled attention

Abstract

The present invention relates to the technical field of crowd counting and discloses a crowd counting method based on scale attention, comprising: extracting features from an input image; aggregating multi-scale features at the highest scale resolution layer through a scale attention module and a feature fusion module, inheriting information from each lower resolution layer; inputting the fused features into a density map generation module at each resolution level to obtain predicted density maps at different scale levels; designing a loss function based on multi-expert thinking to train a crowd counting model; pre-processing the crowd image to be predicted and inputting it into the trained crowd counting model, outputting a predicted density map at the highest scale resolution layer to achieve crowd number prediction. The present invention explores the intrinsic connection between different feature levels and different pedestrian head scales to fully utilize and integrate the scale perception capabilities at different feature levels to achieve accurate crowd counting.

Description

Technical Field

[0001] The present invention relates to the technical field of crowd counting, and in particular to a crowd counting method based on scale attention. Background Art

[0002] Crowd counting involves algorithmically analyzing images or video streams captured within a specific area to calculate the total number of pedestrians in that area. While earlier traditional crowd counting methods relied primarily on manual feature extraction, current popular crowd counting methods are primarily based on deep learning, typically employing convolutional neural networks (CNNs) or Transformer architectures. Among deep learning-based crowd counting algorithms, using density maps to calculate the total number of people is a mainstream approach. The overall process of using density maps for crowd counting is as follows: first, a point is marked at each pedestrian location in the image. Then, using a two-dimensional Gaussian kernel function, these discrete marked points are converted into a true pixel-level density map, which serves as a supervisory signal for network training. Each pixel value in the density map reflects the crowd density at that location. By integrating the entire density map, the total number of pedestrians in the image can be determined.

[0003] Given the frequent occurrence of large-scale crowd gatherings, it is essential to count and estimate the number of people in crowded public places. However, the current application of crowd counting is still significantly limited. The problem of inconsistent head size in images has attracted particular attention from most researchers. Because the distance between people and the camera varies, heads at a distance are smaller than those close by. This change, known as scale variation, leads to perspective distortion in the captured images. In addition, differences in camera placement and height result in different shooting angles. Perspective distortion and differences in shooting angles result in significant differences in the scale of objects in different areas of the image. The heads in a single image exhibit multi-scale variations, making crowd counting difficult.

[0004] To address the problem of drastic changes in crowd size in densely populated scenes, current researchers have improved crowd counting algorithms and network structures, mainly exploring and attempting the following:

[0005] (1) Multi-column network structure:

[0006] The earliest CNN-based crowd counting models adopted a single-branch structure and used fixed convolution kernels to extract features. However, due to the fixed receptive field of this fixed-size convolution kernel, it is unable to capture multi-scale features, which inevitably leads to serious counting errors. To address the scale variation problem, some researchers have used multi-column CNN models (such as MCNN networks) or used a hierarchical structure to learn multi-scale representations from different layers of the backbone network. Each column of MCNN consists of convolution layers with different receptive fields. The features learned by multi-column CNN are helpful in dealing with the problem of uneven head sizes in images. In the multi-column CNN model, larger convolution kernels can extract larger scale information, while smaller convolution kernels extract smaller scale information.

[0007] However, these multi-column networks have significant drawbacks: (I) The performance of these methods depends on the number of network branches. While more columns or branches can be used to enrich the receptive field, each additional column or branch significantly increases the number of network parameters and computational complexity, making real-time counting difficult. (II) The size of each column's convolution kernel can only be set empirically, and the selected convolution kernel sizes, such as 3×3 and 5×5, are discrete values ​​that fail to account for scale continuity. This leads to feature redundancy between columns and branches, ultimately generating unclear density maps.

[0008] (2) A single network integrates multi-scale features:

[0009] Given the limitations of multi-column convolutional networks, subsequent researchers have primarily attempted to fuse multi-scale features within a single network, for example using a hierarchical structure to learn multi-scale representations from different layers of the backbone network. However, these methods simply add or concatenate the extracted features at different scales. This fusion approach is not effective for multi-scale scenarios, as features at different scales are likely to have varying degrees of importance within an image and contain noise and redundant information, requiring them to be filtered or assigned different weights during the fusion process. To this end, some subsequent methods have leveraged visual attention mechanisms to fuse multi-scale features.

[0010] Despite the significant progress made by these multi-scale methods, most existing methods aggregate features from multiple levels in a scale-independent manner by utilizing all feature levels to generate the final prediction, ignoring the inherent correspondence between feature levels and head scales. This feature aggregation method may lead to mutual suppression of features at different scales, as accurately computing the same object at all scales is extremely challenging.

[0011] In summary, existing crowd counting methods are still insufficient when dealing with changes in crowd size. Therefore, further optimizing the solution to the problem of crowd size changes is crucial to improving the performance of crowd counting algorithms. Summary of the Invention

[0012] Significant variations in crowd size are the most significant factor limiting the performance of crowd counting algorithms. In different scenarios, whether due to differences in camera position or natural variations in camera angle within the same environment, the distance between the target group and the camera in the image varies, leading to perspective effects in the image. Specifically, pedestrians closer to the camera appear larger and contain more detailed crowd features, while pedestrians farther away appear smaller and contain significantly less information. This imbalance in information makes it extremely difficult to accurately identify groups of different sizes in the same image.

[0013] To address the shortcomings of existing crowd counting methods in coping with changes in crowd scale, the present invention provides a crowd counting method based on scale attention. It improves the structure of traditional multi-branch convolutional neural networks and explores the intrinsic connection between different feature levels and different pedestrian head scales to fully utilize and integrate the scale perception capabilities at different feature levels. It can ultimately adapt to the complex and changeable changes in human head scale in dense crowd scenes, generate high-quality crowd density maps, and achieve accurate crowd counting.

[0014] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0015] A crowd counting method based on scale attention, wherein the crowd counting model adopted includes a feature extraction module, a scale attention module, a feature fusion module, and a density map generation module; the crowd counting method comprises the following steps:

[0016] Step 1: Use the HRNet network as a feature extraction module to extract features from the input image, thereby obtaining features at multiple resolution levels, namely, multi-scale features;

[0017] Step 2: The multi-scale level features are passed through the scale attention module and the feature fusion module: from the low scale level to the high scale level, the first fusion feature of the current scale level is input into the scale attention module, and the first fusion feature is decoupled into a scale-strong correlation feature whose correlation with the current scale level is higher than the set condition, and a scale-weak correlation feature whose correlation with the current scale level is weaker than the set condition; then the scale-strong correlation feature corresponding to the current scale level and the feature under the resolution layer of the previous scale level are input into the feature fusion module for feature fusion to obtain the second fusion feature Out of the previous scale level. i-1 ; Input the scale-strong correlation features and scale-weak correlation features corresponding to the current scale level, as well as the features under the resolution layer of the previous scale level, into the feature fusion module for feature fusion to obtain the first fusion feature of the previous scale level Finally, the highest-scale resolution layer aggregates and inherits information from lower-scale resolution layers to adapt to the changes in head scale in the image.

[0018] Step 3: At each scale level, the second fusion feature is input into the density map generation module to obtain the predicted density map at different scale levels.

[0019] Step 4: Design a loss function based on multi-expert thinking and train the crowd counting model: Adopt an expert selection strategy based on image patches to adaptively select and learn different density regions at each resolution layer. Calculate the loss between the predicted density and the true density in the corresponding regions at different resolution layers. Finally, obtain the final loss function of the crowd counting model by weighted summing the losses corresponding to each resolution layer. Train the crowd counting model to obtain a trained crowd counting model.

[0020] Step 5: In the inference phase, the crowd image to be predicted is preprocessed and then input into the trained crowd counting model; the crowd counting model will output a predicted density map at the highest scale resolution layer to predict the number of people.

[0021] Furthermore, the scale attention module includes a strong correlation feature extraction module, a weak correlation feature extraction module and a scale feature decoupler;

[0022] Multi-scale features are recorded as N is the total number of scale levels of the resolution layer, F i is the feature at the resolution layer of the i-th scale level, from F0 to F N-1 The number of channels is doubled and the spatial resolution is halved successively. The spatial resolution of the feature at the resolution layer of the i-th scale level is expressed as (h i ,w i )=(H / 2 i+2 ,W / 2 i+2 ); H and W represent the height and width of the input image respectively;

[0023] The scale feature decoupler consists of three convolutional layers. The first two convolutional layers are followed by a batch normalization layer and a ReLU activation function layer. The last convolutional layer is used to reduce the number of channels and generate two attention maps in the channel dimension through the Softmax function, which are respectively denoted as the first attention map A and the second attention map B. s and the second attention map A w ;

[0024] The strong correlation feature extraction module and the weak correlation feature extraction module both include two convolutional layers and corresponding batch normalization layers and ReLU activation function layers. The first fusion feature of the i-th scale level After being processed by the strong correlation feature extraction module and the weak correlation feature extraction module respectively, they will be decoupled into a strong correlation feature with a correlation with the i-th scale level higher than the set condition and a scale weak correlation feature with a correlation with the specific scale level weaker than the set condition; the two attention maps are matrix-multiplied with the scale strong correlation feature and the scale weak correlation feature respectively to obtain the final scale strong correlation feature S of the i-th scale level i and scale-weakly correlated features W i :

[0025]

[0026] in, represents the strong correlation feature extraction module, represents the weak correlation feature extraction module, θ s and θ w are the learnable parameters of the strong correlation feature extraction module and the weak correlation feature extraction module respectively.

[0027] Furthermore, when the feature fusion module performs feature fusion, it specifically includes:

[0028] The feature F at the resolution layer of the i-th scale level i The scale-dependent feature S obtained by decoupling i , and the feature F at the i-1 scale level resolution layer i-1 , input to the feature fusion module EFF for feature fusion, and obtain the second fusion feature Out at the i-1 scale level i-1 , input to the density map generation module:

[0029] Out i-1 =EFF{F i-1 , S i}

[0030] The feature F at the resolution layer of the i-th scale level i The scale-dependent feature S obtained by decoupling i and scale-weakly correlated features W i , and the feature F at the i-1 scale level resolution layer i-1 Input to the feature fusion module for feature fusion, the first fusion feature of the i-1 scale level

[0031]

[0032] Among them, S i +W i Indicates S i and W i Perform element-wise addition.

[0033] Furthermore, step three specifically includes:

[0034] The density map generation module includes two DySample upsamplers, a context-aware module, and a convolution operation;

[0035] The second fusion feature Out i-1 After passing through the DySample upsampler, two convolution operations, DySample upsampler, context-aware module and two convolution operations, the predicted density map is obtained;

[0036] Among them, the context-aware module captures the correlation between different regions in the feature map through multi-scale observation and weighted fusion mechanism, which can enhance the spatial context information of the feature map;

[0037] Density map generation module Only the highest scale level resolution layer is trained, and the other scale levels resolution layers have copies, density map generation module The parameter θ c The resolution layers at other scale levels will remain frozen.

[0038] Furthermore, step four specifically includes:

[0039] Define the binary mask M i , binary mask M i The pixel value is only 0 or 1, and the binary mask M i Indicates the area that is most suitable for prediction at the current resolution layer; introduces the sum of binary masks With M ii The following relationship exists:

[0040]

[0041] After passing through the density map generation module, the predicted density maps on different resolution branches are output Ground truth density map The point annotation is generated by mapping the point annotation with a Gaussian kernel function of a fixed size; the point annotation is a manual annotation of the center point of the head in the image; that is, the point annotation is a point annotation, not a box representation in the detection field.

[0042] Next, the predicted density maps and the true density maps in the resolution layers of different scale levels are divided into X×Y image blocks, where X and Y represent the number of divisions in the height and width directions respectively:

[0043]

[0044] p h ,p w The fixed value is 256. Represents the height and width of the image block at the resolution layer of the i-th scale level;

[0045] In order to determine the optimal resolution of a given image block, the cost calculation formula is defined to obtain the most suitable resolution layer for the image block:

[0046]

[0047] Label MoE =arg min(Cost x，y )

[0048]

[0049] Among them, Cost x,y (i) Records the cost of different scale-level resolution layers on X×Y image blocks. The first term is the true value density of the resolution layer at a certain scale level at (x, y) of the image block. With predicted density The average mean square error between the two, the second term is the instance mean square error, e is a constant; Label MoE Indicates the label that records the best resolution of each of the X×Y image blocks; scatter(·) represents the scatter function, which is used for Label MoE Perform the conversion to generate binary masks M of different scale levels of resolution i , and further calculated

[0050] During the training process, the loss function l is defined for the resolution layers at different scale levels. i ,

[0051]

[0052] Among them, L ED Indicates the calculation of Euclidean distance, is the output of the density map generation module, i.e. the predicted density map By weighting the loss functions of resolution layers at different scale levels, the final loss function L is obtained:

[0053]

[0054] N is the total number of scale levels in the resolution layer, λ i is the weight of the loss function at each resolution layer.

[0055] Compared with the prior art, the beneficial technical effects of the present invention are:

[0056] (1) The continuity of the change of human head size is taken into account.

[0057] Compared with the earliest multi-column CNN models (such as MCNN networks), the present invention does not use multi-column and fixed-size convolution kernels to extract multi-scale features. Instead, it adopts HRNet-W48 as the backbone network to progressively obtain multi-scale features from multiple resolution layers, fully considering the continuity of scale changes, which helps to further refine the extracted multi-scale features.

[0058] (2) The quality of the generated crowd density map is improved to achieve more accurate crowd counting.

[0059] Compared to other networks that use hierarchical structures to learn multi-scale representations, this method explores the inherent connection between different feature levels and different pedestrian head scales. By decoupling the features of each resolution layer, specific feature levels are more focused on predictions within their appropriate scale range. Furthermore, this method fully utilizes and integrates the scale-aware capabilities of different feature levels, ultimately adapting to the varying head scales in dense crowd scenes and achieving accurate crowd counting.

[0060] (3) The training process of the crowd counting model was optimized, and the robustness and generalization ability of the model were enhanced.

[0061] Compared with many crowd counting models that use a single loss function, this method introduces a hybrid loss function based on multi-expert thinking to constrain the density map learning process at each level, adaptively select appropriate prediction areas for each resolution layer, and continuously optimize during training to ensure improved model performance to adapt to complex and changeable head scale changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a flow chart of the method of the present invention;

[0063] Figure 2 It is a schematic diagram of the present invention;

[0064] Figure 3 is a structural diagram of the crowd counting model of the present invention;

[0065] Figure 4 This is a sample visualization of the prediction results of some images on the ShanghaiTech dataset using the method of the present invention;

[0066] Figure 5 Schematic diagram of the structure of a feature extraction module according to an embodiment of the present invention;

[0067] Figure 6 Schematic diagram of the structure of the scale attention module according to an embodiment of the present invention;

[0068] Figure 7 Schematic diagram of the structure of a feature fusion module according to an embodiment of the present invention;

[0069] Figure 8 Schematic diagram of the structure of a density map generation module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0071] Figure 1 The method flow of the present invention is shown. Specifically, it may include the following five steps:

[0072] Step 1: Use the HRNet-W48 network as the feature extraction module (backbone network) to extract features from the input image, thereby obtaining different features of multiple resolution layers, namely multi-scale level features.

[0073] Step 2: For the multi-scale features previously extracted by the backbone network, these features are sequentially passed through the scale attention module and the feature fusion module. Specifically, at each resolution layer from low to high scale levels, the extracted features are first decoupled into scale-strongly correlated features and scale-weakly correlated features, allowing them to be processed independently. Subsequently, the high-resolution feature representations continuously inherit useful feature representations from low-resolution ones, achieving further feature fusion. Finally, at the highest-scale resolution layer, the effective information from each low-resolution layer is aggregated and inherited to better adapt to the drastic changes in human head scale.

[0074] Step 3: At each scale level, one of the output branches after feature fusion is input into the density map generation module to obtain the predicted density maps at different scale levels.

[0075] Step 4: Design a loss function based on multi-expert thinking, train the crowd counting model, and obtain a trained crowd counting model.

[0076] Specifically, during the training phase, the crowd counting model adopts an expert selection strategy based on image blocks to adaptively select and learn different density areas at each scale level of resolution layer; in the corresponding areas of the resolution layers at different scale levels, the loss between the predicted density and the true value density is calculated. Finally, by adding the losses of each layer according to a certain weight, the final loss function of the crowd counting model is obtained, thereby imposing effective constraints on the crowd counting model.

[0077] Step 5: Prepare the crowd image to be predicted and, after certain preprocessing operations, input it into the trained crowd counting model. During the inference phase, the crowd counting model will output a high-quality predicted density map at the highest resolution level and accurately predict the number of people.

[0078] like Figure 3 As shown in Figure 2, the overall process of the crowd counting model is as follows:

[0079] The crowd counting model proposed in the present invention is mainly composed of a feature extraction module, a scale attention module, a feature fusion module and a density map generation module. Specifically, the crowd counting model uses the HRNet-W48 network as a feature extraction module to process the input image to obtain multi-scale level features. The multi-scale level features are then further processed by the scale attention module and the feature fusion module, and are sent to the density map generation module to obtain crowd density maps of different sizes. During the training phase, the crowd counting model adopts an expert selection strategy based on image blocks to adaptively select and learn different density areas at the resolution layers of each scale level, and imposes effective constraints on the crowd counting model by weighting the losses of each resolution layer. Figure 3 As shown in the figure, the dark gray areas in the density map indicate that these image areas will not be used in the loss function calculation. The remaining unobstructed areas reflect the best prediction areas at each resolution level. Ultimately, during the inference phase, the model will accurately predict crowd density at the highest resolution level.

[0080] The present invention conducted relevant experiments on the classic crowd counting dataset, the ShanghaiTech dataset. The dataset consists of two parts, Part A and Part B. The crowd size in Part A varies dramatically, while the crowd distribution in Part B is relatively sparse. The relevant experimental results are shown in Table 1. Figure 4 shown.

[0081] (1) Comparison of commonly used evaluation indicators for crowd counting algorithms

[0082] Table 1 shows the experimental results of the crowd counting model (Scale Attention Network, SANet) used in the scale-attention-based crowd counting method proposed in this paper and the models used by ten other excellent crowd counting algorithms on the ShanghaiTech dataset. Among them, MAE and MSE are the most commonly used evaluation metrics in the field of crowd counting. They represent the mean absolute error and mean square error between the predicted number of people and the actual number of people on the test set, respectively. MAE and MSE are used to evaluate the accuracy and robustness of crowd counting methods, respectively. The lower their values, the higher the counting accuracy of the crowd counting method.

[0083] Table 1 Comparison of experimental results of various crowd counting methods on the ShanghaiTech dataset:

[0084]

[0085] Experimental results show that the SANet model proposed in this invention performs best in the MAE indicator on the densely populated PartA dataset. Compared with the suboptimal ASNet model, its MAE value is reduced by 5.1%, and the MSE indicator reaches the suboptimal level, slightly better than the MBTTBF model. On the PartB dataset with a sparser population, the MAE and MSE of the SANet model proposed in this invention are both ahead of other models, and are reduced by 10.0% and 20.9% respectively compared with the suboptimal RANet model. This shows that in both dense and sparse crowd distribution scenarios, the method of this invention can better adapt to changes in head size and improve the accuracy and robustness of crowd counting.

[0086] (2) Visualization examples of the model predicting crowd density and crowd size

[0087] Figure 4 This visualization shows a sample prediction result for a portion of images from the ShanghaiTech dataset test set using our method. Each row, from left to right, contains the original image, the ground-truth density map, and the estimated density map. GT represents the true number of people, and Pre represents the predicted number of people. Comparing the two density maps with the actual number of people shows that our method can accurately predict both the total number of people and their spatial distribution.

[0088] Example

[0089] This embodiment proposes a crowd counting model based on scale attention network (SANet). The crowd counting model mainly consists of feature extraction module, scale attention module, feature fusion module and density map generation module. The overall structure of the model is as follows: Figure 3 As shown. The overall process of the crowd counting model is as follows:

[0090] First, HRNet-W48 is used as the backbone network to perform preliminary feature extraction on the input image, generating feature representations at multiple resolutions while ensuring that the generated high-resolution representations also contain rich semantic information. Subsequently, a scale-attention module is employed to enable branches at different feature levels to focus on predictions within their appropriate scale ranges, thereby fully leveraging the scale-aware capabilities of each resolution level. Furthermore, since high resolution of feature maps is crucial for crowd counting models, this paper aims to accurately predict crowd density at the highest resolution level. Therefore, it is necessary to further incorporate good semantic information from lower-resolution layers, a process achieved through an efficient feature fusion module. Finally, a density map generation module generates predicted density maps for branches at different feature levels. Furthermore, throughout the model training process, a multi-expert loss function constrains the density map learning process at each level, adaptively selecting appropriate prediction regions for each resolution level and continuously optimizing the model during training. This ultimately enables the model to adapt to complex and variable head scales.

[0091] The important components of the method of the present invention are described in detail as follows:

[0092] (1) Feature extraction module

[0093] The High-Resolution Network (HRNet) architecture was originally designed and proposed by researchers in 2019 specifically for human pose estimation. This network maintains high-resolution representations throughout the process and augments them with low-resolution representations in parallel. Specifically, the high-resolution network maintains high-resolution representations by connecting high-resolution and low-resolution convolutions in parallel and repeatedly performing multi-scale fusion between the parallel convolutions. The resulting high-resolution representations are not only strong but also spatially accurate.

[0094] Figure 5 The network structure of HRNet is shown. Specifically, the network consists of four stages and four parallel sub-networks. The resolution of each stage is gradually halved, and the number of channels of the feature map is correspondingly doubled. The input image first passes through a stem network, which consists of two 3×3 convolutions with a stride of 2, reducing the resolution to 1 / 4 of the original image size, and then enters the first stage. The first stage contains 4 residual units, each of which is consistent with the settings in ResNet-50 and consists of a bottleneck structure with a width of 64. At the end of the first stage, the number of channels of the feature map is reduced to C through a 3×3 convolution. Starting from the second stage, each subnet will continue to exchange information with other subnets, such as Figure 5As shown by the light blue dashed line, this structure is defined as a swap unit. Within this swap unit, information transfer from low to high resolution is achieved through upsampling after a 1×1 convolution. To reduce potential information loss during downsampling, the transition from high to low resolution is accomplished using a 3×3 convolution with a stride of 2. The subsequent two stages operate similarly. The second, third, and fourth stages contain 1, 4, and 3 swap units, respectively. Ultimately, the feature maps output by the four subnetworks representing the four resolutions have C, 2C, 4C, and 8C channels, respectively.

[0095] The backbone network of the present invention adopts HRNet-W48, where "W48" indicates that the highest-resolution feature map has 48 channels, i.e., C = 48. This backbone network outputs feature representations at four different resolutions. The sizes of these feature maps are, from high to low, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size, corresponding to 48, 96, 192, and 384 channels, respectively.

[0096] (2) Scale Attention Module

[0097] The method of the present invention needs to ensure that before generating the density map, the different scale-level features extracted from the backbone network Backbone are first decoupled into features and other features suitable for processing at the current scale level, and defined as scale-strongly correlated features and scale-weakly correlated features, respectively, to ensure that they can be further processed independently. This approach can effectively avoid the introduction of excessive noise and help generate high-quality density maps. Therefore, the present invention proposes a novel scale attention module that adopts a divide-and-conquer strategy, aiming to make full use of the respective advantages of different feature levels, so that specific feature levels are more focused on predictions within the scale range to which they are suitable.

[0098] Specifically, the scale attention module consists of a strong correlation feature extraction module, a weak correlation feature extraction module and a scale feature decoupler. Its structure is as follows: Figure 6 As shown. The input image is denoted as I, H and W represent the height and width of the input image respectively. The features of different scale levels extracted by the backbone network can be expressed as Since the backbone network adopts HRNet-W48, it will output four levels of features, that is, N = 3. From F0 to F3, the number of channels doubles and the spatial resolution is halved. The spatial resolution of each layer can be expressed as (h i ,w i )=(H / 2 i+2 ,W / 2 i+2 ). The scale feature decoupler, strong correlation feature extraction module and weak correlation feature extraction module are respectively denoted as and These three modules are all learnable, where θ d ,θ s and θ w Represents their respective parameters. The scale feature decoupler mainly consists of three convolutional layers. The first two are 3×3 convolutions. Each convolution is followed by a BN layer and a ReLU layer. Then, a 1×1 convolution is used to reduce the number of channels to 2, and two attention maps are generated in the channel dimension through the Softmax function. These two attention maps (respectively A s and A w denoted by ) will then perform matrix dot multiplication operations with the feature maps processed by the strong correlation feature extraction module and the weak correlation feature extraction module respectively. The strong correlation feature extraction module and the weak correlation feature extraction module both contain two 3×3 convolutions and corresponding BN and ReLU layers. After being processed by these two branches, the features at a specific level will be decoupled into scale-strong correlation features and scale-weak correlation features (respectively denoted as S i and W i ), these features will be further used to achieve feature fusion.

[0099] (3) Efficient feature fusion module

[0100] Figure 7 The detailed structure of the efficient feature fusion module is shown. The entire feature fusion process follows a progressive strategy from low-scale resolution layers to high-scale resolution layers, aiming to enable high-resolution representation to absorb deep information from low-resolution representation. i , the scale-dependent feature S obtained by decoupling i Combined with the feature F of the previous scale resolution layer i-1 The feature fusion is performed, and the fusion result will be used as the output of the resolution layer of the i-1 scale level, and then enter the subsequent density map generation module. At the same time, the decoupled scale weak correlation feature W i Still need to work with S i Passed together to the i-1th scale level resolution layer, and the feature F of this layer i-1 Fusion is performed to support feature decoupling at the i-1th resolution level. The rationale for this design is that features that are not suitable for prediction at the current resolution level may have new prediction potential at higher resolution levels, so it is necessary to pass them on to the previous layer to participate in the subsequent feature decoupling process. The specific formula is as follows:

[0101]

[0102] Out i-1 =EFF{F i-1 , S i}

[0103]

[0104] In the initial stage, is set to F3 as the starting input of the scale attention module, and the generated S i and W i It will enter the feature fusion module together with F2 as its initial input, and then produce two output results: Out2 and It is worth noting that in the highest-scale resolution layer, only weakly correlated feature extraction modules are used, because there is no need to pass features upward to higher levels at this level.

[0105] (4) Density map generation module

[0106] At the resolution layer of each scale level, the fused feature Out after feature fusion will be input into the density map generation module to obtain the predicted density maps at different scale levels.

[0107] The density map generation module is denoted as Its parameters are defined as θ c , its structure is as follows Figure 8 As shown. During the whole process, the input features will undergo two upsampling operations, expanding the size to four times the previous size, in order to enhance the expressive power of the feature map, so as to better adapt to the distribution of people at different scale levels and improve the generation quality of the density map. It is worth noting that the upsampler used in the density map generation module will specifically adopt the DySample upsampler, which is a simple, efficient and lightweight dynamic upsampler that can generate more refined upsampling results that conform to the data distribution without introducing a high computational burden. After each upsampling, the feature map will pass through two 3×3 convolutional layers in sequence, gradually reducing the number of channels to extract key features. After the second upsampling, a context-aware module is also introduced. This module does not directly fuse multiple channel information, but captures the correlation between different regions in the feature map through multi-scale observation and weighted fusion mechanism, which can enhance the spatial context information of the feature map. This design helps the model to more deeply understand and represent the complex situation of dense and uneven distribution of people at the current resolution level. It is worth noting that since the resolution layer of the highest scale level ultimately aggregates information from each low-resolution layer, the density map generation module Only the highest-scale resolution layer is trained, and other resolution layers have copies of it. The parameter θ c These layers will remain frozen. By using the same parameters, the characteristic patterns of the estimated ground truth Gaussian kernels at each resolution are ensured to be consistent.

[0108] (5) Hybrid loss function based on multi-expert thinking

[0109] In order to constrain the density map learning process at each resolution level, the method of the present invention also proposes a mixed loss function based on the Mixture of Experts (MoE) idea. It adaptively selects the appropriate prediction area for each resolution layer to ensure the optimization and improvement of model performance, thereby ultimately enabling the model to adapt to complex and changeable changes in human head scale.

[0110] The core concept of multi-expert is to decompose complex tasks into multiple subtasks, each handled by a specific "expert" model. Because the method previously extracted features at different resolution levels, these different resolution levels can be analogized to different experts. By calculating and comparing the scores of experts from different resolutions on a given image region, each resolution level can select an appropriate area for crowd density prediction.

[0111] Specifically, an ideal binary mask (pixel values ​​are only 0 or 1) M will be defined i , which indicates the most suitable area for prediction at the current resolution layer. At the same time, another binary mask is introduced It is related to M i The following relationship exists:

[0112]

[0113] It actually reflects the inheritance of low-resolution representation by high-resolution representation.

[0114] After passing through the density map generation module, the predicted density maps on different resolution branches are output Ground truth density map The point annotations are mapped by a Gaussian kernel function of a fixed size. Next, the two density maps in different resolution layers are divided into X×Y image patches, where X and Y represent the division in the height h and width w directions respectively. The specific division method is as follows:

[0115]

[0116] where (ph,p w ) is set to (256,256) by default, Represents the patch size at different resolutions. For a given patch, in order to determine the corresponding optimal resolution, a cost calculation formula is defined, as shown in Formula 4. Its goal is to find which resolution layer has the lowest cost for a given patch, that is, to find the resolution layer that best suits the patch:

[0117]

[0118] Label MoE =arg min(Cost x，y )

[0119]

[0120] Among them, Cost x,y The cost of the four resolution layers on the X×Y patches is recorded. The first item is the true value density of a certain resolution layer on the patch (x, y) With predicted density The first is the average mean square error (AMSE) between the two patches, and the second is the instance mean square error (IMSE). AMSE focuses on measuring the overall difference and is still effective when there is no human head in the patch; while IMSE emphasizes the ability to predict the heads of the crowd. e is a very small number to avoid the impact caused by the background in the patch. MoE The labels of the best resolutions of each of the X×Y patches are recorded, for example, 0 and 3 represent the highest resolution and the lowest resolution respectively. Finally, the built-in scatter function of Pytorch is used to scatter the labels. MoE Convert and generate binary masks M of different resolution layers i , and can be further calculated

[0121] Many existing density map estimation methods generally train the model by calculating the Euclidean distance loss between the true value density map and the predicted density map. The loss function L den It can be expressed as:

[0122] L den =L ED (D pre , D gt )

[0123] Among them L ED Indicates the calculation of the Euclidean distance between the two. On this basis, the method of the present invention defines the loss function l for different resolution layers during the training process. i , which is used to measure the difference between the regional features that are ideally suitable for processing at the current resolution layer and the regional features that the model actually learns, as shown in the following formula:

[0124]

[0125] in It is the output of the density map generation module, that is, the predicted density map D Pre After that, we can further weight the losses of each layer to get the final loss function:

[0126]

[0127] In formula 7, λ i is the loss weight of each resolution layer. Since the resolution decreases by half from high to low, it can be set

[0128] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

[0129] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A crowd counting method based on scale attention, characterized in that: The crowd counting model used includes a feature extraction module, a scale attention module, a feature fusion module, and a density map generation module; the crowd counting method includes the following steps: Step 1: Use the HRNet network as a feature extraction module to extract features from the input image, thereby obtaining features at multiple resolution levels, namely multi-scale features; Step 2: The multi-scale level features are passed through the scale attention module and the feature fusion module: from the low scale level to the high scale level, the first fusion feature of the current scale level is input into the scale attention module, and the first fusion feature is decoupled into a scale-strong correlation feature whose correlation with the current scale level is higher than the set condition, and a scale-weak correlation feature whose correlation with the current scale level is weaker than the set condition; then the scale-strong correlation feature corresponding to the current scale level and the feature under the resolution layer of the previous scale level are input into the feature fusion module for feature fusion to obtain the second fusion feature of the previous scale level. ; Input the scale-strong correlation features and scale-weak correlation features corresponding to the current scale level, as well as the features under the resolution layer of the previous scale level, into the feature fusion module for feature fusion to obtain the first fusion feature of the previous scale level Finally, the highest-level resolution layer aggregates and inherits information from lower-level resolution layers to adapt to the changes in head scale in the image. Step 3: At each scale level, the second fusion feature is input into the density map generation module to obtain the predicted density map at different scale levels. ; Step 4: Design a loss function based on multi-expert thinking and train the crowd counting model: Adopt an expert selection strategy based on image patches to adaptively select and learn different density regions at each resolution layer. Calculate the loss between the predicted density and the true density in the corresponding regions at different resolution layers. Finally, obtain the final loss function of the crowd counting model by weighted summing the losses corresponding to each resolution layer. Train the crowd counting model to obtain a trained crowd counting model. Step 5: In the inference phase, the crowd image to be predicted is preprocessed and then input into the trained crowd counting model; the crowd counting model will output a predicted density map at the highest scale resolution layer to predict the number of people.

2. A crowd counting method based on scale attention according to claim 1, characterized in that: The scale attention module includes a strong correlation feature extraction module, a weak correlation feature extraction module and a scale feature decoupler; Multi-scale features are recorded as , N is the total number of scale levels of the resolution layer, is the feature at the resolution layer of the i-th scale level, from arrive The number of channels is doubled and the spatial resolution is halved successively. The spatial resolution of the feature at the resolution layer of the i-th scale level is expressed as , and Respectively represent the height and width of the input image; The scale feature decoupler consists of three convolutional layers. The first two convolutional layers are followed by a batch normalization layer and a ReLU activation function layer. The last convolutional layer is used to reduce the number of channels and generate two attention maps in the channel dimension through the Softmax function, which are recorded as the first attention map and the second attention map respectively. and the second attention map ; The strong correlation feature extraction module and the weak correlation feature extraction module both include two convolutional layers and corresponding batch normalization layers and ReLU activation function layers. The first fusion feature of the i-th scale level After being processed by the strong correlation feature extraction module and the weak correlation feature extraction module respectively, they will be decoupled into strong correlation features whose correlation with the i-th scale level is higher than the set condition and weak correlation features whose correlation with a specific scale level is weaker than the set condition; The two attention maps are respectively subjected to matrix dot multiplication with the scale-strong correlation feature and the scale-weak correlation feature to obtain the final scale-strong correlation feature of the i-th scale level. Weakly correlated with scale : in, represents the strong correlation feature extraction module, represents the weak correlation feature extraction module, and are the learnable parameters of the strong correlation feature extraction module and the weak correlation feature extraction module respectively.

3. The crowd counting method based on scale attention according to claim 2, characterized in that: When the feature fusion module performs feature fusion, it specifically includes: The features of the resolution layer at the i-th scale level Decoupled scale-dependent features , and the features at the i-1th scale level resolution layer , input to the feature fusion module EFF for feature fusion, and obtain the second fusion feature of the i-1th scale level , input to the density map generation module: The features of the resolution layer at the i-th scale level Decoupled scale-dependent features Weakly correlated with scale , and the features at the i-1th scale level resolution layer Input to the feature fusion module for feature fusion, the first fusion feature of the i-1 scale level : in, Express and Perform element-wise addition.

4. The crowd counting method based on scale attention according to claim 1, characterized in that: Step three specifically includes: The density map generation module includes two DySample upsamplers, a context-aware module, and a convolution operation; Second fusion feature After passing through the DySample upsampler, two convolution operations, DySample upsampler, context-aware module and two convolution operations, the predicted density map is obtained; Among them, the context-aware module captures the correlation between different regions in the feature map through multi-scale observation and weighted fusion mechanism, which can enhance the spatial context information of the feature map; Density map generation module Only the highest scale level resolution layer is trained, and the resolution layers of other scale levels have copies, density map generation module Parameters The resolution layers at other scale levels will remain frozen.

5. The crowd counting method based on scale attention according to claim 1, characterized in that: Step 4 specifically includes: Define a binary mask , binary mask The pixel value is only 0 or 1, binary mask Indicates the area that is most suitable for prediction at the current resolution layer; introduces the sum of binary masks , and The following relationship exists: After passing through the density map generation module, the predicted density maps on different resolution branches are output , the true value density map The point annotation is generated by mapping the point annotation by a Gaussian kernel function of a fixed size; the point annotation is a manual markup of the center point of the head in the image; Next, the predicted density maps and the true density maps in the resolution layers of different scale levels are divided into X×Y image blocks, where X and Y represent the number of divisions in the height and width directions respectively: The fixed value is 256. Represents the height and width of the image block at the resolution layer of the i-th scale level; Represents the height of the feature at the i-th scale resolution layer, Represents the width of the feature at the resolution layer of the i-th scale level; and Respectively represent the height and width of the input image; In order to determine the optimal resolution of a given image block, the cost calculation formula is defined to obtain the most suitable resolution layer for the image block: in, The cost of different scale-level resolution layers on X×Y image blocks is recorded. The first item is the true value density of the resolution layer at a certain scale level at (x, y) of the image block. With predicted density The second term is the instance mean square error, is a constant; Indicates the label that records the best resolution of each of the X×Y image blocks; Represents the scatter function, Perform conversion to generate binary masks of different scale levels of resolution , and further calculated ; Define loss functions for different scale levels of resolution during training , in, Indicates calculating the Euclidean distance, is the output of the density map generation module, i.e. the predicted density map ; By weighting the loss functions of resolution layers at different scale levels, the final loss function L is obtained: N is the total number of scale levels of the resolution layer, is the weight of the loss function at each resolution layer.

Images