A weakly supervised crowd counting method based on multi-scale dynamic graph convolution

Abstract

A weakly supervised crowd counting method based on a multi-scale dynamic graph convolution network belongs to crowd counting in the fields of public security, city planning and traffic scheduling. Due to the complexity and diversity of traffic scenes, it is very difficult to perform point-level labeling on a large number of crowds, and a large amount of manpower is required. Weakly supervised crowd counting is more suitable for these scenes because they only require counting-level annotations. Existing weakly supervised crowd counting ignores the non-uniformity of cross-distance crowd density distribution and multi-scale crowd head, and cannot obtain similar accurate counting results as the fully supervised crowd counting method. The present application proposes a multi-level regional dynamic graph convolution module to extract the internal relationship between different crowd regions, so as to learn dynamic regional scores and further optimize regional feature representation, and a coarse-grained multi-level feature fusion module is designed to extract multi-scale crowd head information. The present application has high regression accuracy and end-to-end crowd counting capability.

Description

Technical Field

[0001] This invention, based on deep learning technology, studies a weakly supervised method for crowd counting in dense scenes. First, a Transformer network is used to extract deep feature representations from dense crowds. Then, a multi-scale feature fusion module addresses the crowd scale problem in weakly supervised crowd counting tasks, and a feature enhancement module increases the receptive field and enhances the semantic information of the crowd. Finally, a multi-scale dynamic graph convolutional network is used to optimize regional crowd features, establish correlations between similar crowd regions, and use this bias for mutual supervision between regions. This invention belongs to the field of computer vision, specifically involving deep learning, pattern recognition, and other technologies. Background Technology

[0002] Crowd counting is a popular topic in crowd analysis, used to estimate the number of people in an image. It plays a crucial role in social security, early warning of abnormal situations, and traffic control. With population growth and urbanization, transportation scenarios such as subway stations, bus stations, airports, tourist attractions, and docks are becoming increasingly common. At the same time, severe crowd gatherings can lead to congestion and stampedes.

[0003] Based on the content of the labels, existing mainstream crowd counting methods can be divided into two categories: strongly supervised crowd counting and weakly supervised crowd counting. Fully supervised methods require both point-level annotation (annotations of object boxes or heads in the image) and count-level annotation (the number of people in the image). Furthermore, existing fully supervised crowd counting methods can be categorized into object detection, density estimation, point supervision, and bounding box regression methods. In recent years, with the rapid development of deep learning, density estimation-based methods have achieved significant progress. These methods can regress the predicted density map of an image to reflect density variations. However, these methods require accurate point-level annotations to generate density map labels, which is very difficult in complex traffic scenarios. Furthermore, the final counting result is obtained by integrating the density map. However, these methods cannot achieve end-to-end crowd counting.

[0004] The complexity and diversity of traffic scenes make it difficult to accurately locate object coordinates. Weakly supervised methods can address this issue, requiring only count-level annotations for regression counting. However, for weakly supervised crowd counting, the lack of point-level annotations makes it difficult to achieve the same accuracy as fully supervised methods. More useful information should be derived from the image itself. Crowd images contain variations in both region and target scales. Considering multi-scale region information and the internal relationships between similar image regions, cross-distance relationships between different regions can be established. By learning dynamic region counting scores based on multi-level feature maps, relationships between different regions with similar counting scores are established to optimize region features. This overcomes the problem of lacking point-level annotations to some extent. Furthermore, a coarse-grained multi-level feature fusion module is proposed to address the head problem in multi-scale crowds. Summary of the Invention

[0005] The weakly supervised crowd counting task can be viewed as a discrete-valued regression problem. The goal of this task is to regress the number of people in an image, which can be expressed by the following formula:

[0006] Count = F(IM)

[0007] Where F represents the method, IM represents the image, and Count is the predicted crowd count. This method primarily considers two issues: uneven distribution of crowd density across distances and uneven distribution of crowd heads at multiple scales. Due to the lack of point-level annotations, weakly supervised crowd counting does not know the exact location of the crowd and cannot learn density variation information in complex scenes. It was found that region-scale information can reflect certain density information. However, region-scale information is not correlated with location in the image. (See attached image) Figure 1 As shown, the densities of non-adjacent regions can be very similar. In particular, there are many internal connections between other image regions. To better utilize the internal relationships between crowd regions and the internal relationships between region scale information, a multi-layer region dynamic graph convolution module is designed to realize the correlation between similar regions. This module can establish relationships across distances. Furthermore, current weakly supervised crowd counting methods typically only focus on fine-grained feature information in their multi-layer feature fusion modules, neglecting the characteristics of weakly supervised crowd counting. Based on the above ideas, a new weakly supervised method is proposed to achieve end-to-end crowd counting. The overall structure of the model is shown in the attached figure. Figure 2 As shown:

[0008] Swin-Transformer feature extraction network; coarse-grained multi-scale feature fusion module; feature enhancement module; multi-scale dynamic graph convolutional regression module;

[0009] (1) Swin-Transformer Feature Extraction Network

[0010] The Transformer's unique long and short distance attention mechanism is an excellent solution to the lack of point-level annotations and actively focuses on crowd regions in images. It effectively addresses the limited receptive field of convolutional neural networks. This means that the Transformer is more suitable than convolutional neural networks for use as the backbone network in weakly supervised crowd counting tasks. Furthermore, since the Swin-Transformer's window attention mechanism processes image patches, it alleviates the problem of uneven crowd distribution to some extent. Therefore, the Swin-Transformer was chosen as the feature extraction network.

[0011] (2) Coarse-grained multi-scale feature fusion module

[0012] Existing weakly supervised crowd counting methods typically employ multi-level feature fusion modules from general upstream tasks. These upstream tasks usually provide precise coordinate annotations for objects, but weakly supervised crowd counting lacks such labels. Unlike fully supervised methods, weakly supervised crowd counting can regress a number from an image without point-level annotations. However, due to the lack of point-level annotations, the exact location of each object cannot be known, making it impossible to supervise fine-grained, high-resolution, multi-level crowd features. Therefore, coarse-grained features are more suitable for weakly supervised crowd counting. Thus, a coarse-grained multi-level feature fusion module was designed to address the object scale problem in weakly supervised crowd counting. However, inappropriate feature resolution leads to severe crowd head segmentation problems. Therefore, extensive ablation experiments were conducted to determine the size of the fused features. Furthermore, high-level features have sufficient feature representation to target a region in the image. Low-level features have rich detail but lack semantic information. A pyramid structure was constructed to fuse high-level and low-level features.

[0013] (3) Feature enhancement module

[0014] To enhance the semantic information of crowds in the feature maps, this module was added after the coarse-grained multi-level feature fusion module. The feature enhancement module consists of a multi-level expanding convolutional module and a multi-level global pooling module. The multi-level expanding convolutional module is inspired by CCTrans. This module can construct multi-level receptive fields, enhance the receptive field, and detect density changes. Expanded convolution is a commonly used method in computer vision; it can expand the receptive field without introducing additional parameters while capturing contextual information. The multi-level global pooling module uses global pooling operations at different scales to extract crowd region features. Since images contain regions of different scales, a single-resolution feature map cannot satisfy all scenarios. Therefore, it is considered more beneficial to be able to count at different regional scales.

[0015] (4) Multi-scale dynamic graph convolutional regression module

[0016] We discovered strong intrinsic relationships between different crowd regions. Specifically, crowd characteristics should be similar between regions of similar size. This offset can be used to supervise each other between regions to improve counting accuracy. Based on this, a multi-layer dynamic graph convolutional regression module is proposed to utilize this intrinsic relationship information. This module designs a multi-layer dynamic graph convolutional module to establish relationships between different regions. It can learn image region scores based on its own region features. These region scores are used to establish associations between regions with similar scores, and a graph convolutional network is used to optimize the features of the crowd regions. Furthermore, a simple linear regression head is designed to regress the final result.

[0017] The construction of the graph structure is the foundation of GCN, but weakly supervised crowd counting lacks prior information to construct the graph. Instead, it utilizes internal relationships to construct the crowd region map. For the region feature map, the goal is to learn the percentage of counts for each image region within the entire image. First, the features are simplified to a low-dimensional space using Con2d (3×3), resulting in a low-dimensional feature map representation for each region in the image. Then, since the Sigmoid transforms a continuous value into a value between 0 and 1 (which can be viewed as a probability), it is used to map the low-dimensional features to a non-linear score level. Finally, this module obtains a score map using Softmax. The score map is used to calculate relative distances, and KNN is used to obtain other similar crowd regions for each region. Furthermore, since GCN lacks batching parameters, the feature extraction network also has such parameters. Therefore, all batched maps are placed on a sparse graph, solving the problem of unbatched graph convolution. Notably, two layers of stacked graph convolution are designed to optimize region features. Finally, a simple linear regression head is added after the multi-layer dynamic graph convolution module to regress the count results for each image region by stacking linear layers. The count result for the image is obtained by summing the counts of all image regions.

[0018] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:

[0019] 1) A multi-scale dynamic graph convolutional framework (MDGCC) for weakly supervised crowd counting is proposed, which has end-to-end crowd counting capabilities with high regression accuracy. 2) To address the problem of uneven crowd density distribution across distances, a multi-level region dynamic graph convolutional module is proposed. A crowd region map is constructed using the internal relationships between multi-level crowd regions, and then the features of cross-distance crowd regions are optimized using a graph convolutional network. It should be noted that this is the first time a graph convolutional network has been used to solve the weakly supervised crowd counting problem. 3) Weakly supervised crowd counting lacks point-level annotations, cannot know the exact location of each object, and cannot supervise fine-grained, high-resolution, multi-level crowd features. Coarse-grained features are more suitable for weakly supervised crowd counting. Therefore, to address the multi-scale crowd head problem, a coarse-grained multi-level feature fusion module combining global pooling and upsampling operations is proposed, which can extract region-level crowd semantic information. Attached Figure Description

[0020] Figure 1 The image below shows the changes at the regional scale, while the image above shows the changes at the scale of an individual target.

[0021] Figure 2 It refers to the overall structure of the model.

[0022] Based on the above description, the following is a specific implementation process, but the scope of protection of this patent is not limited to this implementation process.

[0023] Step 1: Data Preprocessing

[0024] First, four classic crowd counting databases were used to construct the input sample points: a) the image size was resized to 1152×768 or 768×1152; b) each image was then divided into 6 384×384 image blocks; c) random data augmentation (random horizontal flipping and grayscale transformation) was applied to the dataset.

[0025] Step 2: Model Training

[0026] Step 2.1 Model Parameter Setting

[0027] For the feature extraction backbone network, the Swin-L model was used, and ImageNet-22K pre-trained parameters were used as initial parameters to accelerate model convergence. The training batch size was 30. The learning rate was initially set to 10. -5 It decays to 10 after 100 batches. -6 The total number of training batches is set to 200.

[0028] During the model evaluation phase, MSE and MAE are used as metrics, and the calculation formulas are as follows:

[0029]

[0030]

[0031] Where N is the number of images, P i and G i These represent the predicted value and the actual value, respectively. The Mean Absolute Error (MAE) is the sum of the absolute values ​​of the differences between the target value and the predicted value. It only measures the average error of the predicted values. The Mean Squared Error (MSE) is the most commonly used regression loss function. It is calculated by taking the square of the distance between the predicted value and the target value.

[0032] Step 2.2 Feature Extraction from the Swin-Transformer Backbone Network

[0033] For an image X∈R 384*384*3 (Here, the first and second dimensions represent the length and width of the feature map, and the third dimension represents the number of channels in the feature map.) To input the Swin-Transformer backbone network, the two-dimensional image information first needs to be converted into one-dimensional sequence information that conforms to the backbone network's input. Therefore, the feature extraction backbone network includes image patch partitioning and image patch merging operations, which can solve the input problem. It is worth noting that this feature extraction backbone network contains four stages, each of which reduces the resolution of the input feature map and expands the receptive field of the feature map, similar to a convolutional neural network. Therefore, this step can extract image feature maps of four different scales (X1∈R). 96*96*192 X2∈R 48*48*384 X3∈R 24*24*768 X4∈R 12*12*1536 )

[0034] Step 2.3 Multi-scale feature fusion

[0035] Based on the multi-scale image feature maps output in step 2.2, to address the issue of multi-scale targets in crowd images, the four different scale feature maps output by the backbone network are fused. Specifically, features X1, X2, X3, and X4 are fused so that the final output features contain multi-scale information. 1 / 16 (24*24) of the original image is chosen as the optimal resolution for the feature map output. Therefore, X3 does not need to be resized, while X1 and X2 are reduced to 24*24 using global average pooling. However, X4 is adjusted to 24*24 using bilinear upsampling. Finally, for X1, X2, X3, and X4, a 2D convolution with a 3*3 kernel is used to slide across the feature map to eliminate the negative effects of average pooling and upsampling, and to unify the number of channels in the feature map to 384. Finally, the multi-scale feature maps are added and fused to form the final feature representation Z∈R containing multi-scale information.24*24*384 .

[0036] Step 2.4 Feature Enhancement

[0037] Feature enhancement includes a multi-scale dilated convolution module and a multi-scale global pooling module. The multi-scale dilated convolution module contains four channels, including three dilated convolution channels and one additional regular convolution channel. This module can extract a large receptive field and learn crowd information at different scales in the image through convolutions with different dilation rates. Specifically, the three dilated convolution channels process the feature representation Z output from step 2.3 using dilated convolution layers with dilation rates of 1, 2, and 3, respectively, outputting three feature maps Z1∈R with different receptive field sizes. 24*24*384 Z2∈R 24*24*384 Z3∈R 24*24*384 The additional ordinary convolution channels perform ordinary convolution on the feature map, outputting Z4∈R. 24*24*1152 Finally, Z1, Z2, and Z3 are concatenated along the feature dimension to add information about each pixel in the feature map, resulting in Z123∈R. 24*24*1152 Finally, this feature map is added to Z4 at the pixel level, and the output is M∈R. 24*24*1152 The multi-level global pooling module consists of three channels: pooling size 12×12, pooling size 6×6, and pooling size 3×3. Pooling layers of different scales are used to extract multi-scale region counting features to address the negative problem of crowd segmentation that may occur with single-scale regions. After each pooling layer, a 3×3 2D convolutional kernel slides across the feature map to eliminate the negative effect of average pooling on the feature map and unify the number of channels in the feature map to 64. It should be noted that each convolutional layer is followed by a batch normalization (BN) layer and a ReLU activation function. Finally, the output of step 2.4 is the region features M1∈R at different scales. 12*12*64 M2∈R 6*6*64 M3 ∈R 3*3*64 .

[0038] Step 2.5 Counting Regression Based on Multi-Scale Region Dynamic Graph Convolution

[0039] For multi-scale region feature representation, a 2D convolution with a kernel size of 1*1 is first used to reduce the feature representation to a low-dimensional space, resulting in a ground-dimensional feature map representation of the crowd region. Then, since the Sigmoid function can transform a continuous value into a value between 0 and 1, it is used to map the low-dimensional features to a non-linear score level. A count score for each region is obtained through Softmax. The region count scores are used to calculate relative distances, and KNN is used to obtain similar crowd regions for each region. Specifically, the output of step 2.4 is first subjected to 1D convolution to reduce the feature dimension to M11∈R. 12*12*1 M22∈R 6*6*1 M33∈R 3*3*1 Then, the count score M11 for each region is obtained through the sigmoid activation function and the softmax layer. f ∈R 144 M22 f ∈R 36 M33 f ∈R 9 The relative distance between regions can be calculated by subtracting the count scores of each region; the closer the relative distance, the more similar the features. Specifically, M11, M22, and M33 select the 12, 6, and 3 nearest neighbor regions for each region, respectively, as the nearest neighbor similar population regions. Therefore, the nearest neighbor similar population regions for each region's features can be obtained based on the relative distance, and this relationship of feature similarity between regions can be expressed using an adjacency matrix. The multi-scale region features and adjacency matrix output from step 2.4 are input into a stacked two-layer graph convolutional network (note that the node feature dimension changes from 64 to 64 to 64 during the propagation of the two layers of graph convolution). Finally, the feature Q1∈R output by the graph convolutional network is... 12*12*64 Q2∈R 6*6*64 Q3∈R 3*3*64 The input is fed into a stacked MLP layer, and regression is performed on the count results of each region in the image to obtain Q11∈R. 12*12 Q22∈R 6*6 Q33∈R 3*3 The counts of all regions are summed, and the counts at different scales are averaged to obtain the final count result of the image, P = (Q11 + Q22 + Q33) / 3.

Claims

1. A weakly supervised crowd counting method based on multi-scale dynamic graph convolution, characterized in that, Includes the following steps: Step 1: Data Preprocessing First, four classic crowd counting databases were used to construct the input sample points: a) the image size was resized to 1152×768 or 768×1152; b) each image was then divided into 6 384×384 image blocks; c) random data augmentation was applied to the dataset. Step 2: Model Training Step 2.1 Model Parameter Setting For the feature extraction backbone network, the Swin-L model was adopted, and the pre-trained parameters from ImageNet-22K were used as initial parameters to accelerate the model's convergence speed; the training batch size was 30; and the learning rate was initially set to 10. -5 It decays to 10 after 100 batches. -6 The total number of training batches is set to 200. During the model evaluation phase, MSE and MAE are used as metrics, and the calculation formulas are as follows: Where N is the number of images, P i and G i These represent the predicted value and the actual value, respectively; the mean absolute error (MAE) is the sum of the absolute values ​​of the differences between the target value and the predicted value; it only measures the average value of the error in the predicted value; the mean squared error (MSE) is the most commonly used regression loss function; it is calculated by squaring the distance between the predicted value and the target value. Step 2.2 Feature Extraction from the Swin-Transformer Backbone Network For an image X∈R 384*384*3 The first and second dimensions represent the length and width of the feature map, and the third dimension represents the number of channels in the feature map. To input the Swin-Transformer backbone network, the two-dimensional image information must first be converted into one-dimensional sequence information that conforms to the backbone network's input. Therefore, the feature extraction backbone network includes image patch partitioning and image patch merging operations, which can solve the input problem. It is worth noting that this feature extraction backbone network contains four stages, each of which reduces the resolution of the input feature map and expands the receptive field of the feature map like a convolutional neural network. Therefore, this step can extract image feature maps of four different scales, namely X1∈R. 96*96*192 X2∈R 48*48*384 X3∈R 24 *24*768 X4∈R 12*12*1536 ; Step 2.3 Multi-scale feature fusion Based on the multi-scale image feature maps output in step 2.2, the four different scale feature maps output by the backbone network are fused. Specifically, features X1, X2, X3, and X4 are fused so that the final output features can contain multi-scale information. 1 / 16 of the original image, i.e., 24*24, is selected as the optimal resolution for the feature map output. Therefore, X3 does not need to be resized. X1 and X2 are reduced to 24*24 using global average pooling. However, X4 is adjusted to 24*24 using bilinear upsampling. Finally, for X1, X2, X3, and X4, a 2D convolution with a 3*3 kernel is used to slide across the feature map to eliminate the negative effects of average pooling and upsampling on the feature map, and the number of channels in the feature map is unified to 384. Finally, the multi-scale feature maps are added and fused to form the final feature representation Z∈R containing multi-scale information. 24*24*384 ; Step 2.4 Feature Enhancement Feature enhancement includes a multi-scale dilated convolution module and a multi-scale global pooling module. The multi-scale dilated convolution module has four channels, including three dilated convolution channels and one additional regular convolution channel. This module can extract a large receptive field and learn crowd information at different scales in the image through convolution with different dilation rates. Specifically, the three dilated convolution channels are processed by dilated convolution layers with dilation rates of 1, 2, and 3, respectively, to process the feature representation Z output in step 2.3, outputting three feature maps Z1∈R with different receptive fields. 24*24*384 Z2∈R 24 *24*384 Z3∈R 24*24*384 The additional ordinary convolution channels perform ordinary convolution on the feature map, outputting Z4∈R. 24*24*1152 Finally, Z1, Z2, and Z3 are concatenated along the feature dimension to add information about each pixel in the feature map, resulting in Z123∈R. 24*24*1152 Finally, this feature map is added to Z4 at the pixel level, and the output is M∈R. 24*24*1152 The multi-level global pooling module consists of three channels: pooling size 12×12, pooling size 6×6, and pooling size 3×3. Pooling layers of different scales are used to extract multi-scale region counting features to address the negative problem of crowd segmentation that may occur in single-scale regions. After each pooling layer, a 3*3 2D convolutional kernel slides across the feature map to eliminate the negative effect of average pooling on the feature map and unify the number of channels in the feature map to 64. It should be noted that each convolutional layer is followed by a batch normalization (BN) layer and a ReLU activation function. Finally, the output of step 2.4 is the region features M1∈R at different scales. 12*12*64 M2∈R 6*6*64 M3∈R 3*3*64 ; Step 2.5 Counting Regression Based on Multi-Scale Region Dynamic Graph Convolution For multi-scale region feature representation, firstly, a 2D convolution with a kernel size of 1*1 is used to reduce the feature representation to a low-dimensional space, resulting in a ground-dimensional feature map representation of the crowd region. A Sigmoid function is then used to map the low-dimensional features to a non-linear score level. A count score for each region is obtained through Softmax. The region count scores are used to calculate relative distances, and KNN is used to obtain similar crowd regions for each region. Specifically, firstly, a 1D convolution is used to reduce the feature dimension of the output from step 2.4 to M11∈R. 12*12*1 M22∈R 6*6*1 M33∈R 3*3*1 Then, the count score M11 for each region is obtained through the sigmoid activation function and the softmax layer. f ∈R 144 M22 f ∈R 36 M33 f ∈R 9 The relative distance between regions can be calculated by subtracting the count scores of each region; the closer the relative distance, the more similar the features. Specifically, M11, M22, and M33 take the 12, 6, and 3 regions with the closest relative distance to each region as the nearest neighbor similar population regions, respectively. Therefore, the nearest neighbor similar population regions for each region's features can be obtained based on the relative distance, and this relationship of feature similarity between regions can be expressed using an adjacency matrix. The multi-scale region features and adjacency matrix output from step 2.4 are input into a stacked two-layer graph convolutional network. Finally, the feature Q1∈R output by the graph convolutional network is... 12*12*64 Q2∈R 6*6*64 Q3∈R 3*3*64 The input is fed into a stacked MLP layer, and regression is performed on the count results of each region in the image to obtain Q11∈R. 12*12 Q22∈R 6*6 Q33∈R 3*3 Add up the counts of all regions and average the counts at different scales to obtain the final count result of the image, P = (Q11 + Q22 + Q33) / 3.

Images