A crowd density estimation method and system based on content-aware module

Abstract

The present invention provides a crowd density estimation method and system based on a content-aware module, relating to the technical field of crowd density estimation. The method comprises: preprocessing image data to be predicted; extracting deep semantic information and multi-layer fused context information based on the processed image data; dynamically generating content-aware parameters based on the deep semantic information based on the content-aware module; initializing the weights and offsets of the convolutional layer of the crowd density estimation module based on the content-aware parameters; and inputting the multi-layer fused context information into the initialized crowd density estimation module to obtain a crowd density image. Each time the present invention predicts crowd density, it obtains the deep semantic information of the image data and dynamically generates parameters for the convolutional layer of the crowd density estimation module, enabling more flexible and targeted crowd density estimation and achieving better prediction results in complex and changing scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of crowd density estimation, and in particular to a crowd density estimation method and system based on a content perception module. Background Art

[0002] In recent years, the human population has increased significantly, and crowd density in public places has increased significantly. Stampedes caused by overcrowding have occurred worldwide, resulting in significant loss of life and property. With the rapid development of computer vision technology, crowd counting based on surveillance images and videos has become crucial for security and traffic control. However, the ever-changing environment, camera angles, and occlusion caused by dense crowds pose significant challenges to this task.

[0003] With the development of deep learning and convolutional neural networks (CNNs), researchers have begun using CNNs to accurately estimate crowd sizes from images and videos. Currently, most advanced CNN methods use pre-trained models (VGG, ResNet) combined with complex functional modules (attention modules) to predict crowd density maps from input images. Other methods have designed multi-column architectures (MCNN) and multi-task architectures (PCCNET) to improve task accuracy. However, current networks used in the crowd prediction field typically use fixed parameters, making them inflexible and difficult to handle. Crowd images are subject to varying viewing angles, irregular crowd distribution, perspective, and occlusion. Summary of the Invention

[0004] To address the above problems, the present invention provides a crowd density estimation method and system based on a content-aware module, designs a crowd density estimation network based on the content-aware module, and dynamically generates parameters based on image semantic information through the content-aware module and loads them into the model, so that the model can obtain better prediction results in complex and changing scenarios.

[0005] To achieve the above object, the present invention provides a crowd density estimation method based on a content-aware module, comprising:

[0006] Preprocess the image data to be predicted;

[0007] Extracting deep semantic information and multi-layer fusion context information based on the processed image data;

[0008] Dynamically generating content-aware parameters based on the deep semantic information based on a content-aware module;

[0009] Initializing the weights and offsets of the convolutional layer of the crowd density estimation module according to the content-aware parameters;

[0010] The multi-layer fusion context information is input into the initialized crowd density estimation module to obtain a crowd density image.

[0011] As a further improvement of the present invention, the processed data is input into a convolutional neural network model;

[0012] The convolutional neural network model includes a feature extraction module, a content perception module and a crowd density estimation module;

[0013] The feature extraction module extracts deep semantic information and multi-layer fusion context information based on the processed image data.

[0014] As a further improvement of the present invention, the feature extraction module includes a backbone network and a feature fusion network;

[0015] The backbone network includes one convolutional layer and four residual layers, and performs a series of convolution, down-collection, and up-collection on the processed image data to obtain the deep semantic information;

[0016] The feature fusion network adopts a feature pyramid network, and fuses the output features of one convolutional layer and four residual layers in the backbone network through feature splicing to obtain the feature fusion context information.

[0017] As a further improvement of the present invention, the training process of the convolutional neural network model includes:

[0018] Perform data preprocessing, data enhancement and multi-scale transformation on training image data;

[0019] Initialize the parameters of the backbone network, feature fusion network and content perception module;

[0020] The transformed image data is fed into the convolutional neural network model, and the deep semantic information and multi-layer fusion context information are obtained through the feature extraction module;

[0021] The content perception module converts the deep semantic information into weights and offsets of the convolution layer, and loads them into the convolution layer of the crowd density estimation module;

[0022] The multi-layer fusion context information is fed into the crowd density estimation module to obtain the crowd density image;

[0023] The crowd density image is trained through the mean square error function to obtain the parameters of the backbone network, feature fusion network and content perception module.

[0024] As a further improvement of the present invention, the multi-scale transformation method is:

[0025] Generate the ground-truth crowd density map of the image using adaptive Gaussian kernel;

[0026] Perform bilinear interpolation scaling on the image and the ground-truth crowd density map every 5 epochs;

[0027] For the crowd density image obtained by estimating the scaled image, the pixel value transformation process is finally performed using the formula:

[0028]

[0029] in,

[0030] Ratio is the transformation coefficient;

[0031] Represents the sum of pixel values ​​of the crowd density image before bilinear interpolation scaling;

[0032] Represents the sum of pixel values ​​of the crowd density image after bilinear interpolation scaling;

[0033] Represents the pixel value of the final crowd density image.

[0034] As a further improvement of the present invention, the data enhancement processing includes random cropping, random translation, random horizontal flipping, random brightness and random erasure of the image data.

[0035] As a further improvement of the present invention, the content perception module includes three layers of 1×1 convolution and four groups of parameter generation modules, each group of the parameter generation modules includes a 3×3 convolution and an average pooling layer;

[0036] The deep semantic information is processed through three layers of 1×1 convolution and four sets of parameter generation modules to obtain four sets of content-aware parameters.

[0037] As a further improvement of the present invention, the crowd density prediction module includes four groups of 1×1 convolutions;

[0038] The weights and offsets of the four groups of 1×1 convolutions of the crowd density prediction module are initialized respectively according to the four groups of content-aware parameters.

[0039] As a further improvement of the present invention, the four groups of 1×1 convolutions of the crowd density prediction module have channel numbers of 512, 256, 128, and 2 respectively;

[0040] The four groups of 1×1 convolutions sequentially perform image channel dimensionality reduction on the multi-layer fusion context information to obtain a crowd density feature map with a channel number of 2;

[0041] Performing heat map analysis and overall integration on the crowd density feature map to obtain the crowd density image and the total number of people in the image.

[0042] The present invention also provides a crowd density estimation system based on a content perception module, comprising: an image data processing module, a feature extraction module, a content perception module and a crowd density estimation module;

[0043] The image data processing module is used to:

[0044] Preprocess the image data to be predicted;

[0045] The feature extraction module is used to:

[0046] Extracting deep semantic information and multi-layer fusion context information based on the processed image data;

[0047] The content perception module is used to:

[0048] dynamically generating content-aware parameters based on the deep semantic information;

[0049] The crowd density estimation module is used to:

[0050] Initializing the weights and offsets of the convolutional layer according to the content-aware parameters;

[0051] Crowd density is estimated based on the multi-layer fusion context information to obtain a crowd density image.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] Compared with traditional crowd density estimation methods such as MCNN, CSRNET, PCCNET, and CAN, in which parameters are fixed after training, the present invention obtains the deep semantic information of the image data each time the crowd density is predicted, and dynamically generates the parameters of the convolutional layer of the crowd density estimation module, thereby performing crowd density estimation more flexibly and targetedly, and can obtain better prediction results in complex and changeable scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of a crowd density estimation method based on a content-aware module disclosed in an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of a crowd density estimation system based on a content-aware module disclosed in an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of the structure of a convolutional neural network model disclosed in one embodiment of the present invention;

[0057] Figure 4A schematic diagram of a feature extraction module in a convolutional neural network model disclosed in one embodiment of the present invention;

[0058] Figure 5 A schematic diagram of the detailed structure of a convolutional neural network model disclosed in one embodiment of the present invention. DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0060] The present invention is described in further detail below with reference to the accompanying drawings:

[0061] Example:

[0062] like Figure 1 As shown, the present invention provides a crowd density estimation method based on a content-aware module, comprising:

[0063] S1. Preprocessing the image data to be predicted;

[0064] in,

[0065] Preprocessing is required to enable input into the convolutional neural network model for prediction. The width and height of the input image must be limited to 512×512. To ensure that the image does not deform, we do not choose to directly resize it to 512×512. Instead, we choose to fill and randomly crop it. Images smaller than this size are filled with black pixels in the lower right corner to make it 512×512. Images larger than this size are first randomly cropped, and the undersized parts after cropping are filled with black pixels.

[0066] S2, input the processed data into the convolutional neural network model;

[0067] in,

[0068] like Figure 3 、 5 As shown in the figure, the convolutional neural network model includes a feature extraction module, a content perception module and a crowd density estimation module. The feature extraction module includes a backbone network and a feature fusion network.

[0069] The backbone network consists of one convolutional layer and four residual layers, and the feature fusion network adopts a feature pyramid network;

[0070] S3, the feature extraction module extracts deep semantic information and multi-layer fusion context information based on the processed image data;

[0071] in,

[0072] The processed image data is subjected to a series of convolutions, down-collections, and up-collections through one convolutional layer and four residual layers of the backbone network to obtain deep semantic information.

[0073] like Figure 4 As shown in the figure, through the feature fusion network, the output features of one convolutional layer and four residual layers in the backbone network are fused by feature splicing to obtain feature fusion context information.

[0074] S4, based on the content-aware module, dynamically generates content-aware parameters according to deep semantic information, such as Figure 4 As shown;

[0075] in,

[0076] The content perception module consists of three layers of 1×1 convolution and four groups of parameter generation modules. Each group of parameter generation modules includes a 3×3 convolution and an average pooling layer.

[0077] The deep semantic information is passed through three layers of 1×1 convolution and four sets of parameter generation modules to obtain four sets of content-aware parameters.

[0078] S4, initializing the weights and offsets of the convolutional layer of the crowd density estimation module according to the content-aware parameters;

[0079] in,

[0080] like Figure 5 As shown, the crowd density prediction module includes four groups of 1×1 convolutions;

[0081] The weights and offsets of the four groups of 1×1 convolutions of the crowd density prediction module are initialized according to the four groups of content-aware parameters.

[0082] S5. Input the multi-layer fusion context information into the initialized crowd density estimation module to obtain a crowd density image.

[0083] in,

[0084] The crowd density prediction module has four groups of 1×1 convolutions with channel numbers of 512, 256, 128, and 2 respectively;

[0085] Four groups of 1×1 convolutions sequentially perform image channel dimensionality reduction on the multi-layer fusion context information to obtain a crowd density feature map with 2 channels;

[0086] Perform heat map analysis and overall integration on the crowd density feature map to obtain the crowd density image and the total number of people in the image.

[0087] Furthermore, the formula for generating the crowd density image is:

[0088]

[0089] in,

[0090] i represents the i-th head index;

[0091] N represents the total number of people;

[0092] δ(x-xi) represents the function of the head position in the image;

[0093] represents the adaptive Gaussian kernel;

[0094] σ i is the Gaussian kernel standard deviation;

[0095] Represents the average of the sum of the Euclidean distances between the head and its three adjacent heads in the image;

[0096] β is the weight coefficient.

[0097] The configuration and training process of the convolutional neural network model in the present invention includes:

[0098] Step 1: Perform data preprocessing, data enhancement processing and multi-scale transformation on the training image data;

[0099] in,

[0100] This paper uses three public datasets: ShanghaiTech-A, ShanghaiTech-B, and UCF-QNRF;

[0101] In order to be able to input the network for training, the image needs to be preprocessed. The width and height of the input image must be limited to 512×512. To ensure that the image does not deform, the direct resize to 512×512 method is not chosen. Instead, the padding and random cropping method is chosen. Images smaller than this size are padded with black pixels in the lower right corner to make the size 512×512. Images larger than this size are first randomly cropped, and the undersized parts after cropping are padded with black pixels.

[0102] In order to enrich the dataset samples, this paper adopts data enhancement methods such as random cropping, random translation, random horizontal flipping, and random brightness. In addition, random erasing is added to improve the robustness of the model.

[0103] In order to further enrich the data samples, this paper designs a multi-scale training method suitable for crowd density estimation. It uses an adaptive Gaussian kernel to generate a true crowd density map. Then, the image and the true crowd density map are bilinearly interpolated and scaled every 5 epochs. The scaling range is randomly selected in the sequence [0.5, 0.75, 1, 1.25, 1.5]. Since bilinear interpolation scaling will cause the sum of all pixel values ​​in the final crowd density image (representing the number of pedestrians in the image) to change, for this part of the data, we need to perform pixel value transformation processing on each pixel value after the change. The formula is:

[0104]

[0105] in,

[0106] Ratio is the transformation coefficient;

[0107] Represents the sum of pixel values ​​of the crowd density image before bilinear interpolation scaling;

[0108] Represents the sum of pixel values ​​of the crowd density image after bilinear interpolation scaling;

[0109] Represents the pixel value of the final crowd density image.

[0110] Step 2: Configure and initialize the parameters of the backbone network, feature fusion network, and content perception module;

[0111] in,

[0112] The backbone network is modified from ResNet18. The main modifications include: 1) removing the final fully connected layer of ResNet18; 2) replacing the first convolutional layer of ResNet18 with a 7×7 convolution kernel, a stride of 2, and a padding of 3 with a 3×3 convolution kernel, a stride of 1, and a padding of 1, reducing image downsampling. Specifically, the backbone network consists of the first convolutional layer and four residual layers. Each residual layer has two residual blocks, and each residual block has two sets of convolution operations. The structure is 3×3 convolution, batch processing, and ReLu activation function. The residual blocks are connected by addition. The fourth residual layer of the backbone network outputs deep semantic information (feature map size 512×16×16), which serves as the input of the content-aware module.

[0113] The feature fusion module uses a feature pyramid module. Its input is the output of the first convolutional layer and the maximum pooling layer of the backbone network (feature map dimensions 64×256×256), the output of the first residual layer (feature map dimensions 64×128×128), the output of the second residual block (feature map dimensions 128×64×64), and the output of the third residual layer (feature map dimensions 512×32×32). The feature fusion module performs bilinear interpolation upsampling on the outputs of the first three residual layers, unifying the feature map width and height to 256×256. It then fuses the features by splicing to obtain feature fusion context information (feature map dimensions 512×256×256).

[0114] The content-aware module uses deep semantic information to dynamically generate the weights and offsets of the convolution layer and loads them into the convolution layer of the crowd density estimation module. The front end of the content-aware module is three layers of 1×1 convolution (output feature map dimension 512×16×16), and the back end is four groups of content-aware generation layers consisting of 3×3 convolution kernels, convolution layers with a stride of 1 and padding of 1, full connections, and average pooling layers. Four groups of weights and offsets are obtained after the content-aware layer. The first group has a weight dimension of 512×512×1×1 and an offset dimension of 512×1; the second group has a weight dimension of 256×512×1×1 and an offset dimension of 256×1; the third group has a weight dimension of 128×256×1×1 and an offset dimension of 128×1; the fourth group has a weight dimension of 2×128×1×1 and an offset dimension of 2×1;

[0115] Finally, the ResNet18 pre-trained model was used to initialize the parameters of the backbone network, and the Xavier method was used to initialize the parameters of the content perception module and feature fusion network.

[0116] Step 3: The transformed image data is fed into a convolutional neural network model that has been configured and initialized. The feature extraction module obtains deep semantic information and multi-layer fusion context information.

[0117] in,

[0118] The UCF-QNRF dataset contains 1201 images for training and 334 images for testing. It features complex scenes, high crowd density, and varied shooting angles, making it more realistic and difficult.

[0119] The ShanghaiTech-A dataset uses 300 images for training and 182 for testing. The images have large size variations, dense crowds, and large scene variations.

[0120] The ShanghaiTech-B dataset has 400 images for training and 316 for testing. The images have fixed size, irregular crowd distribution, and relatively simple scenes and shooting angles.

[0121] Step 4: The content-aware module converts the deep semantic information into weights and offsets of the convolutional layer and loads them into the convolutional layer of the crowd density estimation module.

[0122] in,

[0123] The crowd density prediction module consists of four layers of convolution, with convolution kernel sizes of 1×1×512, 1×1×256, 1×1×128, and 1×1×2, corresponding to the weights and offsets generated by the content-aware module. The four layers of convolution are initialized by the weights and offsets generated by the content-aware module.

[0124] Step 5: Send the multi-layer fusion context information to the crowd density estimation module to obtain the crowd density image;

[0125] in,

[0126] The mean square error loss function is used to train the crowd density image. The optimizer uses the momentum adaptive optimizer, the initial learning rate is set to 0.001, and a total of more than 200 iterations are performed.

[0127] Step 6: Test the accuracy of the convolutional neural network model using test data. If the accuracy meets the target requirements, the parameters of the backbone network, feature fusion network, and content perception module can be obtained.

[0128] Generally, the evaluation indicators of convolutional neural network models are mean absolute error (MAE) and mean square error (MSE), and the formulas are as follows:

[0129]

[0130] The prediction performance of the algorithm was evaluated on the UCF-QNRF, ShanghaiTech-A, and ShanghaiTech-B datasets. The prediction performance of the crowd density estimation method of the present invention was compared with that of conventional methods as shown in the following table. The method of the present invention achieved competitive results.

[0131]

[0132] like Figure 2 As shown, the present invention also provides a crowd density estimation system based on a content perception module, comprising: an image data processing module, a feature extraction module, a content perception module and a crowd density estimation module;

[0133] Image data processing module, used for:

[0134] Preprocess the image data to be predicted;

[0135] Feature extraction module, used to:

[0136] Extract deep semantic information and multi-layer fusion context information based on the processed image data;

[0137] Content-aware module for:

[0138] Dynamically generate content-aware parameters based on deep semantic information;

[0139] Crowd density estimation module, used for:

[0140] Initialize the weights and biases of the convolutional layers according to content-aware parameters;

[0141] Crowd density is estimated based on multi-layer fusion context information to obtain a crowd density image.

[0142] Advantages of the present invention:

[0143] Compared with traditional crowd density estimation methods such as MCNN, CSRNET, PCCNET, CAN, etc., where the parameters are fixed after training, this method obtains the deep semantic information of the image data each time the crowd density is predicted, and dynamically generates the parameters of the convolutional layer of the crowd density estimation module, thereby performing crowd density estimation more flexibly and targetedly, and obtaining better prediction results in complex and changing scenarios.

[0144] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A crowd density estimation method based on a content-aware module, characterized in that: include: Preprocess the image data to be predicted; Extracting deep semantic information and multi-layer fusion context information based on the processed image data, which includes: inputting the processed data into a convolutional neural network model; the convolutional neural network model includes a feature extraction module, a content perception module and a crowd density estimation module; the feature extraction module extracts deep semantic information and multi-layer fusion context information based on the processed image data; further, the feature extraction module includes a backbone network and a feature fusion network; the backbone network includes a layer of convolutional layer and four layers of residual layer, and performs a series of convolution, down-collection and up-collection on the processed image data to obtain the deep semantic information; the feature fusion network adopts a feature pyramid network, and fuses the output features of the layer of convolutional layer and the four layers of residual layer in the backbone network by feature splicing to obtain the feature fusion context information; Dynamically generating content-aware parameters based on the deep semantic information based on a content-aware module, wherein the content-aware module includes three layers of 1×1 convolution and four groups of parameter generation modules, each group of parameter generation modules includes a 3×3 convolution and an average pooling layer; the deep semantic information is passed through the three layers of 1×1 convolution and the four groups of parameter generation modules to obtain four groups of content-aware parameters; Initializing the weights and offsets of the convolutional layer of the crowd density estimation module according to the content-aware parameters; The multi-layer fusion context information is input into the initialized crowd density estimation module to obtain a crowd density image.

2. The method according to claim 1, wherein: The training process of the convolutional neural network model includes: Perform data preprocessing, data enhancement and multi-scale transformation on training image data; Initialize the parameters of the backbone network, feature fusion network and content perception module; The transformed image data is fed into the convolutional neural network model, and the deep semantic information and multi-layer fusion context information are obtained through the feature extraction module; The content perception module converts the deep semantic information into weights and offsets of the convolution layer, and loads them into the convolution layer of the crowd density estimation module; The multi-layer fusion context information is fed into the crowd density estimation module to obtain the crowd density image; The crowd density image is trained through the mean square error function to obtain the parameters of the backbone network, feature fusion network and content perception module.

3. The method according to claim 2, characterized in that The multi-scale transformation method is: Generate the ground-truth crowd density map of the image using adaptive Gaussian kernel; Perform bilinear interpolation scaling on the image and the ground-truth crowd density map every 5 epochs; For the crowd density image obtained by estimating the scaled image, the pixel value transformation process is finally performed using the formula: in, Ratio is the transformation coefficient; Represents the sum of pixel values ​​of the crowd density image before bilinear interpolation scaling; Represents the sum of pixel values ​​of the crowd density image after bilinear interpolation scaling; Indicates the pixel value of the final crowd density image.

4. The method according to claim 2, wherein: The data enhancement processing includes random cropping, random translation, random horizontal flipping, random brightness and random erasing of the image data.

5. The method according to claim 1, wherein: The crowd density prediction module includes four groups of 1×1 convolutions; The weights and offsets of the four groups of 1×1 convolutions of the crowd density prediction module are initialized respectively according to the four groups of content-aware parameters.

6. The method according to claim 5, characterized in that: The four groups of 1×1 convolutions in the crowd density prediction module have channel numbers of 512, 256, 128, and 2 respectively; The four groups of 1×1 convolutions sequentially perform image channel dimensionality reduction on the multi-layer fusion context information to obtain a crowd density feature map with a channel number of 2; Performing heat map analysis and overall integration on the crowd density feature map to obtain the crowd density image and the total number of people in the image.

7. A system for implementing the crowd density estimation method according to any one of claims 1 to 6, characterized in that: include: Image data processing module, feature extraction module, content perception module and crowd density estimation module; The image data processing module is used to: Preprocess the image data to be predicted; The feature extraction module is used to: Extracting deep semantic information and multi-layer fusion context information based on the processed image data; The content perception module is used to: dynamically generating content-aware parameters based on the deep semantic information; The crowd density estimation module is used to: Initializing the weights and offsets of the convolutional layer according to the content-aware parameters; Crowd density is estimated based on the multi-layer fusion context information to obtain a crowd density image.

Images