Method, device and equipment for estimating crowd density and storage medium

Abstract

The application provides a crowd density estimation method, device, equipment and storage medium. The method comprises: acquiring a plurality of video frames in a to-be-processed video; sequentially inputting the plurality of video frames into a crowd density estimation model to output a crowd density map corresponding to each video frame and having context information. The crowd density estimation model comprises a 3D (three-dimensional) convolution (3D Conv) module, N P3D (pseudo 3D) convolution (P3D Conv) modules and M 2D (two-dimensional) convolution (2D Conv) modules connected in sequence. The N P3D (pseudo 3D) convolution (P3D Conv) modules each comprise an attention unit, and the attention unit is used to extract the time correlation between the plurality of video frames. Through the method, the time correlation information between the video frames can be extracted and combined to determine the crowd density map corresponding to the video frame, thereby improving the accuracy of video crowd counting.

Description

Technical Field

[0001] This application relates to image recognition technology, and more particularly to a method, apparatus, device, and storage medium for estimating crowd density. Background Technology

[0002] In recent years, with social development, video surveillance equipment has been widely deployed. In areas such as train stations and shopping malls, in order to better serve users, the density of people in video frames is usually estimated to determine the corresponding pedestrian flow at different times.

[0003] Existing technologies determine the crowd density of video frame images using a three-dimensional convolutional neural network model (C3D) or an improved three-dimensional convolutional neural network model (E3D).

[0004] However, due to limitations in the structure of 3D convolutional neural networks and the size of video crowd counting datasets, the accuracy of existing C3D and E3D-based video crowd counting models needs to be improved. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for estimating crowd density, which can extract and combine temporal correlation information between video frames to determine the crowd density map corresponding to the video frame, thereby improving the accuracy of video crowd counting.

[0006] Firstly, this application provides a method for estimating crowd density. The method includes: acquiring multiple video frames from a video to be processed; sequentially inputting the multiple video frames into a crowd density estimation model to output a crowd density map with contextual information corresponding to each video frame. The crowd density estimation model includes one sequentially connected 3D convolutional (3D Conv) module, N pseudo-3D convolutional (P3D Conv) modules, and M 2D convolutional (2D Conv) modules. Each of the N P3D Conv modules includes an attention unit, which is used to extract the temporal correlation between multiple video frames. Optionally, each of the N P3D Conv modules includes a feature extraction convolutional unit, and each of the M 2D Conv modules includes a dimensionality reduction convolutional unit. The 3D Conv module, the N P3D Conv modules, and the M 2D Conv modules are sequentially connected in series. Within the same P3D Conv module, every two feature extraction convolutional units and attention units are alternately connected in series.

[0007] Multiple video frames are sequentially input into the crowd density estimation model to output a crowd density map with contextual information for each video frame. This process includes: for each video frame, inputting the video frame into the 3DConv module of the crowd density estimation model to obtain a first multidimensional feature map; starting from the feature extraction convolutional unit of the first P3D Conv module, inputting the first multidimensional feature map into the feature extraction convolutional unit of the first P3D Conv module to obtain a second multidimensional feature map; processing the second multidimensional feature map according to the attention unit in the first P3D Conv module, and inputting the multidimensional feature map processed by the first P3D Conv module into a new feature extraction convolutional unit, until the next module is a 2D Conv module; inputting the third multidimensional feature map output by the last P3D Conv module into the 2D Conv module to perform dimensionality reduction processing on the third multidimensional feature map to obtain a crowd density map with contextual information for each video frame.

[0008] Optionally, the multidimensional feature map is processed according to the attention unit in the P3D Conv module, including: performing global average pooling and global maximum pooling on the multidimensional feature map through the attention unit to obtain a first processing result and a second processing result; concatenating the first processing result and the second processing result to obtain a concatenated result; and processing the concatenated result through a multilayer perceptron and a Sigmund activation function to obtain the weight values ​​of the multidimensional feature map for different channels, wherein the weight values ​​are used to represent the importance of the channel to the multidimensional feature map.

[0009] Optionally, the stride of the convolution kernel in the 3D Conv module is 1x1x1.

[0010] Optionally, the feature extraction convolutional unit includes two spatial convolutional layers with 1x3x3 kernels and two temporal convolutional layers with 3x1x1 kernels; wherein, the first spatial convolutional layer and the first temporal convolutional layer are connected in series, the first temporal convolutional layer is connected to the second spatial convolutional layer and the second temporal convolutional layer respectively, and the sum of the outputs of the second spatial convolutional layer and the second temporal convolutional layer, and the sum of the result with the input of the first spatial convolutional layer, is used as the output of the feature extraction convolutional layer.

[0011] Optionally, the feature extraction convolutional unit includes two spatial convolutional layers with 1x3x3 kernels and two temporal convolutional layers with 3x1x1 kernels; wherein the first spatial convolutional layer and the first temporal convolutional layer are respectively connected to the second spatial convolutional layer, the second spatial convolutional layer is connected to the second temporal convolutional layer, and the sum of the outputs of the second spatial convolutional layer and the second temporal convolutional layer, and the sum of the result with the input of the first spatial convolutional layer, is used as the output of the feature extraction convolutional layer.

[0012] Optionally, the feature extraction convolutional unit includes two spatial convolutional layers with 1x3x3 kernels and two temporal convolutional layers with 3x1x1 kernels; wherein, the first spatial convolutional layer, the first temporal convolutional layer, the second spatial convolutional layer, and the second temporal convolutional layer are connected in series in sequence, and the sum of the outputs of the second spatial convolutional layer and the second temporal convolutional layer, and the sum of the result with the input of the first spatial convolutional layer, is used as the output of the feature extraction convolutional layer.

[0013] Secondly, this application provides the structure of a population density estimation device, the device comprising:

[0014] The acquisition module is used to acquire multiple video frames from the video to be processed.

[0015] The processing module is used to input multiple video frames sequentially into the crowd density estimation model to output a crowd density map with contextual information for each video frame. The crowd density estimation model includes one 3D Conv module, N P3D Conv modules and M 2D Conv modules connected in sequence. Each of the N P3D Conv modules includes an attention unit, which is used to extract the temporal correlation between multiple video frames.

[0016] Optionally, each of the N P3D Conv modules includes a feature extraction convolutional unit, and each of the M 2D Conv modules includes a dimensionality reduction convolutional unit. The 3D Conv modules, the N P3D Conv modules, and the M 2D Conv modules are connected in series sequentially. In the same P3D Conv module, every two feature extraction convolutional units and attention units are connected in series alternately.

[0017] The processing module is specifically used to input the video frame into the 3DConv module of the crowd density estimation model for each video frame to obtain a first multidimensional feature map. Starting from the feature extraction convolutional unit of the first P3D Conv module, the first multidimensional feature map is input into the feature extraction convolutional unit of the first P3D Conv module to obtain a second multidimensional feature map. The second multidimensional feature map is processed according to the attention unit in the first P3D Conv module, and the multidimensional feature map processed by the first P3D Conv module is input into a new feature extraction convolutional unit until the next module is a 2D Conv module. The third multidimensional feature map output by the last P3D Conv module is input into the 2D Conv module to perform dimensionality reduction processing on the third multidimensional feature map to obtain a crowd density map with contextual information corresponding to each video frame.

[0018] Optionally, the processing unit 82 is specifically used to perform global average pooling and global maximum pooling on the multidimensional feature map through the attention unit to obtain a first processing result and a second processing result; to concatenate the first processing result and the second processing result to obtain a concatenated result; and to process the concatenated result through a multilayer perceptron and a Sigmund activation function to obtain the weight values ​​of the multidimensional feature map of different channels, wherein the weight values ​​are used to represent the importance of the channel to the multidimensional feature map.

[0019] Optionally, the stride of the convolution kernel in the 3D Conv module is 1x1x1.

[0020] Optionally, the feature extraction convolutional unit includes two spatial convolutional layers with 1x3x3 kernels and two temporal convolutional layers with 3x1x1 kernels; wherein, the first spatial convolutional layer and the first temporal convolutional layer are connected in series, the first temporal convolutional layer is connected to the second spatial convolutional layer and the second temporal convolutional layer respectively, and the sum of the outputs of the second spatial convolutional layer and the second temporal convolutional layer, and the sum of the result with the input of the first spatial convolutional layer, is used as the output of the feature extraction convolutional layer.

[0021] Optionally, the feature extraction convolutional unit includes two spatial convolutional layers with 1x3x3 kernels and two temporal convolutional layers with 3x1x1 kernels; wherein the first spatial convolutional layer and the first temporal convolutional layer are respectively connected to the second spatial convolutional layer, the second spatial convolutional layer is connected to the second temporal convolutional layer, and the sum of the outputs of the second spatial convolutional layer and the second temporal convolutional layer, and the sum of the result with the input of the first spatial convolutional layer, is used as the output of the feature extraction convolutional layer.

[0022] Optionally, the feature extraction convolutional unit includes two spatial convolutional layers with 1x3x3 kernels and two temporal convolutional layers with 3x1x1 kernels; wherein, the first spatial convolutional layer, the first temporal convolutional layer, the second spatial convolutional layer, and the second temporal convolutional layer are connected in series in sequence, and the sum of the outputs of the second spatial convolutional layer and the second temporal convolutional layer, and the sum of the result with the input of the first spatial convolutional layer, is used as the output of the feature extraction convolutional layer.

[0023] Thirdly, this application provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in the first aspect or an alternative method of the first aspect.

[0024] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement a method as described in the first aspect or an alternative method of the first aspect.

[0025] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement a method as described in the first aspect or an alternative method of the first aspect.

[0026] The crowd density estimation method, apparatus, device, and storage medium provided in this application acquire multiple video frames from a video to be processed; input the multiple video frames sequentially into a crowd density estimation model to output a crowd density map with contextual information corresponding to each video frame. The crowd density estimation model includes one 3D Conv module, N P3D Conv modules, and M 2D Conv modules connected in sequence. Each of the N P3D Conv modules includes an attention unit, which is used to extract the temporal correlation between multiple video frames. It can extract and combine the temporal correlation information between video frames to determine the crowd density map corresponding to the video, thereby improving the accuracy of video crowd counting. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A flowchart illustrating a method for estimating population density provided in this application;

[0029] Figure 2 A schematic diagram of the architecture of a population density estimation model provided in this application;

[0030] Figure 3 A schematic diagram of a P3D Conv module architecture is provided for this application;

[0031] Figure 4 This application provides a schematic diagram of the structure of a feature extraction convolutional unit;

[0032] Figure 5 A schematic diagram of another feature extraction convolutional unit provided in this application;

[0033] Figure 6 A schematic diagram of the structure of another feature extraction convolutional unit provided in this application;

[0034] Figure 7 A schematic diagram of the structure of an attention unit provided in this application;

[0035] Figure 8 A flowchart illustrating another method for estimating population density provided in this application;

[0036] Figure 9 A schematic diagram of the structure of a population density estimation device provided in this application;

[0037] Figure 10 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] C3D and E3D are commonly used to determine crowd density in video frames. However, since a 3D convolutional neural network is four times the size of a 2D convolutional neural network of the same depth, achieving the same depth with a 3D convolutional neural network as a 2D neural network would result in a large model size. Without significantly increasing the model size, a 3D convolutional neural network may not be able to achieve the desired depth, thus limiting the accuracy of crowd density estimation in video frames.

[0040] Based on this, this application provides a method for estimating crowd density. This method acquires multiple video frames from a video to be processed, sequentially inputs these frames into a crowd density estimation model including an attention unit, and outputs a crowd density map with contextual information for each video frame. By adding an attention module to the model, the temporal correlation information between video frames can be extracted and combined to determine the crowd density map corresponding to the video, thereby improving the accuracy of crowd counting in the video, i.e., improving the accuracy of crowd density recognition for video frames.

[0041] Figure 1 A flowchart illustrating a method for estimating crowd density provided in this application, which is applied to electronic devices, such as... Figure 1 As shown, the method includes:

[0042] S101. Obtain multiple video frames from the video to be processed.

[0043] Electronic devices can receive multiple video frames input by the user through their input ports; they can also obtain multiple video frames from virtual or physical storage media.

[0044] Electronic devices can also receive user-inputted video to be processed through their input ports, or obtain the video to be processed from virtual or physical storage media, and process the video to be processed to obtain multiple video frames.

[0045] S102. Input multiple video frames sequentially into the crowd density estimation model to output a crowd density map with contextual information for each video frame.

[0046] The crowd density estimation model consists of one 3D Conv module, N P3D Conv modules, and M 2D Conv modules connected in sequence.

[0047] Each of the N P3D Conv modules includes an attention unit, which is used to extract the temporal correlation between multiple video frames.

[0048] Specifically, M is a positive integer greater than or equal to 1, and N is a positive integer greater than or equal to 2.

[0049] A crowd density map is an image used to characterize the density of people in an image.

[0050] Crowd density maps with contextual information are derived from the spatiotemporal information of several video frames before and after a specific video frame, which contribute to the crowd density map results for that specific video frame.

[0051] For example, the crowd density estimation model can be constructed using the pseudo-3D convolutional network P3D ResNet-34 as the backbone. Specifically, the model components after the fifth convolutional module (Conv5) in the P3D ResNet-34 model are removed. All pooling operations in the P3D ResNet-34 model are removed to reduce the loss of temporal information caused by pooling operations in the time dimension. The stride of the convolutional kernel operation in the first convolutional module (Conv1) of the P3D ResNet-34 model is set to 1x1x1 to improve the resolution of the model's output feature map. After Conv5 in the P3D ResNet-34 model, two different convolutional modules, Conv6 and Conv7, are added to reduce the dimensionality of the multidimensional feature image output by Conv5, ultimately obtaining a crowd density map with a size of 1 / 8 of the original input image size and 1 channel.

[0052] Furthermore, the 3x3x3 convolutional modules in the second to fifth convolutional modules are decomposed into a two-dimensional spatial convolutional module with a kernel size of 1x3x3 (S, 1x3x3) and a one-dimensional temporal convolutional module with a kernel size of 3x1x1 (T, 3x1x1). These two convolutional modules encode the spatial and temporal information in the image, respectively. The spatial and temporal convolutional modules can each employ different architectures of the P3DConv module to improve the diversity of the backbone network of the crowd density estimation model.

[0053] Furthermore, temporal channel attention modules are introduced into the second to fifth convolutional modules respectively to extract the temporal correlation between different frames in the video, while encoding global contextual information to improve the accuracy of the model's video crowd counting.

[0054] This application embodiment acquires multiple video frames from the video to be processed, inputs these multiple video frames sequentially into a crowd density estimation model including an attention unit, and outputs a crowd density map with contextual information corresponding to each video frame. It can extract and combine the temporal correlation information between video frames to determine the crowd density map corresponding to the video, thereby improving the accuracy of video crowd counting.

[0055] Figure 2 A schematic diagram of the architecture of a population density estimation model provided in this application is shown below. Figure 2 As shown, the population density estimation model includes one 3D Conv module, N P3D Conv modules, and M 2D Conv modules.

[0056] One 3D Conv module, N P3D Conv modules, and M 2D Conv modules are connected in series. Figure 2 In the example shown, N is 5 and M is 2. It should be understood that this is merely an example and not a limiting description.

[0057] Specifically, each P3D Conv module includes a feature extraction unit and an attention unit. Specifically, within the same P3D Conv module, every two feature extraction convolutional units and one attention unit are alternately connected in series. Figure 3 This application provides an architectural diagram of a P3D Conv module. Figure 3The P3D Conv module shown includes 6 feature extraction convolutional units and 3 attention units. Specifically, the input of this P3D Conv module is the output of the previous module. The first and second feature extraction convolutional units are concatenated and then concatenated with the first attention unit; the output of the first attention unit is the input of the third feature extraction convolutional unit; the third and fourth feature extraction convolutional units are concatenated and then concatenated with the second attention unit; the output of the second attention unit is the input of the fifth feature extraction convolutional unit; the fifth and sixth feature extraction convolutional units are concatenated and then concatenated with the third attention unit; the output of the third attention unit is the input of the next module.

[0058] The attention unit is a time-channel attention unit.

[0059] The P3D Conv module's convolutional layers include layers used for extracting image features.

[0060] Optionally, the stride of the convolutional sum of this convolutional layer is 1x1x1.

[0061] This method can improve the resolution of the feature map output by the P3D Conv module, thereby improving the accuracy of the crowd density estimation model.

[0062] Optionally, the feature extraction convolutional unit includes two spatial convolutional layers with 1x3x3 kernels and two temporal convolutional layers with 3x1x1 kernels.

[0063] Figure 4 This application provides a schematic diagram of the structure of a feature extraction convolutional unit, as shown below. Figure 4 As shown, in one possible implementation, the connection method of the two spatial convolutional layers with 1x3x3 kernels and the two temporal convolutional layers with 3x1x1 kernels includes: the first spatial convolutional layer and the first temporal convolutional layer are connected in series; the first temporal convolutional layer is connected to the second spatial convolutional layer and the second temporal convolutional layer respectively; the sum of the outputs of the second spatial convolutional layer and the second temporal convolutional layer, and the sum of the result with the input of the first spatial convolutional layer, is used as the output of the feature extraction convolutional layer.

[0064] Figure 5 A schematic diagram of another feature extraction convolutional unit provided in this application is shown below. Figure 5As shown, in one possible implementation, the connection method of the two spatial convolutional layers with 1x3x3 kernels and the two temporal convolutional layers with 3x1x1 kernels includes: the first spatial convolutional layer and the first temporal convolutional layer are respectively connected to the second spatial convolutional layer, the second spatial convolutional layer is connected to the second temporal convolutional layer, and the sum of the outputs of the second spatial convolutional layer and the second temporal convolutional layer, and the sum of the result, and the input of the first spatial convolutional layer, is used as the output of the feature extraction convolutional layer.

[0065] Figure 6 A schematic diagram of another feature extraction convolutional unit provided in this application is shown below. Figure 6 As shown, in one possible implementation, the connection method of two spatial convolutional layers with 1x3x3 kernels and two temporal convolutional layers with 3x1x1 kernels includes: the first spatial convolutional layer, the first temporal convolutional layer, the second spatial convolutional layer, and the second temporal convolutional layer are connected in series in sequence, and the sum of the outputs of the second spatial convolutional layer and the second temporal convolutional layer, and the sum of the result with the input of the first spatial convolutional layer, is used as the output of the feature extraction convolutional layer.

[0066] Specifically, the spatial convolution module is a 2D Conv module used to process spatial information in video frame images. The temporal convolution module is a one-dimensional convolution module used to process temporal information in video frame images.

[0067] Correspondingly, the connection methods of the temporal and spatial convolutional layers in the feature extraction convolutional units of any P3D Conv module in the crowd density estimation model can be the same or different; the connection methods of the temporal and spatial convolutional layers in the feature extraction convolutional units of different P3D Conv modules can be the same or different.

[0068] This method can improve the diversity of the backbone network of the population density estimation model and improve the accuracy of the estimation results.

[0069] Figure 7 A schematic diagram of the structure of an attention unit provided in this application is shown below. Figure 7 As shown, the attention unit includes an average pooling unit, a global pooling unit, a concatenation unit, a processing unit, and a scaling unit;

[0070] The outputs of the average pooling unit and the global pooling unit are connected to the input of the splicing unit; the output of the splicing unit is connected to the input of the processing unit; and the output of the processing unit is connected to the input of the scaling unit.

[0071] The average pooling unit is used to perform global average pooling on the multidimensional feature map to obtain the first processing structure.

[0072] The global pooling unit is used to perform global max pooling on the multidimensional feature map to obtain the second processing result.

[0073] The splicing unit is used to splice the first processing result and the second processing result to obtain the spliced ​​result.

[0074] The processing unit includes a Multilayer Perceptron (MLP) and a Sigmoid activation function, used to process the stitched result through the MLP and Sigmoid activation function to obtain weight values ​​for multidimensional feature maps of different channels. The weight value corresponding to any channel is used to characterize the contribution of that channel's feature map to the final population counting result.

[0075] The scaling unit is used to multiply the tensor corresponding to the multidimensional feature map with the corresponding weight scalar output by the processing unit, and output the corresponding result to realize the scaling of the multidimensional feature map.

[0076] Figure 8 A flowchart illustrating another crowd density estimation method provided in this application, applied to an electronic device that includes the aforementioned crowd density estimation model, such as... Figure 8 As shown, the method includes:

[0077] S801. Obtain multiple video frames from the video to be processed.

[0078] S801 and S101 have the same technical features. For a detailed description, please refer to S101. They will not be repeated here.

[0079] S802. For each video frame, input the video frame into the 3D Conv module in the crowd density estimation model to obtain the first multidimensional feature map.

[0080] The video frame image is input into a 3D Conv module with a stride of 1x1x1. For example, the convolution kernel of this 3D Conv module is 3x3x3.

[0081] S803. Starting from the feature extraction convolutional unit of the first P3D Conv module, the first multidimensional feature map is input into the feature extraction convolutional unit of the first P3D Conv module to obtain the second multidimensional feature map. The second multidimensional feature map is processed according to the attention unit in the first P3D Conv module, and the multidimensional feature map processed by the first P3D Conv module is input into the new feature extraction convolutional unit, until the next module is a 2D Conv module.

[0082] That is, the first multidimensional feature map is processed by two feature extraction convolutional units and one attention unit alternately connected in the preceding P3D Conv module, and then output to the subsequent P3D Conv module. The subsequent P3D Conv module then processes it further by two feature extraction convolutional units and one attention unit alternately connected in the preceding P3D Conv module, and outputs it after the processing is completed.

[0083] In one possible implementation, processing the multidimensional feature map according to the attention unit in the P3D Conv module includes: performing global average pooling and global maximum pooling on the multidimensional feature map through the attention unit to obtain a first processing result and a second processing result; concatenating the first processing result and the second processing result to obtain a concatenated result; and processing the concatenated result through a multilayer perceptron and a Sigmund activation function to obtain weight values ​​of the multidimensional feature map for different channels, whereby the weight values ​​represent the importance of the channels to the multidimensional feature map.

[0084] This method, through the processing of attention units, can extract the temporal correlation between different frames in a video and encode global contextual information, thereby improving the accuracy of the crowd density estimation model in estimating crowd density in a video.

[0085] S804. Input the third multidimensional feature map output by the last P3D Conv module into the 2D Conv module to perform dimensionality reduction processing on the third multidimensional feature map, and obtain the crowd density map with contextual information corresponding to each video frame.

[0086] The 2D Conv module is used to perform dimensionality reduction on the feature image. There can be one or more 2D Conv modules.

[0087] For example, when there are two 2D Conv modules, the parameters of the first 2D Conv module can be 1x1, 256, stride = 1; the parameters of the second 2D Conv module can be 1x1, 1, stride = 1. Through the processing of the above two 2D Conv modules, the number of channels of the image can be reduced to 1, resulting in a crowd density map with a size of 1 / 8 of the original input image and a number of channels of 1.

[0088] This application embodiment acquires multiple video frames from the video to be processed. For each video frame, the video frame is input into the 3D Conv module of the crowd density estimation model to obtain a first multidimensional feature map. Starting from the feature extraction convolutional unit of the first P3D Conv module, the first multidimensional feature map is input into the feature extraction convolutional unit of the first P3D Conv module to obtain a second multidimensional feature map. The second multidimensional feature map is processed according to the attention unit in the first P3D Conv module, and the multidimensional feature map processed by the first P3D Conv module is input into a new feature extraction convolutional unit until the next module is a 2D Conv module. The third multidimensional feature map output by the last P3D Conv module is input into the 2D Conv module to perform dimensionality reduction processing on the third multidimensional feature map, thereby obtaining a crowd density map with contextual information corresponding to each video frame. This can improve the resolution of the feature map output by the model, increase the diversity of the model structure, and thus improve the accuracy of video crowd counting and the accuracy of the determined crowd density map.

[0089] Figure 9 A schematic diagram of a population density estimation device provided in this application is shown below. Figure 9 As shown, the device includes:

[0090] The acquisition module 91 is used to acquire multiple video frames from the video to be processed.

[0091] The processing module 92 is used to input multiple video frames sequentially into the crowd density estimation model to output a crowd density map with contextual information corresponding to each video frame. The crowd density estimation model includes one 3D Conv module, N P3D Conv modules and M 2D Conv modules connected in sequence. Each of the N P3D Conv modules includes an attention unit, which is used to extract the temporal correlation between multiple video frames.

[0092] Optionally, each of the N 3D Conv modules includes a feature extraction convolutional unit, and each of the M 2D Conv modules includes a dimensionality reduction convolutional unit. The 3D Conv modules, the N P3D Conv modules, and the M 2D Conv modules are connected in series sequentially. In the same P3D Conv module, every two feature extraction convolutional units and attention units are connected in series alternately.

[0093] Processing module 92 is specifically used to input the video frame into the 3DConv module of the crowd density estimation model for each video frame to obtain a first multidimensional feature map; starting from the feature extraction convolutional unit of the first P3D Conv module, the first multidimensional feature map is input into the feature extraction convolutional unit of the first P3D Conv module to obtain a second multidimensional feature map; the second multidimensional feature map is processed according to the attention unit in the first P3D Conv module, and the multidimensional feature map processed by the first P3D Conv module is input into a new feature extraction convolutional unit until the next module is a 2D Conv module; the third multidimensional feature map output by the last P3D Conv module is input into the 2D Conv module to perform dimensionality reduction processing on the third multidimensional feature map to obtain a crowd density map with contextual information corresponding to each video frame.

[0094] Optionally, the processing unit 92 is specifically used to perform global average pooling and global maximum pooling on the multidimensional feature map through the attention unit to obtain a first processing result and a second processing result; to concatenate the first processing result and the second processing result to obtain a concatenated result; and to process the concatenated result through a multilayer perceptron and a Sigmund activation function to obtain the weight values ​​of the multidimensional feature map of different channels, wherein the weight values ​​are used to represent the importance of the channel to the multidimensional feature map.

[0095] Optionally, the stride of the convolution kernel in the 3D Conv module is 1x1x1.

[0096] Optionally, the feature extraction convolutional unit includes two spatial convolutional layers with 1x3x3 kernels and two temporal convolutional layers with 3x1x1 kernels; wherein, the first spatial convolutional layer and the first temporal convolutional layer are connected in series, the first temporal convolutional layer is connected to the second spatial convolutional layer and the second temporal convolutional layer respectively, and the sum of the outputs of the second spatial convolutional layer and the second temporal convolutional layer, and the sum of the result with the input of the first spatial convolutional layer, is used as the output of the feature extraction convolutional layer.

[0097] Optionally, the feature extraction convolutional unit includes two spatial convolutional layers with 1x3x3 kernels and two temporal convolutional layers with 3x1x1 kernels; wherein the first spatial convolutional layer and the first temporal convolutional layer are respectively connected to the second spatial convolutional layer, the second spatial convolutional layer is connected to the second temporal convolutional layer, and the sum of the outputs of the second spatial convolutional layer and the second temporal convolutional layer, and the sum of the result with the input of the first spatial convolutional layer, is used as the output of the feature extraction convolutional layer.

[0098] Optionally, the feature extraction convolutional unit includes two spatial convolutional layers with 1x3x3 kernels and two temporal convolutional layers with 3x1x1 kernels; wherein, the first spatial convolutional layer, the first temporal convolutional layer, the second spatial convolutional layer, and the second temporal convolutional layer are connected in series in sequence, and the sum of the outputs of the second spatial convolutional layer and the second temporal convolutional layer, and the sum of the result with the input of the first spatial convolutional layer, is used as the output of the feature extraction convolutional layer.

[0099] The crowd density estimation device can perform the crowd density estimation method described above. Its content and effects can be found in the method embodiment section, and will not be repeated here.

[0100] Figure 10 A schematic diagram of the structure of the electronic device provided in this application, such as Figure 10 As shown, the electronic device of this embodiment includes a processor 101 and a memory 102; the processor 101 and the memory 102 are communicatively connected. The memory 102 is used to store computer programs. The processor 101 is used to call the computer program stored in the memory 102 to implement the method in the above embodiment.

[0101] Optionally, the electronic device also includes a transceiver 103 for communicating with other devices.

[0102] The electronic device can perform the above-described method for estimating crowd density, the details of which can be found in the method implementation section and will not be repeated here.

[0103] This disclosure also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the methods as described in any of the above method embodiments.

[0104] When the computer-readable storage medium stores the computer-executable instructions, it can implement the above-mentioned method for estimating population density when executed by a processor. The content and effects can be found in the method embodiment section, and will not be repeated here.

[0105] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the methods as described in any of the above method embodiments.

[0106] When the computer program is executed by the processor, it can implement the above-mentioned method for estimating population density. Its content and effects can be found in the method implementation section, and will not be repeated here.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method of estimating crowd density, characterized by, include: Retrieve multiple video frames from the video to be processed; The multiple video frames are sequentially input into the crowd density estimation model to output a crowd density map with contextual information for each video frame. The crowd density estimation model includes one 3D convolutional module, N pseudo-3D convolutional modules, and M 2D convolutional modules connected in sequence. Each of the N pseudo-3D convolutional modules includes an attention unit, which is used to extract the temporal correlation between multiple video frames. Each of the N P3D Conv modules includes a feature extraction convolutional unit, and each of the M 2D Conv modules includes a dimensionality reduction convolutional unit. The 3D Conv module, the N P3D Conv modules, and the M 2D Conv modules are connected in series in sequence. In the same P3D Conv module, every two feature extraction convolutional units and the attention unit are connected in series alternately. The step of sequentially inputting the plurality of video frames into the crowd density estimation model to output a crowd density map with contextual information corresponding to each video frame includes: For each video frame, the video frame is input into the 3D Conv module in the crowd density estimation model to obtain the first multidimensional feature map; Starting from the feature extraction convolutional unit of the first P3D Conv module, the first multidimensional feature map is input into the feature extraction convolutional unit of the first P3D Conv module to obtain the second multidimensional feature map. The second multidimensional feature map is processed according to the attention unit in the first P3D Conv module, and the multidimensional feature map processed by the first P3D Conv module is input into the feature extraction convolutional unit of the next P3D Conv module, until the next module is a 2DConv module. The third multidimensional feature map output by the last P3D Conv module is input into the 2D Conv module to perform dimensionality reduction processing on the third multidimensional feature map, thereby obtaining a crowd density map with contextual information corresponding to each video frame.

2. The method of claim 1, wherein, The multidimensional feature map is processed using the attention unit in the P3D Conv module, including: The attention unit performs global average pooling and global maximum pooling on the multidimensional feature map to obtain the first processing result and the second processing result. The first processing result and the second processing result are concatenated to obtain the concatenated result; The splicing result is processed by a multilayer perceptron and a Sigmund activation function to obtain the weight values ​​of the multidimensional feature maps of different channels. The weight values ​​are used to represent the importance of the channel to the multidimensional feature map.

3. The method according to any of claims 1-2, characterized in that, The convolution kernel in the 3D Conv module has a stride of 1x1x1.

4. The method according to claim 1 or 2, characterized in that, The feature extraction convolutional unit includes two spatial convolutional layers with 1x3x3 kernels and two temporal convolutional layers with 3x1x1 kernels; In this configuration, a first spatial convolutional layer and a first temporal convolutional layer are connected in series. The first temporal convolutional layer is connected to a second spatial convolutional layer and a second temporal convolutional layer, respectively. The sum of the outputs of the second spatial convolutional layer and the second temporal convolutional layer, and the sum of the input of the first spatial convolutional layer, is used as the output of the feature extraction convolutional layer.

5. The method according to claim 1 or 2, characterized in that, The feature extraction convolutional unit includes two spatial convolutional layers with 1x3x3 kernels and two temporal convolutional layers with 3x1x1 kernels; The first spatial convolutional layer and the first temporal convolutional layer are respectively connected to the second spatial convolutional layer, and the second spatial convolutional layer is connected to the second temporal convolutional layer. The sum of the outputs of the second spatial convolutional layer and the second temporal convolutional layer, and the sum of the outputs of the first spatial convolutional layer and the input of the first spatial convolutional layer, is used as the output of the feature extraction convolutional layer.

6. The method of claim 1 or 2, wherein, The feature extraction convolutional unit includes two spatial convolutional layers with 1x3x3 kernels and two temporal convolutional layers with 3x1x1 kernels; The first spatial convolutional layer, the first temporal convolutional layer, the second spatial convolutional layer, and the second temporal convolutional layer are connected in series. The sum of the outputs of the second spatial convolutional layer and the second temporal convolutional layer, and the sum of the input of the first spatial convolutional layer, is used as the output of the feature extraction convolutional layer.

7. A device for estimating population density, characterized in that, include: The acquisition module is used to acquire multiple video frames from the video to be processed; The processing module is used to input the multiple video frames sequentially into the crowd density estimation model to output a crowd density map with contextual information corresponding to each video frame. The crowd density estimation model is sequentially connected to one 3D Conv module, N P3D Conv modules and M 2D Conv modules. Each of the N P3D Conv modules includes an attention unit, which is used to extract the temporal correlation between multiple video frames. Each of the N P3D Conv modules includes a feature extraction convolutional unit, and each of the M 2D Conv modules includes a dimensionality reduction convolutional unit. The 3D Conv module, the N P3D Conv modules, and the M 2D Conv modules are connected in series in sequence. In the same P3D Conv module, every two feature extraction convolutional units and the attention unit are connected in series alternately. The processing module is specifically configured to: for each video frame, input the video frame into the 3D Conv module of the crowd density estimation model to obtain a first multidimensional feature map; starting from the feature extraction convolutional unit of the first P3D Conv module, input the first multidimensional feature map into the feature extraction convolutional unit of the first P3D Conv module to obtain a second multidimensional feature map; process the second multidimensional feature map according to the attention unit in the first P3D Conv module, and input the multidimensional feature map processed by the first P3D Conv module into the feature extraction convolutional unit of the next P3D Conv module, until the next module is a 2D Conv module; input the third multidimensional feature map output by the last P3D Conv module into the 2D Conv module to perform dimensionality reduction processing on the third multidimensional feature map to obtain a crowd density map with contextual information corresponding to each video frame.

8. An electronic device, comprising: The invention includes a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, implements the method as described in any one of claims 1-6.

9. A storage medium, characterized by The storage medium stores instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1-6.

Images