Crowd counting method and system based on multi-scale fusion and distribution matching
Through the multi-scale fusion and distribution matching population counting method, the VGG backbone network and attention mechanism are used, combined with multi-scale feature fusion and distribution matching technology, the problem of low accuracy of deep learning population counting method in dense population scenarios is solved. A more accurate population count is achieved.
Patent Information
- Application Number
- CN202410622387.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-20
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-05-20
AI Technical Summary
The existing deep learning-based population counting method has low accuracy in dense population scenarios, ignoring the detailed information of low-order features and population density scale differences, resulting in a reduced prediction accuracy.
Using a multi-scale fusion and distribution matching population counting method, the attention mechanism and multi-scale feature fusion module are introduced through the VGG backbone network, a multi-scale branch prediction model is constructed, a density map is generated, and the optimal transmission loss and count loss functions are used for training. , combine distribution matching and multi-scale fusion to generate density maps.
It improves the accuracy of density map prediction, can more accurately identify and detect people, reduce the error detection rate, take into account high-order semantic information and low-order detailed information, and is suitable for counting of dense crowd scenarios.
Smart Images

Figure CN118447452B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a crowd counting method and system based on multi-scale fusion and distribution matching. Background Art
[0002] In recent years, with the continuous acceleration of urbanization, the agglomeration effect of urban population has become more and more obvious. The crowd density in public places such as public transportation, stadiums, and exhibition centers has become increasingly high. Timely acquisition and analysis of crowd information can play an important role in managing crowds and ensuring safety. Crowd counting refers to the method of using computer vision technology to monitor and count the number of people in cameras in real time in high-traffic areas. At the same time, good crowd counting methods can also be extended to other fields, such as counting cells or bacteria from microscopic images, estimating animal populations in wildlife reserves, and estimating the number of vehicles at transportation hubs or traffic jams.
[0003] Most of the early crowd counting methods were based on detection and regression. The detection-based method can estimate the number of people in a crowd relatively accurately in sparse crowd scenes, but it is not effective in dense crowd scenes. Compared with the detection-based method, the regression-based method has significantly improved the detection accuracy, but lacks the positioning information of the crowd. With the widespread application of deep learning technology in the field of computer vision, a large number of crowd counting methods based on deep learning have been proposed.
[0004] In the crowd counting method based on deep learning, when the neural network extracts features from the input image, most of the current mainstream crowd counting methods use a single last layer of features for prediction. The advantages are simple calculation and low requirements on the computer's computing power and memory size. However, because it only uses the semantic information of high-order features during prediction, and ignores the detailed information of low-order features and crowd density scale differences, it is not very effective in detecting small targets, so the accuracy of detecting dense crowds will be reduced. In addition, the problem of large differences in crowd density scales was ignored during training, resulting in low prediction accuracy. Summary of the invention
[0005] In view of this, the present invention proposes a crowd counting method and system based on multi-scale fusion and distribution matching, which are used to solve the problem of low accuracy of existing crowd counting methods based on deep learning.
[0006] In a first aspect, the present invention discloses a crowd counting method based on multi-scale fusion and distribution matching, the method comprising:
[0007] Obtain crowd images and preprocess them to build a data set;
[0008] A multi-scale branch prediction model is constructed with VGG as the backbone network, attention mechanism and multi-scale feature fusion module are introduced, and the multi-scale branch prediction model is pre-trained based on the dataset;
[0009] The crowd image to be tested is input into the pre-trained multi-scale branch prediction model to generate a density map and perform crowd counting.
[0010] Multi-scale branch prediction model Multi-scale branch prediction model Based on the above technical solution, preferably, the preprocessing includes: performing data enhancement and cropping and scaling processing on each batch of crowd images.
[0011] On the basis of the above technical solution, preferably, the multi-scale branch prediction model includes a front-end network and a back-end network;
[0012] The front-end network includes a VGG backbone network, a multi-scale feature fusion module and a feature enhancement module connected in sequence;
[0013] The VGG backbone network is used to extract features of different scales from the input crowd image;
[0014] The multi-scale feature fusion module is used to perform bottom-up feature fusion on the outputs of the VGG backbone network at different scales, and input the feature fusion results of different scales into the feature enhancement module respectively;
[0015] The feature enhancement module is used to encode information of the horizontal and vertical positions of each scale, and output a feature prediction map of the corresponding scale after a convolution operation;
[0016] The backend network is used to perform bottom-up feature fusion on the feature prediction map of the corresponding scale and output a density map.
[0017] On the basis of the above technical solution, preferably, the VGG backbone network includes an input layer, a first scale feature extraction module, a second scale feature extraction module, a third scale feature extraction module, a fourth scale feature extraction module and a fifth scale feature extraction module connected in sequence;
[0018] The first scale feature extraction module includes two convolutional layers with 64 channels and one pooling layer;
[0019] The second scale feature extraction module includes two convolutional layers with 128 channels and one pooling layer;
[0020] The third scale feature extraction module includes 4 convolution layers with 256 channels and one pooling layer;
[0021] The fourth scale feature extraction module includes 4 convolution layers with 512 channels and one pooling layer;
[0022] The fifth scale feature extraction module includes 4 convolutional layers with 512 channels.
[0023] On the basis of the above technical solution, preferably, the multi-scale feature fusion module includes a first scale feature fusion module, a second scale feature fusion module and a third scale feature fusion module connected in sequence;
[0024] The first scale feature fusion module is used to splice the output of the fifth scale feature extraction module after the convolution operation and upsampling processing are performed in sequence with the output of the third scale feature extraction module after the convolution operation, and output the spliced result to the second scale feature fusion module and the feature enhancement module after upsampling processing;
[0025] The second scale feature fusion module is used to splice the output of the second scale feature extraction module after the convolution operation with the output of the first scale feature fusion module, and output the spliced result to the third scale feature fusion module and the feature enhancement module after upsampling;
[0026] The third scale feature fusion module is used to splice the output of the first scale feature extraction module after the convolution operation with the output of the second scale feature fusion module, and output the spliced result to the feature enhancement module after upsampling;
[0027] The front-end network outputs a first-scale feature prediction map, a second-scale feature prediction map, a third-scale feature prediction map, and a fourth-scale feature prediction map, respectively.
[0028] On the basis of the above technical solution, preferably, the backend network includes a first backend fusion module, a second backend fusion module and a third backend fusion module connected in sequence;
[0029] The first back-end fusion module is used to splice the feature prediction map of the fourth scale with the feature prediction map of the third scale after the convolution attention unit and upsampling processing, and output the splicing result to the second back-end fusion module;
[0030] The convolutional attention unit includes a first branch unit, a second branch unit and a splicing unit, the first branch unit includes a convolution layer, the second branch unit includes a convolution layer and a CBAM attention module connected in sequence, and the splicing unit is used to output the splicing result of the first branch unit and the second branch unit;
[0031] The second back-end fusion module is used to splice the output result of the first back-end fusion module with the feature prediction map of the second scale after the convolution attention unit and upsampling processing, and output the splicing result to the third back-end fusion module;
[0032] The third back-end fusion module is used to splice the output result of the second back-end fusion module with the feature prediction map of the first scale after the convolution attention unit and upsampling processing, and output the density map after the splicing result is subjected to convolution operation.
[0033] On the basis of the above technical solution, preferably, the overall loss function of the multi-scale branch prediction model is:
[0034]
[0035] Among them, z, Represent the vectorized actual density map and predicted density map respectively, is the overall loss function, λ is the weight coefficient, To count the losses:
[0036]
[0037] is the optimal transmission loss:
[0038]
[0039] W(·) represents the similarity function between the predicted density map and the actual density map.
[0040] In a second aspect, the present invention discloses a crowd counting system based on multi-scale fusion and distribution matching, the system comprising:
[0041] Data acquisition module: used to acquire crowd images and perform preprocessing to establish a data set;
[0042] Model building module: used to build a multi-scale branch prediction model with VGG as the backbone network, introduce the attention mechanism and multi-scale feature fusion module, and perform pre-training based on the data set;
[0043] Model prediction module: used to input the crowd image to be tested into the pre-trained multi-scale branch prediction model, generate a density map and perform crowd counting.
[0044] In a third aspect of the present invention, an electronic device is disclosed, comprising: at least one processor, at least one memory, a communication interface and a bus;
[0045] Wherein, the processor, memory, and communication interface communicate with each other via the bus;
[0046] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method according to the first aspect of the present invention.
[0047] According to a fourth aspect of the present invention, a computer-readable storage medium is disclosed, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to implement the method described in the first aspect of the present invention.
[0048] Compared with the prior art, the present invention has the following beneficial effects:
[0049] 1) The present invention uses VGG as the backbone network, introduces the attention mechanism and the multi-scale feature fusion module to construct a multi-scale branch prediction model. The front-end network of the multi-scale branch prediction model can extract and fuse features of different scales to obtain high-level semantic feature maps of various scales. It can take into account the differences in the density scales of different crowds, and perform channel fusion on the feature maps of different scales output by the front-end to generate a density map, thereby improving the accuracy of density map prediction.
[0050] 2) The present invention uses the VGG backbone network to extract features of different scales on the input crowd image, and quotes the idea of the FPN feature image pyramid to design a bottom-up feature fusion channel that is horizontally linked to the top-down feature extraction channel of the original VGG backbone network, and improves the last layer of the VGG backbone network into a multi-scale fused branch feature prediction. The fused high-order features still have rich semantic information and are suitable for identifying and detecting most objects. The fused low-order features have not only detail information but also partial semantic information, which can more accurately identify and detect crowds, and can guide low-order features to identify and detect small targets, reducing the false detection rate.
[0051] 3) The present invention also adds a data enhancement module in the fusion process to encode the horizontal and vertical position information of each scale, increase the range of position information, and improve the ability to receive semantic information of each scale, thereby generating a multi-scale feature representation and outputting a multi-scale feature map.
[0052] 4) The present invention designs a convolutional attention unit in the back-end network, and combines channel attention and spatial attention to adjust the positions of different channels and spaces, thereby improving the ability to capture key information and aggregating context information to generate a density map, thereby improving performance while also achieving a lightweight effect.
[0053] 5) In the density map estimation of the present invention, the optimal transmission loss function of distribution matching and the counting loss function are used to calculate the overall loss function, so as to replace the original definition of training loss based on the pixel-by-pixel difference between the Gaussian smoothed density map and the predicted density map, and the optimal transmission loss is used to measure the similarity between the predicted density map and the true density map. The density map generated by the distribution matching and the semantically rich multi-scale fusion is combined to achieve a more accurate counting result under the dual effect of the two. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0055] Figure 1 It is a flow chart of the crowd counting method based on multi-scale fusion and distribution matching of the present invention;
[0056] Figure 2 is a structural diagram of a multi-scale branch prediction model of the present invention;
[0057] Figure 3 It is a schematic diagram of the back-end network structure of the present invention;
[0058] Figure 4 Schematic diagram of the structure of the convolutional attention unit of the present invention. DETAILED DESCRIPTION
[0059] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0060] See also Figure 1 The present invention proposes a crowd counting method based on multi-scale fusion and distribution matching, the method comprising:
[0061] S1. Obtain crowd images and preprocess them to establish a dataset.
[0062] The acquired crowd images are preprocessed in batches. Each batch of input image sets is subjected to data augmentation. Each RGB image is cropped and scaled. Each image is cropped to a size of 512*512 as the input of the backbone network. The input parameters are (10, 3, 512, 512), which represent batch size, number of channels, length, and width, respectively.
[0063] S2. Using VGG as the backbone network, introducing the attention mechanism and multi-scale feature fusion module to build a multi-scale branch prediction model, and pre-training based on the dataset.
[0064] For target groups that are far away in the image, on the one hand, we need high-resolution feature maps to pay more attention to small area information, and on the other hand, we need more global information to more accurately determine the existence and location of the crowd. However, it is difficult for existing crowd counting methods to take both aspects into account.
[0065] Therefore, the present invention proposes a network structure framework for multi-scale branch prediction. The front end of the method framework extracts features of different sizes and fuses features from bottom to top based on the VGG-19 network, and uses the fused multi-scale feature map for simultaneous prediction. The fused high-order features still have rich semantic information and are suitable for identifying and detecting most objects. The fused low-order features have not only detailed information but also partial semantic information, which can more accurately identify and detect people, and can guide low-order features to identify and detect small targets, reducing the false detection rate.
[0066] Specifically, Figure 2 As shown, the multi-scale branch prediction model of the present invention includes a front-end network and a back-end network;
[0067] The front-end network includes a VGG backbone network, a multi-scale feature fusion module and a feature enhancement module which are connected in sequence.
[0068] The VGG backbone network is used to extract features of different scales from the input crowd images.
[0069] Specifically, VGG-19 can be used as the VGG backbone network, and the first 16 layers of VGG-19 (excluding the last pooling layer) can be used for feature extraction. Figure 2 As shown on the left, the VGG backbone network is roughly divided into five modules from top to bottom according to the number of channels.
[0070] The VGG backbone network includes an input layer, a first scale feature extraction module, a second scale feature extraction module, a third scale feature extraction module, a fourth scale feature extraction module and a fifth scale feature extraction module which are connected in sequence.
[0071] The first-scale feature extraction module includes two convolutional layers (Conv3-64) with a size of 3*3 and a number of channels of 64, and one pooling layer;
[0072] The second-scale feature extraction module includes two convolutional layers (Conv3-128) with a size of 3*3 and a channel number of 128 and one pooling layer;
[0073] The third scale feature extraction module includes 4 convolutional layers (Conv3-256) with a size of 3*3 and a channel number of 256 and a pooling layer;
[0074] The fourth scale feature extraction module includes 4 convolutional layers (Conv3-512) with a size of 3*3 and a channel number of 512 and a pooling layer;
[0075] The fifth scale feature extraction module includes 4 convolutional layers (Conv3-512) with a size of 3*3 and a channel number of 512.
[0076] Among them, each convolutional layer is activated by the ReLU function, and the pooling layer (max poling) is used for pooling operation.
[0077] The multi-scale feature fusion module includes a corresponding bottom-up feature fusion channel for bottom-up feature fusion of the outputs of the VGG backbone network at different scales, and inputs the feature fusion results of different scales into the feature enhancement module respectively.
[0078] Specifically, the multi-scale feature fusion module includes a first scale feature fusion module, a second scale feature fusion module and a third scale feature fusion module which are connected in sequence.
[0079] The first scale feature fusion module is used to splice the output of the fifth scale feature extraction module after 1*1 convolution operation and 2 times upsampling processing and the output of the third scale feature extraction module after convolution operation, and output the spliced result to the second scale feature fusion module and the feature enhancement module after 2 times upsampling processing.
[0080] The second scale feature fusion module is used to splice the output result of the second scale feature extraction module after the 1*1 convolution operation with the output result of the first scale feature fusion module, and output the spliced result to the third scale feature fusion module and the feature enhancement module after 2 times upsampling.
[0081] The third scale feature fusion module is used to splice the output of the first scale feature extraction module after 1*1 convolution operation with the output of the second scale feature fusion module, and output the spliced result to the feature enhancement module after 2 times upsampling. The feature enhancement module includes a CA attention module and a convolution layer connected in sequence, which is used to encode the horizontal and vertical position information of each scale through the CA attention module, and output the feature prediction map of the corresponding scale after convolution operation of the convolution layer.
[0082] The present invention improves adaptability and generalization capabilities through a feature enhancement module, wherein the CA attention module encodes information on the horizontal and vertical positions of each scale, increases the range of position information, and improves the ability to receive semantic information at each scale. The 3*3 convolution kernel of the convolution layer performs a convolution operation on each fusion, the purpose is to get rid of the aliasing effect of upsampling. Finally, a total of four feature prediction maps of different sizes are output, namely, the feature prediction map Pre1 of the first scale, the feature prediction map Pre2 of the second scale, the feature prediction map Pre3 of the third scale, and the feature prediction map Pre4 of the fourth scale.
[0083] Figure 3 It is a schematic diagram of the back-end network structure of the present invention, wherein the back-end network is used to perform bottom-up feature fusion on feature prediction maps of corresponding scales and output a density map.
[0084] The backend network includes a first backend fusion module, a second backend fusion module and a third backend fusion module which are connected in sequence.
[0085] The first back-end fusion module is used to splice the feature prediction map of the fourth scale with the feature prediction map of the third scale after the convolution attention unit and upsampling processing, and output the splicing result to the second back-end fusion module.
[0086] The convolutional attention unit (Conv-CB unit) includes a first branch unit, a second branch unit and a splicing unit. Figure 4 The schematic diagram of the structure of the convolutional attention unit is shown in Figure 1. The first branch unit includes a 1*1 convolutional layer, the second branch unit includes a 3*3 convolutional layer with 64 input channels and 32 output channels and a CBAM attention module connected in sequence, and the splicing unit is used to output the splicing result of the first branch unit and the second branch unit. Through the convolutional attention unit, two feature maps of different sizes can be fused into a feature map with 32 channels in sequence. Through the designed Conv-CB unit, the introduction of channel attention and spatial attention modules, as well as multi-scale attention fusion, can effectively enhance the convolutional neural network's attention to important information in the feature map, thereby improving the lightweight performance and generalization ability of the model.
[0087] The second back-end fusion module is used to splice the output result of the first back-end fusion module with the feature prediction map of the second scale after the convolution attention unit and upsampling processing, and output the splicing result to the third back-end fusion module.
[0088] The third back-end fusion module is used to splice the output result of the second back-end fusion module with the feature prediction map of the first scale after the convolution attention unit and upsampling processing, and output the density map after the splicing result passes through a convolution layer with a size of 1*1, an input channel number of 128, and an output channel number of 1.
[0089] Existing density map estimation methods usually define training loss based on the pixel-by-pixel difference between the Gaussian smoothed density map and the predicted density map. A single kernel width is used to smooth the point annotations, but the density scales of the groups vary greatly, and a fixed Gaussian kernel cannot be used for fitting, resulting in low accuracy of the predicted Gaussian density map. Subsequently, an adaptive kernel width was used, which was selected based on the distance to the nearest neighbor of the annotation point, but the distribution of the crowd was very irregular, making it difficult to achieve the desired effect. In order to improve the accuracy of the counting, the present invention combines distribution matching with multi-scale fused density maps to perform crowd counting. The combination of optimal transmission loss and counting loss is used to replace the original definition of training loss based on the pixel-by-pixel difference between the Gaussian smoothed density map and the predicted density map, and the similarity between the normalized predicted density map and the normalized true density map is measured. The network input image for crowd counting and the mapping of the output density value are used to predict the number of people.
[0090] (1) The distance between the predicted density map and the actual density map is evaluated using the optimal transmission loss. The optimal transmission loss is defined as follows:
[0091]
[0092] Among them, W(·) represents the distance, μ is the set of pixels corresponding to the predicted density map The probability distribution of ν is the set of pixels corresponding to the actual density map. The probability distribution of α, β is the weight vector to be learned, <,> represents the inner product operation; c(x i ,y i ) is the cost function, indicating A point x on i Move to A point y on i The required cost is i,j=1,2,…,n.
[0093] (2) Order Vectorized binary image representing point annotations, true value, It is represented as the vectorized prediction density map returned by the neural network, that is, the predicted value. Taking ||*|| as the L1 norm of the vector, then ||z||, As the calculation of true value and predicted value, the goal of crowd counting is to make As close to ||z|| as possible. The counting loss used The definition is as follows:
[0094]
[0095] z and Divided by their respective total mass to convert into probability density function, the optimal transmission loss is obtained as follows:
[0096]
[0097] Among them, α * , β * is the corresponding weight vector.
[0098] (3) Finally, the overall loss function It is defined as the sum of two loss functions. Where λ is an adjustable hyperparameter.
[0099]
[0100] S3. Input the crowd image to be tested into the pre-trained multi-scale branch prediction model to generate a density map and perform crowd counting.
[0101] The multi-scale branch prediction model pre-trained in step S2 can be used for crowd counting. The crowd image to be measured is collected, and after pre-processing, it is input into the pre-trained multi-scale branch prediction model to obtain the crowd counting result.
[0102] On the basis of the above method embodiments, the present invention further proposes a crowd counting system based on multi-scale fusion and distribution matching, the system comprising:
[0103] Data acquisition module: used to acquire crowd images and perform preprocessing;
[0104] Model building module: used to build a multi-scale branch prediction model with VGG as the backbone network, introduce the attention mechanism and multi-scale feature fusion module, and perform pre-training;
[0105] Model prediction module: used to input the preprocessed crowd image into the pre-trained multi-scale branch prediction model, generate density map and perform crowd counting.
[0106] The above system embodiments and method embodiments correspond one to one, and for a brief description of the system embodiments, please refer to the method embodiments.
[0107] The present invention also discloses an electronic device, comprising: at least one processor, at least one memory, a communication interface and a bus; wherein the processor, memory, and communication interface communicate with each other through the bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the aforementioned method of the present invention.
[0108] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to implement all or part of the steps of the method described in the embodiment of the present invention. The storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk, and other media that can store program codes.
[0109] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be distributed to multiple network units. A person skilled in the art may select some or all of the modules according to actual needs to achieve the purpose of the solution of this embodiment without creative effort.
[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A crowd counting method based on multi-scale fusion and distribution matching, characterized in that: The method comprises: Obtain crowd images and preprocess them to build a data set; A multi-scale branch prediction model is constructed with VGG as the backbone network, attention mechanism and multi-scale feature fusion module are introduced, and the multi-scale branch prediction model is pre-trained based on the dataset; Input the crowd image to be tested into the pre-trained multi-scale branch prediction model to generate a density map and perform crowd counting; The preprocessing includes: performing data enhancement and cropping and scaling processing on each batch of crowd images; The multi-scale branch prediction model includes a front-end network and a back-end network; The front-end network includes a VGG backbone network, a multi-scale feature fusion module and a feature enhancement module connected in sequence; The VGG backbone network is used to extract features of different scales from the input crowd image; The multi-scale feature fusion module is used to perform bottom-up feature fusion on the outputs of the VGG backbone network at different scales, and input the feature fusion results of different scales into the feature enhancement module respectively; The feature enhancement module includes a CA attention module and a convolution layer connected in sequence, and is used to encode information of the horizontal and vertical positions of each scale through the CA attention module, and output a feature prediction map of the corresponding scale after a convolution operation; The backend network is used to perform bottom-up feature fusion on the feature prediction map of the corresponding scale and output a density map; The VGG backbone network includes an input layer, a first scale feature extraction module, a second scale feature extraction module, a third scale feature extraction module, a fourth scale feature extraction module and a fifth scale feature extraction module connected in sequence; The first scale feature extraction module includes two convolutional layers with 64 channels and one pooling layer; The second scale feature extraction module includes two convolutional layers with 128 channels and one pooling layer; The third scale feature extraction module includes 4 convolution layers with 256 channels and one pooling layer; The fourth scale feature extraction module includes 4 convolution layers with 512 channels and one pooling layer; The fifth scale feature extraction module includes 4 convolutional layers with a channel number of 512; The multi-scale feature fusion module includes a first scale feature fusion module, a second scale feature fusion module and a third scale feature fusion module connected in sequence; The first scale feature fusion module is used to splice the output of the fifth scale feature extraction module after the convolution operation and upsampling processing are performed in sequence with the output of the third scale feature extraction module after the convolution operation, and output the spliced result to the second scale feature fusion module and the feature enhancement module after upsampling processing; The second scale feature fusion module is used to splice the output of the second scale feature extraction module after the convolution operation with the output of the first scale feature fusion module, and output the spliced result to the third scale feature fusion module and the feature enhancement module after upsampling; The third scale feature fusion module is used to splice the output of the first scale feature extraction module after the convolution operation with the output of the second scale feature fusion module, and output the spliced result to the feature enhancement module after upsampling; The front-end network outputs a feature prediction map of a first scale, a feature prediction map of a second scale, a feature prediction map of a third scale, and a feature prediction map of a fourth scale respectively; The backend network includes a first backend fusion module, a second backend fusion module and a third backend fusion module connected in sequence; The first back-end fusion module is used to splice the feature prediction map of the fourth scale with the feature prediction map of the third scale after the convolution attention unit and upsampling processing, and output the splicing result to the second back-end fusion module; The convolutional attention unit includes a first branch unit, a second branch unit and a splicing unit, the first branch unit includes a convolution layer, the second branch unit includes a convolution layer and a CBAM attention module connected in sequence, and the splicing unit is used to output the splicing result of the first branch unit and the second branch unit; The second back-end fusion module is used to splice the output result of the first back-end fusion module with the feature prediction map of the second scale after the convolution attention unit and upsampling processing, and output the splicing result to the third back-end fusion module; The third back-end fusion module is used to splice the output result of the second back-end fusion module with the feature prediction map of the first scale after the convolution attention unit and upsampling processing, and output a density map after the splicing result is subjected to a convolution operation; The overall loss function of the multi-scale branch prediction model is: Among them, z, Represent the vectorized actual density map and predicted density map respectively, is the overall loss function, λ is the weight coefficient, To count the losses: is the optimal transmission loss: W(·) represents the similarity function between the predicted density map and the actual density map.
2. A crowd counting system based on multi-scale fusion and distribution matching, which is implemented using the crowd counting method based on multi-scale fusion and distribution matching according to claim 1, characterized in that: The system comprises: Data acquisition module: used to acquire crowd images and perform preprocessing to establish a data set; Model building module: used to build a multi-scale branch prediction model with VGG as the backbone network, introduce the attention mechanism and multi-scale feature fusion module, and pre-train the multi-scale branch prediction model based on the data set; Model prediction module: used to input the crowd image to be tested into the pre-trained multi-scale branch prediction model, generate a density map and perform crowd counting.
3. An electronic device, characterized in that: include: at least one processor, at least one memory, a communication interface, and a bus; Wherein, the processor, memory, and communication interface communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method according to claim 1.
4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions cause a computer to implement the method of claim 1.
Citation Information
Patent Citations
Crowd counting method and device, electronic equipment and computer storage medium
CN112232140A
Crowd counting method based on multi-scale feature fusion
CN112597985A
Method for detecting and counting distribution of dense crowds in video
CN114005085A
Crowd counting method in combination with learnable attention mechanism
CN116453048A
Crowd counting method based on trans-scale pyramid Transform
CN116740439A
