Crowd counting method based on attention feature fusion and multi-column feature enhancement
By adopting the method of attention feature fusion and multi-column feature enhancement in the population counting technology, combining probability map generation and density map regression network, the problem of counting error in complex and dense scenarios is solved, the counting accuracy and efficiency are improved, and the counting accuracy and efficiency are reduced. Calculate the cost.
Patent Information
- Application Number
- CN202411125494.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-16
AI Technical Summary
The existing crowd counting technology has errors and limitations in complex and dense scenarios, especially in scenarios where there are many occlusions. Single-stage networks are difficult to alleviate the problems of overlapping crowd targets and changing scales, and the calculation cost of multiple column models is high, and the attention mechanism is It is difficult to unify the semantic information of feature maps at different levels in feature fusion.
Using a population counting method based on attention feature fusion and multi-column feature enhancement, a probability map generation network and density map regression network are combined, and a dual attention feature fusion network and multi-column feature enhancement network are used to adaptively combine channels and space. Attention, perform multi-scale feature fusion, generate density maps and perform integral operations to improve counting accuracy.
It improves the accuracy and efficiency of population counting, can better deal with occlusion and scale changes in complex and dense scenarios, reduces calculation costs, and enhances the counting ability of dense areas.
Smart Images

Figure CN118644821B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a crowd counting method based on attention feature fusion and multi-column feature enhancement. Background Art
[0002] In crowd counting tasks, counting accuracy directly affects the credibility of counting results and application effects. Although crowd counting technology has made some progress, it still has certain errors and limitations when facing complex crowd-dense scenes, especially scenes with many occluded targets. With the strengthening of population mobility, dense crowd scenes will become more common, and the detection of overly dense crowd targets is a challenge that cannot be ignored in crowd counting tasks.
[0003] Most of the current research in the field of crowd counting is based on the regression of density maps using a single-stage network. However, it is difficult to fundamentally alleviate the problems of crowd target overlap and occlusion using only a single-stage network model. In past studies, in order to alleviate the problem of scale change, some studies proposed multi-column architectures such as MCNN and DecidedNet. Although multi-column models can alleviate the problem of scale change to a certain extent, their counting performance is limited by the number of columns. In addition, multi-column models also greatly increase the computational cost due to multiple branches of parallel training. In order to deal with the problems of complex background and non-uniform distribution, some studies use attention mechanisms to detect changes in crowd density in images, such as ADCrowdNet, Attend to Count, etc. However, when the attention mechanism is directly applied to the feature fusion structure, it is difficult to unify the semantic information of feature maps at different levels. Therefore, it is difficult to combine contextual information in the subsequent feature fusion and density map generation process, resulting in low crowd counting accuracy. Summary of the invention
[0004] Based on this, it is necessary to provide a crowd counting method based on attention feature fusion and multi-column feature enhancement that can improve the accuracy of crowd counting in response to the above technical problems.
[0005] A crowd counting method based on attention feature fusion and multi-column feature enhancement, the method comprising:
[0006] Obtaining images of people to be identified;
[0007] Inputting the crowd image to be identified into the probability map generation network module of the trained crowd counting network model for processing, and outputting a probability map;
[0008] Inputting the crowd image to be identified and the probability map into the density map regression network module of the trained crowd counting network model for processing, and outputting a density map, wherein the density map regression network module includes a multi-column feature enhancement network and a dual-attention feature fusion network;
[0009] An integration operation is performed on the density map to obtain the crowd count in the to-be-identified crowd image.
[0010] In one embodiment, the probability map generation network module includes an encoder composed of the first 13 convolutional layers and the first 4 maximum pooling layers of the VGG-16 network, a first upsampling, a first splicing operation, a second splicing operation, a third splicing operation, a fourth splicing operation, a first feature adjustment unit, a second feature adjustment unit, a third feature adjustment unit, and a fourth feature adjustment unit;
[0011] The step of inputting the crowd image to be identified into a probability map generation network module of a trained crowd counting network model for processing and outputting a probability map comprises:
[0012] The image of the crowd to be identified is input into the encoder for feature extraction to obtain five feature maps of different scales, which are, in descending order of scale, a first feature map, a second feature map, a third feature map, a fourth feature map, and a fifth feature map;
[0013] After the fifth feature map is input into the first upsampling and bilinear interpolation upsampling is performed, the first splicing operation is used to splice it with the fourth feature map to obtain a first splicing feature;
[0014] After the first splicing feature is input into the first feature adjustment unit for feature adjustment, the second splicing operation is used to splice the first splicing feature with the third feature map to obtain a second splicing feature;
[0015] After the second splicing feature is input into the second feature adjustment unit for feature adjustment, the second splicing feature is spliced with the second feature map using the third splicing operation to obtain a third splicing feature;
[0016] After the third splicing feature is input into the third feature adjustment unit for feature adjustment, the third splicing feature is spliced with the first feature map using the fourth splicing operation to obtain a fourth splicing feature;
[0017] The fourth splicing feature is input into the fourth feature adjustment unit for feature adjustment to obtain a probability map.
[0018] In one of the embodiments, the density map regression network module includes: a feature extraction unit consisting of the first 13 convolutional layers and the first 5 maximum pooling layers of the VGG-16 network, a first multi-column feature enhancement unit, a second multi-column feature enhancement unit, a first dual attention feature fusion unit, a second dual attention feature fusion unit, and a density map generation unit;
[0019] The step of inputting the to-be-identified crowd image and the probability map into the density map regression network module of the trained crowd counting network model for processing and outputting the density map comprises:
[0020] Inputting the to-be-identified crowd image and the probability map into the feature extraction unit for feature extraction, obtaining a sixth feature map output by the 10th convolutional layer of the feature extraction unit, a seventh feature map output by the 13th convolutional layer of the feature extraction unit, and an eighth feature map output by the 5th maximum pooling layer of the feature extraction unit;
[0021] The eighth feature map is input into the first multi-column feature enhancement unit for feature enhancement to obtain a first enhanced feature map;
[0022] Inputting the first enhanced feature map and the seventh feature map into the first dual attention feature fusion unit for feature fusion to obtain a first fused feature map;
[0023] Inputting the first fused feature map into a second multi-column feature enhancement unit for feature enhancement to obtain a second enhanced feature map;
[0024] Inputting the second enhanced feature map and the sixth feature map into the second dual attention feature fusion unit for feature fusion to obtain a second fused feature map;
[0025] The second fused feature map is input into the density map generating unit to generate a density map to obtain a density map.
[0026] In one embodiment, the first multi-column feature enhancement unit and the second multi-column feature enhancement unit are multi-column feature enhancement networks with the same structure, and the multi-column feature enhancement network includes a multi-column enhancement structure and an attention structure;
[0027] The feature maps of the multi-column feature enhancement network are respectively input into the multi-column enhancement structure and the attention structure, the attention weight is obtained through the attention structure, and the enhanced feature is obtained through the multi-column enhancement structure, and then the attention weight and the enhanced feature are dot-multiplied to output the enhanced feature map.
[0028] In one embodiment, the multi-column enhancement structure includes a first convolution, a second convolution, a third convolution, a fourth convolution, a fifth convolution, a sixth convolution, a seventh convolution and an eighth convolution;
[0029] Input the feature map of the multi-column enhanced structure, respectively input the first convolution, the second convolution, the third convolution and the fourth convolution processing channel number, and obtain the first processed feature, the second processed feature, the third processed feature and the fourth processed feature accordingly;
[0030] The second processed feature is input into the fifth convolution for processing to obtain the fifth processed feature;
[0031] The third processed feature is added to the fifth processed feature and then subjected to a sixth convolution process to obtain a sixth processed feature;
[0032] The fourth processed feature is added to the sixth processed feature and then subjected to a seventh convolution process to obtain a seventh processed feature;
[0033] The first processed feature, the fifth processed feature, the sixth processed feature and the seventh processed feature are concatenated and then subjected to an eighth convolution process to output enhanced features.
[0034] In one embodiment, the attention structure includes: a ninth convolution, a first Relu activation function, a tenth convolution, a first global average pooling, an eleventh convolution, a second Relu activation function, a twelfth convolution and a first Sigmoid function;
[0035] The feature map of the input attention structure is processed by the ninth convolution, the first Relu activation function and the tenth convolution of the first branch, and the first global average pooling, the eleventh convolution, the second Relu activation function and the twelfth convolution of the second branch respectively. The outputs of the first branch and the second branch are added and activated by the first Sigmoid function to obtain the attention weight.
[0036] In one embodiment, the first dual attention feature fusion unit and the second dual attention feature fusion unit are dual attention feature fusion networks with the same structure, and the dual attention feature fusion network includes: a thirteenth convolution, a fourteenth convolution, a third Relu activation function, a fifteenth convolution, a second global average pooling, a sixteenth convolution, a fourth Relu activation function, a seventeenth convolution, a second Sigmoid function, average pooling, maximum pooling, an eighteenth convolution and a third Sigmoid function;
[0037] After upsampling, the deep enhanced feature map input into the dual attention feature fusion network is concatenated with the shallow feature map input into the dual attention feature fusion network and then input into the thirteenth convolution for processing to obtain the first feature;
[0038] The first feature is operated sequentially through the fourteenth convolution, the third ReLU activation function and the fifteenth convolution, and a first operation result is output;
[0039] The first feature is operated sequentially through the second global average pooling, the sixteenth convolution, the fourth ReLU activation function and the seventeenth convolution, and a second operation result is output;
[0040] The first operation result and the second operation result are added together and activated using the second Sigmoid function to obtain a weight between 0 and 1. w ;
[0041] The weight w Multiplying the deep enhancement feature map by the deep enhancement feature map to obtain a processed deep feature map;
[0042] The weight (1- w ) is multiplied by the shallow feature map to obtain a processed shallow feature map;
[0043] Add the processed deep feature map and the processed shallow feature map to obtain a preliminary fusion feature map;
[0044] The shallow feature map is spliced with the results of the average pooling and the maximum pooling operations respectively, and then input into the eighteenth convolution process and activated by the third Sigmoid function. The activated output is multiplied by the preliminary fusion feature map as a weight to obtain a fusion feature map.
[0045] In one of the embodiments, the density map generation unit includes: 2 3×3 convolutions and 1 1×1 convolution;
[0046] The second fused feature map is subjected to two 3×3 convolutions to reduce the channels of the fused feature map, and then a 1×1 convolution is performed to generate a density map output.
[0047] In one embodiment, the crowd counting network model is trained as follows:
[0048] Acquire a crowd image data sample set, wherein the crowd image data samples in the crowd image data sample set include crowd image data and corresponding real crowd probability maps and real crowd density maps;
[0049] Dividing the crowd image data sample set into a first training set, a validation set and a test set;
[0050] Analyze the local count value of each 32×32 block in each crowd image data in the first training set, and generate an artificial synthetic map of the dense area and a corresponding real crowd probability map and real crowd density map as artificial synthetic dense data samples based on the blocks whose local count values exceed a preset value;
[0051] Adding the artificially synthesized dense data sample to the first training set to obtain a second training set;
[0052] The probability map generation network module of the crowd counting network model is trained using the first training set and the validation set, and the density map regression network module of the crowd counting network model is trained using the second training set and the validation set, a triple counting loss function is used to calculate the gradient during the training process, and an Adam optimizer is used to update and optimize the network parameters of the probability map generation network module and the density map regression network module of the crowd counting network model until the training termination condition is met, thereby obtaining an optimal crowd counting network model;
[0053] The test set is used to input the optimal crowd counting network model for testing. If the test result meets the accuracy requirement, a trained crowd counting network model is obtained.
[0054] In one embodiment, the triple counting loss function L for:
[0055] ,
[0056] in,
[0057] ,
[0058] in, L pmg The loss of the probability map generation network module, is the number of crowd image data samples in the first training set, is the input crowd image data, is the network weight, Express The predicted probability plot of Represents the true population probability map;
[0059] ,
[0060] in, L dmr is the loss of the density map regression network module, M is the number of crowd image data samples in the second training set, are the network weights that are not shared with the probabilistic graph generation network module, Express The predicted density map of is the real population density map;
[0061] ,
[0062] ,
[0063] in, L cis the loss between the final counting result and the actual counting, express The difference between the predicted density map and the actual crowd density map.
[0064] The crowd counting method based on attention feature fusion and multi-column feature enhancement obtains a crowd image to be identified, inputs the crowd image to be identified into the probability map generation network module of the trained crowd counting network model for processing, outputs a probability map, and then inputs the crowd image to be identified and the probability map into the density map regression network module of the trained crowd counting network model for processing, outputs a density map, and the density map regression network module includes a multi-column feature enhancement network and a dual-attention feature fusion network; then the density map is integrated to obtain the crowd count in the crowd image to be identified. Thus, in the probability map generation network module, the probability of each point as a counting target is calculated pixel by pixel by generating a probability map; in the density map regression network module, the multi-column feature enhancement network combines the information of receptive fields of different sizes, and uses the dual-attention feature fusion network for attention-guided feature enhancement, and adaptively combines channel and spatial attention to complete the fusion of multi-scale features, thereby improving the counting accuracy of crowd images. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 It is a flowchart of a crowd counting method based on attention feature fusion and multi-column feature enhancement in one embodiment;
[0066] Figure 2 A schematic diagram of a network structure of a crowd counting network model in one embodiment;
[0067] Figure 3 A schematic diagram of a network structure of a probability graph generation network module in one embodiment;
[0068] Figure 4 A schematic diagram of a network structure of a density map regression network module in one embodiment;
[0069] Figure 5 A schematic diagram of a network structure of a multi-column feature enhancement network in one embodiment;
[0070] Figure 6 Schematic diagram of the network structure of a dual attention feature fusion network in one embodiment;
[0071] Figure 7 FIG. 4 is a schematic diagram of a network structure of a density map generating unit in an embodiment. DETAILED DESCRIPTION
[0072] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0073] In one embodiment, Figure 1 and Figure 2 As shown, a crowd counting method based on attention feature fusion and multi-column feature enhancement is provided, including the following steps:
[0074] Step S220, obtaining an image of a crowd to be identified.
[0075] Step S240: input the crowd image to be identified into the probability map generation network module of the trained crowd counting network model for processing, and output a probability map.
[0076] Among them, the crowd counting network model is a two-stage crowd counting network based on attention feature fusion and multi-column feature enhancement. The first stage of the crowd counting network is a probability map generation network, and the second stage is a density map regression network.
[0077] In one embodiment, the probability map generation network module includes an encoder composed of the first 13 convolutional layers and the first 4 maximum pooling layers of the VGG-16 network, a first upsampling, a first splicing operation, a second splicing operation, a third splicing operation, a fourth splicing operation, a first feature adjustment unit, a second feature adjustment unit, a third feature adjustment unit, and a fourth feature adjustment unit;
[0078] The crowd image to be identified is input into the probability map generation network module of the trained crowd counting network model for processing, and the probability map is output, including:
[0079] The image of the crowd to be identified is input into the encoder for feature extraction to obtain five feature maps of different scales, which are, in descending order of scale: the first feature map, the second feature map, the third feature map, the fourth feature map and the fifth feature map; the fifth feature map is input into the first upsampling for bilinear interpolation upsampling, and then is spliced with the fourth feature map by a first splicing operation to obtain a first splicing feature; the first splicing feature is input into the first feature adjustment unit for feature adjustment, and then is spliced with the third feature map by a second splicing operation to obtain a second splicing feature; the second splicing feature is input into the second feature adjustment unit for feature adjustment, and then is spliced with the second feature map by a third splicing operation to obtain a third splicing feature; the third splicing feature is input into the third feature adjustment unit for feature adjustment, and then is spliced with the first feature map by a fourth splicing operation to obtain a fourth splicing feature; the fourth splicing feature is input into the fourth feature adjustment unit for feature adjustment to obtain a probability map.
[0080] In one embodiment, the network structure of the probability graph generation network module is as follows: Figure 3 As shown in the figure, the encoder-decoder structure is adopted, which maps the input image (i.e., the image of the crowd to be identified) into a probability map. It uses the first 13 convolutional layers and the first 4 maximum pooling layers of the VGG-16 network as encoders to extract features. Among them, the first 13 convolutional layers of VGG-16 are all 3×3 convolutions, and the number of channels of the convolutional layers is 64, 64, 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, and 512, respectively; after the second convolutional layer, the fourth convolutional layer, the seventh convolutional layer, and the tenth convolutional layer, there is a maximum pooling layer with a stride of 2 and a pooling area size of 2, which are the first maximum pooling layer, the second maximum pooling layer, The third and fourth maximum pooling layers; the feature map output by the 13th convolutional layer (i.e., the fifth feature map) is upsampled (i.e., the first upsampling) for bilinear interpolation upsampling, and is spliced with the output feature map of the 10th convolutional layer (i.e., the fourth feature map) of the same size in the feature extraction stage (i.e., the first splicing operation) to obtain the first splicing feature, and then the decoder 1 composed of two 3×3 convolutions is used to adjust its channel, and then the feature map output by the decoder 1 is upsampled (wherein the upsampling and the decoder 1 constitute the first feature adjustment unit) for bilinear interpolation upsampling, and The output feature map of the 7th convolutional layer of the same size as that of the feature extraction stage (i.e., the third feature map) is concatenated (i.e., the second concatenation operation) to obtain the second concatenated feature, and then the decoder 2 composed of two 3×3 convolutions is used to adjust its channel, and then the feature map output by the decoder 2 is upsampled (wherein the upsampling and the decoder 2 constitute the second feature adjustment unit) for bilinear interpolation upsampling, and is concatenated (i.e., the third concatenation operation) with the output feature map of the 4th convolutional layer of the same size as that of the feature extraction stage (i.e., the second feature map), and then the decoder 3 composed of two 3×3 convolutions is used. Adjust its channels, then perform bilinear interpolation upsampling on the feature map output by decoder 3 by upsampling (wherein the upsampling and decoder 3 constitute the third feature adjustment unit), and concatenate it with the output feature map of the second convolutional layer (i.e., the first feature map) of the same size in the feature extraction stage (i.e., the fourth concatenation operation), and then use decoder 4 composed of two 3×3 convolutions to adjust its channels, and its output feature map has the same size as the input image. Finally, use 3×3 convolution (wherein the 3×3 convolution and decoder 4 constitute the fourth feature adjustment unit) to adjust the number of channels to 1, and obtain the generated probability map P.
[0081] Step S260, input the crowd image to be identified and the probability map into the density map regression network module of the trained crowd counting network model for processing, and output the density map. The density map regression network module includes a multi-column feature enhancement network and a dual attention feature fusion network.
[0082] Among them, a multi-column feature enhancement network is used to combine information of receptive fields of different sizes, so that the enhanced features can obtain information of different scales to better cope with the problem of scale change; a dual-attention feature fusion network is used to enable the crowd counting network model to combine features at different levels, better distinguish background and foreground information, pay more attention to dense areas in the image to alleviate the problem of uneven target distribution, and enhance the crowd counting network model's ability to count dense crowds in dense areas.
[0083] In one embodiment, Figure 4 As shown, the density map regression network module includes: a feature extraction unit consisting of the first 13 convolutional layers and the first 5 maximum pooling layers of the VGG-16 network, a first multi-column feature enhancement unit, a second multi-column feature enhancement unit, a first dual-attention feature fusion unit, a second dual-attention feature fusion unit, and a density map generation unit;
[0084] The crowd image to be identified and the probability map are input into the density map regression network module of the trained crowd counting network model for processing, and the density map is output, including:
[0085] The image of the crowd to be identified and the probability map are input into the feature extraction unit for feature extraction, and the sixth feature map output by the 10th convolution layer of the feature extraction unit, the seventh feature map output by the 13th convolution layer of the feature extraction unit, and the eighth feature map output by the 5th maximum pooling layer of the feature extraction unit are obtained; the eighth feature map is input into the first multi-column feature enhancement unit for feature enhancement to obtain a first enhanced feature map; the first enhanced feature map and the seventh feature map are input into the first dual-attention feature fusion unit for feature fusion to obtain a first fused feature map; the first fused feature map is input into the second multi-column feature enhancement unit for feature enhancement to obtain a second enhanced feature map; the second enhanced feature map and the sixth feature map are input into the second dual-attention feature fusion unit for feature fusion to obtain a second fused feature map; the second fused feature map is input into the density map generation unit for density map generation to obtain a density map.
[0086] In one embodiment, Figure 5 As shown, the first multi-column feature enhancement unit and the second multi-column feature enhancement unit are multi-column feature enhancement networks with the same structure, and the multi-column feature enhancement network includes a multi-column enhancement structure and an attention structure; the feature map input into the multi-column feature enhancement network is respectively input into the multi-column enhancement structure and the attention structure, and the attention weight is obtained through the attention structure, and the enhanced feature is obtained through the multi-column enhancement structure, and then the attention weight and the enhanced feature are point-multiplied to output the enhanced feature map.
[0087] In one embodiment, a multi-column enhancement structure includes a first convolution, a second convolution, a third convolution, a fourth convolution, a fifth convolution, a sixth convolution, a seventh convolution and an eighth convolution; the feature map of the multi-column enhancement structure is input into the first convolution, the second convolution, the third convolution and the fourth convolution processing channel numbers respectively, and the first processed feature, the second processed feature, the third processed feature and the fourth processed feature are obtained correspondingly; the second processed feature is input into the fifth convolution for processing to obtain the fifth processed feature; the third processed feature is added to the fifth processed feature and then processed by the sixth convolution to obtain the sixth processed feature; the fourth processed feature is added to the sixth processed feature and then processed by the seventh convolution to obtain the seventh processed feature; the first processed feature, the fifth processed feature, the sixth processed feature and the seventh processed feature are concatenated and then processed by the eighth convolution to output the enhanced feature.
[0088] In one embodiment, Figure 5 As shown in the figure, there are four branches in the multi-column enhancement structure. First, the input feature is processed by 1×1 Conv (i.e., the first convolution, the second convolution, the third convolution, and the fourth convolution) in each branch to reduce the number of channels, so that the number of channels of each branch is 1 / 4 of the input feature. Then, branch 1 does not perform additional processing, branch 2 passes through a 3×3 Conv with a hole rate r of 1 (i.e., the fifth convolution), branch 3 first adds the features after reducing the channel to the output features of branch 2, and then passes through a 3×3 Conv with a hole rate r of 2 (i.e., the sixth convolution), branch 4 adds the features after reducing the channel to the output features of branch 3, and then passes through a 3×3 Conv with a hole rate r of 3 (i.e., the seventh convolution), then the results of the four branches are concatenated and passed through a 1×1 Conv (i.e., the eighth convolution), and then the output features (i.e., enhanced features) are output.
[0089] In one embodiment, the attention structure includes: a ninth convolution, a first Relu activation function, a tenth convolution, a first global average pooling, an eleventh convolution, a second Relu activation function, a twelfth convolution and a first Sigmoid function; the feature map of the input attention structure is processed by the ninth convolution, the first Relu activation function and the tenth convolution of the first branch, and the first global average pooling, the eleventh convolution, the second Relu activation function and the twelfth convolution of the second branch respectively, and the outputs of the first branch and the second branch are added and activated by the first Sigmoid function to obtain the attention weight.
[0090] In one embodiment, Figure 5As shown in the figure, in the attention structure, the first branch, i.e., the input features, are processed in sequence by 1×1 Conv (i.e., the ninth convolution), Relu activation function (i.e., the first Relu activation function), and 1×1 Conv (i.e., the tenth convolution). The second branch performs global average pooling on the input features (i.e., global average pooling is performed using the first global average pooling), and the pooled results are passed through 1×1 Conv (i.e., the eleventh convolution), Relu activation function (i.e., the second Relu activation function), and 1×1 Conv (i.e., the twelfth convolution). The results of the two branches are added together and activated through the Sigmoid function (i.e., the first Sigmoid function) to obtain the attention weight.
[0091] In one embodiment, the first dual attention feature fusion unit and the second dual attention feature fusion unit are dual attention feature fusion networks with the same structure, and the dual attention feature fusion network includes: a thirteenth convolution, a fourteenth convolution, a third Relu activation function, a fifteenth convolution, a second global average pooling, a sixteenth convolution, a fourth Relu activation function, a seventeenth convolution, a second Sigmoid function, average pooling, maximum pooling, an eighteenth convolution and a third Sigmoid function;
[0092] After upsampling the deep enhanced feature map input to the dual attention feature fusion network, it is spliced with the shallow feature map input to the dual attention feature fusion network and then input into the thirteenth convolution for processing to obtain the first feature; the first feature is operated through the fourteenth convolution, the third Relu activation function and the fifteenth convolution in sequence, and the first operation result is output; the first feature is operated through the second global average pooling, the sixteenth convolution, the fourth Relu activation function and the seventeenth convolution in sequence, and the second operation result is output; the first operation result and the second operation result are added and activated using the second Sigmoid function to obtain a weight between 0 and 1. w ; The weight w Multiply it with the deep enhancement feature map to obtain the processed deep feature map; w ) is multiplied with the shallow feature map to obtain a processed shallow feature map; the processed deep feature map is added to the processed shallow feature map to obtain a preliminary fused feature map; the shallow feature map is spliced with the results of average pooling and maximum pooling respectively, and then input into the eighteenth convolution and activated by the third Sigmoid function. The activated output is used as the weight to multiply the preliminary fused feature map to obtain a fused feature map.
[0093] In one embodiment, the network structure of the dual attention feature fusion network is as follows: Figure 6As shown, first, the deep enhanced feature map (wherein, in the first dual attention feature fusion unit, the deep enhanced feature map is the first enhanced feature map, and in the second dual attention feature fusion unit, the deep enhanced feature map is the second enhanced feature map) is bilinearly interpolated and upsampled and concatenated with the shallow feature map (wherein, in the first dual attention feature fusion unit, the shallow feature map is the seventh feature map, and in the second dual attention feature fusion unit, the shallow feature map is the sixth feature map), and then the number of channels is adjusted to 512 through a 1×1 Conv (i.e., the thirteenth convolution) to obtain the first feature, and then the following two operations are performed on the first feature: the first operation is to directly perform 1×1 Conv (i.e., the fourteenth convolution), Relu activation (i.e., the third Relu activation function), and 1×1 Conv (i.e., the fifteenth convolution) operations on the first feature in sequence, and output the first operation result; the second operation is to first perform global average pooling (i.e., the second global average pooling) on the first feature, and then use 1×1 Conv (the sixteenth convolution), Relu activation function (the fourth Relu activation function), 1×1 Conv (the seventeenth convolution) are operated to output the second operation result; then the results of these two operations (i.e. the first operation result and the second operation result) are added, and then Sigmoid activation (i.e. the second Sigmoid function) is used to obtain a weight between 0 and 1 after activation. w ,Will w Multiply it with the deep enhanced feature map to increase the weight (1- w ) is multiplied with the shallow feature map, and then the processed deep feature map is added to the processed shallow feature map to obtain a preliminary fused feature map combined with channel attention; then the spatial attention is calculated for the input shallow feature map: first, the shallow feature map is average pooled and the maximum pooled are respectively performed, and the results are concatenated and passed through a 7×7 Conv (i.e., the eighteenth convolution), and then Sigmoid activation (i.e., the third Sigmoid function) is used, and the output is used as the weight to multiply the preliminary fused feature map combined with channel attention to obtain a fused feature map.
[0094] In one embodiment, Figure 7 As shown in the figure, the density map generation unit includes: 2 3×3 convolutions and 1 1×1 convolution; the second fused feature map is subjected to 2 3×3 convolutions to reduce the channels of the fused feature map, and then a 1×1 convolution is performed to generate a density map output.
[0095] The second fused feature map is input into the density map generation unit, and a crowd density map of 1 / 8 the size of the image of the crowd to be identified is output, and then the crowd density map is integrated to obtain the number of people.
[0096] Among them, two 3×3 convolutions with gradually decreasing output channel numbers are used to reduce the number of channels of the fused feature map, and finally a 1×1 convolution with a channel number of 1 is used to generate the density map.
[0097] Step S280, performing an integration operation on the density map to obtain the crowd count in the crowd image to be identified.
[0098] The crowd counting method based on attention feature fusion and multi-column feature enhancement obtains the crowd image to be identified, inputs the crowd image to be identified into the probability map generation network module of the trained crowd counting network model for processing, outputs the probability map, and then inputs the crowd image to be identified and the probability map into the density map regression network module of the trained crowd counting network model for processing, outputs the density map, and the density map regression network module includes a multi-column feature enhancement network and a dual-attention feature fusion network; then the density map is integrated to obtain the crowd count in the crowd image to be identified. Therefore, in the probability map generation network module, the probability of each point as a counting target is calculated pixel by pixel by generating a probability map; in the density map regression network module, the multi-column feature enhancement network combines the information of receptive fields of different sizes, and uses the dual-attention feature fusion network for attention-guided feature enhancement, and adaptively combines the channel and spatial attention to complete the fusion of multi-scale features, thereby improving the counting accuracy of the crowd image.
[0099] In one embodiment, the crowd counting network model is trained as follows:
[0100] A crowd image data sample set is obtained, wherein the crowd image data samples in the crowd image data sample set include crowd image data and corresponding real crowd probability maps and real crowd density maps; the crowd image data sample set is divided into a first training set, a validation set and a test set; the local count value of each 32×32 block in each crowd image data in the first training set is analyzed, and based on the blocks whose local count values exceed a preset value, an artificial synthetic map of a dense area and a corresponding real crowd probability map and real crowd density map are generated as artificially synthesized dense data samples; the artificially synthesized dense data samples are added to the first training set to obtain a second training set; the probability map generation network module of the crowd counting network model is trained using the first training set and the validation set, and the density map regression network module of the crowd counting network model is trained using the second training set and the validation set, and the gradient is calculated using a triple counting loss function during the training process, and the network parameters of the probability map generation network module and the density map regression network module of the crowd counting network model are updated and optimized using an Adam optimizer until the training termination condition is met, and the optimal crowd counting network model is obtained; the optimal crowd counting network model is input into the test set for testing, and if the test result meets the accuracy requirement, the trained crowd counting network model is obtained.
[0101] Among them, the crowd image data can be downloaded from an open source dataset website or self-made.
[0102] The real crowd probability map and the real crowd density map corresponding to the crowd image data can be generated using the annotation map of the crowd image data. When downloading crowd image data from the open source dataset website, the corresponding annotation map can be downloaded at the same time to generate the real crowd probability map and the real crowd density map.
[0103] Among them, the crowd image data and its annotation graph can also be scaled using the resize function in the torchvision.transforms toolkit so that the longest side of the image does not exceed 1920 pixels before being used as samples.
[0104] In one example, the 1201 training images and 334 test images and their annotated images in the downloaded dataset are scaled using the resize function in the torchvision.transforms toolkit so that the longest side of the image does not exceed 1920 pixels. The scaled training images and annotated images are placed in the training set, and the scaled test images and annotated images are placed in the validation set and the test set at the same time; a probability map generation program is used to generate a real crowd probability map of the same size as the original image for each annotated image; a Gaussian kernel function of σ=10 is used to generate a real crowd density map of 1 / 8 the size of the original image for each annotated image, where the parameter σ of the Gaussian kernel function uses different sizes on different datasets, thereby obtaining the first training set, validation set, and test set. Then analyze the local count value of each 32×32 block in each crowd image data in the first training set (that is, divide the real crowd density map corresponding to the crowd image data into 4×4 pixel areas, and count the crowd values contained in each area). Then, based on the blocks with higher crowd values (that is, the blocks with local count values exceeding the preset value), generate an artificial synthetic map of the dense area and the corresponding real crowd probability map and real crowd density map. In this way, an artificially synthesized dense data sample is generated. The artificially synthesized dense data sample is added to the first training set to obtain the second training set.
[0105] The first training set is used for the first-stage probability map generation network module training, and the second training set is used for the second-stage density map regression network module training.
[0106] Among them, the initial parameters of the training crowd counting network model are set as follows:
[0107] Initial learning rate: 0.00001;
[0108] Optimizer: Adam optimizer.
[0109] Weight decay: 0.005
[0110] Batch size: 4
[0111] Number of training rounds (epoch): 500;
[0112] Data augmentation: The input image (i.e., the crowd image data in the crowd image data sample) is randomly cropped into 8 image blocks of size 384×384, randomly scaled, and horizontally flipped with a probability of 0.5;
[0113] Before training, the convolutional layers of the crowd counting network model all use normal distribution to initialize parameters, where: μ = 0, σ = 0.01.
[0114] It should be understood that the triple counting loss function is applied to the two-stage learning task to better fit the true probability map and the true density map, thereby improving the counting accuracy of crowd images.
[0115] The training process is repeated, and the validation set is used to evaluate the crowd counting network model after each round of training. The network parameters are saved according to the optimal mean absolute error (MAE) value until 500 rounds of training are completed (that is, the training termination condition is met), and the optimal crowd counting network model is saved.
[0116] Among them, the mean absolute error MAE and mean square error MSE are used to evaluate the trained crowd counting network model. The calculation methods of MAE and MSE are as follows:
[0117] ,
[0118] ;
[0119] in, Q is the number of crowd image data samples in the validation set, is the predicted count, is the true count. The lower the values of MAE and MSE, the better the counting performance of the trained crowd counting network model.
[0120] Among them, the test set is input into the optimal crowd counting network model for testing to obtain the crowd image counting result, and then the MAE and MSE scores are calculated by combining the real crowd probability map and the real crowd density map of the test set, and the test results are saved. If the test results meet the accuracy requirements, the trained crowd counting network model is obtained; if it does not meet the accuracy requirements, the crowd counting network model is retrained until it meets the accuracy requirements, and the trained crowd counting network model is obtained.
[0121] In one embodiment, the triple counting loss function Lfor:
[0122] ,
[0123] in,
[0124] ,
[0125] in, L pmg The loss of the network module for probability map generation, is the number of crowd image data samples in the first training set, is the input crowd image data, are the network weights (a set of learnable network weights), Express The predicted probability plot of Represents the true population probability map;
[0126] ,
[0127] in, L dmr is the loss of the density map regression network module, M is the number of crowd image data samples in the second training set, are network weights that are not shared with the probabilistic graph generation network module (a set of learnable network weights that are not shared with the first stage), Express The predicted density map of is the real population density map;
[0128] ,
[0129] ,
[0130] in, L c is the loss between the final counting result and the actual counting, express The difference between the predicted density map and the actual crowd density map.
[0131] Among them, a triple counting loss function that is more suitable for a two-stage network is adopted. L2 loss is selected in the two stages of the crowd counting network model to better fit the real crowd probability map and the real crowd density map, and SmoothL1 loss is used for the real count and density map counting results to improve the counting ability of the model.
[0132] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0134] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A crowd counting method based on attention feature fusion and multi-column feature enhancement, characterized in that: The method comprises: Obtaining images of people to be identified; Inputting the crowd image to be identified into the probability map generation network module of the trained crowd counting network model for processing, and outputting a probability map; Inputting the crowd image to be identified and the probability map into the density map regression network module of the trained crowd counting network model for processing, and outputting a density map, wherein the density map regression network module includes a multi-column feature enhancement network and a dual-attention feature fusion network; Performing an integration operation on the density map to obtain a crowd count in the to-be-identified crowd image; The probability map generation network module includes an encoder composed of the first 13 convolutional layers and the first 4 maximum pooling layers of the VGG-16 network, a first upsampling, a first splicing operation, a second splicing operation, a third splicing operation, a fourth splicing operation, a first feature adjustment unit, a second feature adjustment unit, a third feature adjustment unit, and a fourth feature adjustment unit; The step of inputting the crowd image to be identified into a probability map generation network module of a trained crowd counting network model for processing and outputting a probability map comprises: The image of the crowd to be identified is input into the encoder for feature extraction to obtain five feature maps of different scales, which are, in descending order of scale, a first feature map, a second feature map, a third feature map, a fourth feature map, and a fifth feature map; After the fifth feature map is input into the first upsampling and bilinear interpolation upsampling is performed, the first splicing operation is used to splice it with the fourth feature map to obtain a first splicing feature; After the first splicing feature is input into the first feature adjustment unit for feature adjustment, the second splicing operation is used to splice the first splicing feature with the third feature map to obtain a second splicing feature; After the second splicing feature is input into the second feature adjustment unit for feature adjustment, the second splicing feature is spliced with the second feature map using the third splicing operation to obtain a third splicing feature; After the third splicing feature is input into the third feature adjustment unit for feature adjustment, the third splicing feature is spliced with the first feature map using the fourth splicing operation to obtain a fourth splicing feature; The fourth splicing feature is input into the fourth feature adjustment unit for feature adjustment to obtain a probability map; The density map regression network module includes: a feature extraction unit consisting of the first 13 convolutional layers and the first 5 maximum pooling layers of the VGG-16 network, a first multi-column feature enhancement unit, a second multi-column feature enhancement unit, a first dual-attention feature fusion unit, a second dual-attention feature fusion unit and a density map generation unit; The step of inputting the to-be-identified crowd image and the probability map into the density map regression network module of the trained crowd counting network model for processing and outputting the density map comprises: Inputting the to-be-identified crowd image and the probability map into the feature extraction unit for feature extraction, obtaining a sixth feature map output by the 10th convolutional layer of the feature extraction unit, a seventh feature map output by the 13th convolutional layer of the feature extraction unit, and an eighth feature map output by the 5th maximum pooling layer of the feature extraction unit; The eighth feature map is input into the first multi-column feature enhancement unit for feature enhancement to obtain a first enhanced feature map; Inputting the first enhanced feature map and the seventh feature map into the first dual attention feature fusion unit for feature fusion to obtain a first fused feature map; Inputting the first fused feature map into a second multi-column feature enhancement unit for feature enhancement to obtain a second enhanced feature map; Inputting the second enhanced feature map and the sixth feature map into the second dual attention feature fusion unit for feature fusion to obtain a second fused feature map; Inputting the second fused feature map into the density map generating unit to generate a density map, thereby obtaining a density map; The first multi-column feature enhancement unit and the second multi-column feature enhancement unit are multi-column feature enhancement networks with the same structure, and the multi-column feature enhancement network includes a multi-column enhancement structure and an attention structure; The feature maps input into the multi-column feature enhancement network are respectively input into the multi-column enhancement structure and the attention structure, the attention weight is obtained through the attention structure, and the enhanced feature is obtained through the multi-column enhancement structure, and then the attention weight and the enhanced feature are multiplied to output the enhanced feature map; The first dual attention feature fusion unit and the second dual attention feature fusion unit are dual attention feature fusion networks with the same structure, and the dual attention feature fusion network includes: a thirteenth convolution, a fourteenth convolution, a third Relu activation function, a fifteenth convolution, a second global average pooling, a sixteenth convolution, a fourth Relu activation function, a seventeenth convolution, a second Sigmoid function, average pooling, maximum pooling, an eighteenth convolution and a third Sigmoid function; After upsampling, the deep enhanced feature map input into the dual attention feature fusion network is concatenated with the shallow feature map input into the dual attention feature fusion network and then input into the thirteenth convolution for processing to obtain the first feature; The first feature is operated sequentially through the fourteenth convolution, the third ReLU activation function and the fifteenth convolution, and a first operation result is output; The first feature is operated sequentially through the second global average pooling, the sixteenth convolution, the fourth ReLU activation function and the seventeenth convolution, and a second operation result is output; The first operation result and the second operation result are added together and activated using the second Sigmoid function to obtain a weight between 0 and 1. w ; The weight w Multiplying the deep enhancement feature map by the deep enhancement feature map to obtain a processed deep feature map; The weight (1- w ) is multiplied by the shallow feature map to obtain a processed shallow feature map; Add the processed deep feature map and the processed shallow feature map to obtain a preliminary fusion feature map; The shallow feature map is spliced with the results of the average pooling and the maximum pooling operations respectively, and then input into the eighteenth convolution process and activated by the third Sigmoid function. The activated output is multiplied by the preliminary fusion feature map as a weight to obtain a fusion feature map.
2. The crowd counting method based on attention feature fusion and multi-column feature enhancement according to claim 1 is characterized in that: The multi-column enhancement structure includes a first convolution, a second convolution, a third convolution, a fourth convolution, a fifth convolution, a sixth convolution, a seventh convolution and an eighth convolution; Input the feature map of the multi-column enhanced structure, respectively input the first convolution, the second convolution, the third convolution and the fourth convolution processing channel number, and obtain the first processed feature, the second processed feature, the third processed feature and the fourth processed feature accordingly; The second processed feature is input into the fifth convolution for processing to obtain the fifth processed feature; The third processed feature is added to the fifth processed feature and then subjected to a sixth convolution process to obtain a sixth processed feature; The fourth processed feature is added to the sixth processed feature and then subjected to a seventh convolution process to obtain a seventh processed feature; The first processed feature, the fifth processed feature, the sixth processed feature and the seventh processed feature are concatenated and then subjected to an eighth convolution process to output enhanced features.
3. The crowd counting method based on attention feature fusion and multi-column feature enhancement according to claim 1 is characterized in that: The attention structure includes: a ninth convolution, a first Relu activation function, a tenth convolution, a first global average pooling, an eleventh convolution, a second Relu activation function, a twelfth convolution and a first Sigmoid function; The feature map of the input attention structure is processed by the ninth convolution, the first Relu activation function and the tenth convolution of the first branch, and the first global average pooling, the eleventh convolution, the second Relu activation function and the twelfth convolution of the second branch respectively. The outputs of the first branch and the second branch are added and activated by the first Sigmoid function to obtain the attention weight.
4. The crowd counting method based on attention feature fusion and multi-column feature enhancement according to claim 1 is characterized in that: The density map generation unit includes: 2 3×3 convolutions and 1 1×1 convolution; The second fused feature map is subjected to two 3×3 convolutions to reduce the channels of the fused feature map, and then a 1×1 convolution is performed to generate a density map output.
5. The crowd counting method based on attention feature fusion and multi-column feature enhancement according to claim 1 is characterized in that: The training method of the crowd counting network model is: Acquire a crowd image data sample set, wherein the crowd image data samples in the crowd image data sample set include crowd image data and corresponding real crowd probability maps and real crowd density maps; Dividing the crowd image data sample set into a first training set, a validation set and a test set; Analyze the local count value of each 32×32 block in each crowd image data in the first training set, and generate an artificial synthetic map of the dense area and a corresponding real crowd probability map and real crowd density map as artificial synthetic dense data samples based on the blocks whose local count values exceed a preset value; Adding the artificially synthesized dense data sample to the first training set to obtain a second training set; The probability map generation network module of the crowd counting network model is trained using the first training set and the validation set, and the density map regression network module of the crowd counting network model is trained using the second training set and the validation set, a triple counting loss function is used to calculate the gradient during the training process, and an Adam optimizer is used to update and optimize the network parameters of the probability map generation network module and the density map regression network module of the crowd counting network model until the training termination condition is met, thereby obtaining an optimal crowd counting network model; The test set is used to input the optimal crowd counting network model for testing, and if the test result meets the accuracy requirement, a trained crowd counting network model is obtained; The triple counting loss function L for: , in, , in, L pmg The loss of the network module for probability map generation, is the number of crowd image data samples in the first training set, is the input crowd image data, is the network weight, Express The predicted probability plot of Represents the true population probability map; , in, L dmr is the loss of the density map regression network module, M is the number of crowd image data samples in the second training set, are the network weights that are not shared with the probabilistic graph generation network module, Express The predicted density map of is the real population density map; , , in, L c is the loss between the final counting result and the actual counting, express The difference between the predicted density map and the actual crowd density map.
