Crowd counting method and system based on dual-path multi-scale fusion network
Through a dual-channel multi-scale fusion network, combining self-attention, spatial attention and residual attention feature fusion modules, the accuracy of population counting in complex scenarios is solved, and a higher quality density map is generated, which improves counting accuracy .
Patent Information
- Application Number
- CN202411117386.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-15
AI Technical Summary
The existing population counting method performs poorly in complex scenarios. Due to local information and assumptions of specific scenarios, it is difficult to effectively deal with the problems of scale changes, complex backgrounds and non-uniform distribution.
The population counting method based on a dual-channel multi-scale fusion network is adopted, including feature extraction network, self-attention feature enhancement module, spatial attention multi-scale feature fusion module and residual attention feature fusion module, to generate higher levels through multi-layer feature fusion Density graph of quality to improve counting accuracy.
Capturing the global and local features of the image through self-attention and spatial attention mechanisms, solving the problems of background complexity and scale changes. The residual attention feature fusion module is designed to enhance feature fusion and generate higher quality density maps, which significantly improves the Accuracy of population counting.
Smart Images

Figure CN118644486B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a crowd counting method and system based on a dual-path multi-scale fusion network. Background Art
[0002] As one of the important problems in the field of computer vision, crowd counting is of vital importance to modern society. Accurate crowd counting can not only provide important support for urban planning, security monitoring, and traffic management, but also play a key role in the security and crowd management of various large-scale events.
[0003] In practical application scenarios, crowd counting research is still challenging due to problems such as target occlusion, scale variation, complex background, and non-uniform distribution, and a broader research perspective is needed to improve algorithm performance. Previous crowd counting methods are often limited by local information and specific scene assumptions, resulting in poor performance in complex scenes. Summary of the invention
[0004] The technical problem to be solved by the present invention is: in order to cope with the problems of scale changes, complex background and non-uniform distribution in crowd images, a crowd counting method and system based on a dual-path multi-scale fusion network are provided, which fully utilizes the features of each stage to generate a higher quality density map and improve the counting accuracy of the overall image.
[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0006] The crowd counting method based on dual-path multi-scale fusion network includes the following steps:
[0007] S1. Obtain a crowd image dataset and preprocess it, divide the preprocessed dataset into a training set, a validation set, and a test set; use the labeled image to obtain a real crowd density map.
[0008] S2. Construct a dual-path multi-scale fusion network, which includes a feature extraction network, a self-attention feature enhancement module, a spatial attention multi-scale feature fusion module, a residual attention feature fusion module and a density map generation module.
[0009] The feature extraction network includes the first 13 convolutional layers and the first 4 pooling layers of the VGG-16 network. The outputs of the 10th and 13th convolutional layers of the feature extraction network are transmitted to the self-attention feature enhancement module, and the outputs of the 7th, 10th and 13th convolutional layers are transmitted to the spatial attention multi-scale feature fusion module. The outputs of the self-attention feature enhancement module and the spatial attention multi-scale feature fusion module are transmitted to the residual attention feature fusion module, and the output of the residual attention feature fusion module is transmitted to the density map generation module.
[0010] S3. Use the training set and the validation set in step S1 to train the dual-path multi-scale fusion network to obtain a trained dual-path multi-scale fusion network.
[0011] S4. Input the test set in step S1 into the trained dual-path multi-scale fusion network to obtain the crowd image counting result.
[0012] Furthermore, in step S1, the image processing includes the following steps:
[0013] S101. Download a crowd image dataset from an open source dataset website or create your own, and scale the crowd images and their annotation images in the dataset.
[0014] S102, dividing the scaled crowd image and its annotated image into a training set, a validation set and a test set according to a set ratio.
[0015] S103, using a Gaussian kernel function to generate a real crowd density map for the labeled map.
[0016] Furthermore, in step S2, constructing a dual-path multi-scale fusion network includes the following steps:
[0017] S201. Input the preprocessed crowd image into the first 13 convolutional layers and the first 4 pooling layers of the VGG-16 network for feature extraction, and obtain the first feature map F1 output by the 7th convolutional layer, the second feature map F2 output by the 10th convolutional layer, and the third feature map F3 output by the 13th convolutional layer.
[0018] S202. Input F2 and F3 obtained in step S201 into the self-attention feature enhancement module in sequence, and output the fourth feature map S; input the feature maps F1, F3 and F2 obtained in step S201 into the spatial attention multi-scale feature fusion module in sequence, and output the fifth feature map M.
[0019] S203. Input S and M obtained in step S202 into the residual attention feature fusion module in sequence, and output the sixth feature map R.
[0020] S204: Input R obtained in step S203 into a density map generation module to obtain a predicted crowd density map of the same size as the input image.
[0021] Among them, the self-attention feature enhancement module includes a shallow feature self-attention branch and a deep and shallow feature fusion branch, the spatial attention multi-scale feature fusion module includes a shallow spatial attention branch and a deep feature fusion branch, the residual attention feature fusion module includes a self-attention feature enhancement branch and a residual attention feature fusion branch, and the density map generation module includes three 3×3 convolutions and one 1×1 convolution connected in sequence.
[0022] Further, in step S202, outputting the fourth feature map S includes the following contents:
[0023] After the second feature map F2 is input into the self-attention feature enhancement module, it passes through the shallow feature self-attention branch and uses 3×3 convolution to reduce the number of channels of the second feature map F2 to obtain the seventh feature map T1. T1 is respectively subjected to three 1×1 convolutions to generate feature maps Q, K, and V. The feature map Q is transposed and dot-multiplied with the feature map K. After activation by the Softmax function, the attention weight is obtained. The attention weight is dot-multiplied with the feature map V, and then a 1×1 convolution is performed to obtain the eighth feature map T2. T1 and T2 are added to obtain the self-attention feature map.
[0024] After the third feature map F3 is input into the self-attention feature enhancement module, it is upsampled through the deep and shallow feature fusion branch and then concatenated with the second feature map F2. Then, a 1×1 convolution is performed to obtain the ninth feature map T3. T3 is dot-multiplied with the self-attention feature map to obtain the fourth feature map S.
[0025] Further, in step S202, outputting the fifth feature map M includes the following contents:
[0026] After the first feature map F1 is input into the spatial attention multi-scale feature fusion module, F1 is average pooled and max pooled respectively after the shallow spatial attention branch to obtain the tenth feature map T4 and the eleventh feature map T5. After adding T4 and T5, a 7×7 convolution is performed and then activated by the Sigmoid function to obtain the spatial attention weight. F1 is dot-multiplied with the spatial attention weight and then down-sampled to obtain the twelfth feature map T6.
[0027] After the second feature map F2 and the third feature map F3 are input into the spatial attention multi-scale feature fusion module, F3 is upsampled and concatenated with F2 through the deep feature fusion branch, and then a 1×1 convolution is performed to obtain the thirteenth feature map T7; T6 and T7 are concatenated and concatenated, and then a 1×1 convolution is performed to obtain the fifth feature map M.
[0028] Further, in step S203, outputting the sixth feature map R includes the following contents:
[0029] After the fourth feature map S is input into the residual attention feature fusion module, it passes through the self-attention feature enhancement branch and uses global average pooling to capture the context information of S. The context information passes through a 1×1 convolution and a BN (BatchNormalization) to obtain the fourteenth feature map T8. T8 is activated with the Sigmoid function to obtain the weight, which is multiplied by S to obtain the fifteenth feature map T9.
[0030] After the fifth feature map M is input into the residual attention feature fusion module, M and T9 are concatenated in series through the residual attention feature fusion branch, and then a 3×3 convolution is performed to obtain the sixteenth feature map T10. T10 is sequentially subjected to global average pooling, 1×1 convolution, Relu activation, 1×1 convolution and Sigmoid function activation to obtain the fused channel attention weight, which is dot-multiplied with T10 and then added to T10 to obtain the sixth feature map R.
[0031] Furthermore, in step S204, the output crowd density map includes the following contents:
[0032] The sixth feature map R is input into the density map generation module, and the number of channels of R is reduced through three 3×3 convolutions in sequence, and then a 1×1 convolution with a channel number of 1 is used to generate the predicted crowd density map.
[0033] Furthermore, in step S3, obtaining the trained dual-path multi-scale fusion network includes the following steps:
[0034] S301, set the initial training parameters, including the initial learning rate, optimizer, weight decay, batch size and number of training rounds.
[0035] S302, use SmoothL 1. Loss, L 2 Loss and Total Variation Loss L TV The combined loss function calculates the loss, and the specific formula is:
[0036] ;
[0037] in, L represents the loss function of the combination; N represents the total number of crowd images in the training set; y i Represents the output of the dual-path multi-scale fusion network i The difference between the predicted density map and the true density map of the crowd image; represents the loss weight; L TV represents the total variation loss function, , xi Indicates i images, represents the network weight, D represents the predicted crowd density map output by the dual-path multi-scale fusion network, Represents the real crowd density map; ; .
[0038] The gradient is calculated according to the combined loss function, and the Adam optimizer is used to update the network model parameters.
[0039] S303, using the mean absolute error, mean square error and validation set to evaluate the updated dual-path multi-scale fusion network, the specific formulas of the mean absolute error and mean square error are:
[0040] ;
[0041] ;
[0042] in, MAE represents the mean absolute error, MSE represents the mean square error, C i represents the predicted count, Indicates the true count.
[0043] MAE and MSE The lower the value of , the better the crowd counting performance of the dual-path multi-scale fusion network.
[0044] S304, repeating steps S302-S303 until a preset number of training rounds is reached to obtain a trained dual-path multi-scale fusion network.
[0045] Furthermore, in step S4, obtaining the crowd image counting result includes the following steps:
[0046] S401, input the test set obtained in step S1 into the trained dual-path multi-scale fusion network to obtain a predicted crowd density map, and perform an integral operation on the crowd density map to obtain the predicted number of people.
[0047] S402: Based on the actual number of people and the predicted number of people in the crowd images in the test set, calculate the scores of the mean absolute error and the mean square error, and save the score results and the predicted number of people.
[0048] The crowd counting system based on dual-path multi-scale fusion network includes:
[0049] The image preprocessing module is used to obtain and preprocess the crowd image dataset, divide the preprocessed dataset into a training set, a validation set, and a test set; and obtain a crowd density map using the labeled map.
[0050] The dual-path multi-scale fusion network building module uses the Python programming language and deep learning toolkit to build a dual-path multi-scale fusion network, which includes a feature extraction network, a self-attention feature enhancement module, a spatial attention multi-scale feature fusion module, a residual attention feature fusion module and a density map generation module.
[0051] The network training module is used to train the dual-path multi-scale fusion network using the training set and the verification set in the image preprocessing module to obtain the trained dual-path multi-scale fusion network.
[0052] The result acquisition module is used to input the test set in the image preprocessing module into the trained dual-path multi-scale fusion network to obtain the crowd image counting result.
[0053] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:
[0054] The present invention proposes a dual-path multi-scale fusion network, which uses a self-attention feature enhancement module to capture the global information of shallow feature maps to deal with the problems of complex background and non-uniform distribution; uses a spatial attention multi-scale feature fusion module to calculate the spatial attention of shallow features to enhance the results after deep feature fusion to solve the problem of scale change; and designs a residual attention feature fusion module to enhance and fuse the features of the two branches through two residual connections, making full use of the features of each stage to generate a higher quality density map and improve the counting accuracy of the overall image. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 It is an overall implementation flow chart of the present invention.
[0056] Figure 2 It is a schematic diagram of the dual-path multi-scale fusion network structure of the present invention.
[0057] Figure 3 It is a schematic diagram of the structure of the self-attention feature enhancement module of the present invention.
[0058] Figure 4 It is a schematic diagram of the structure of the spatial attention multi-scale feature fusion module of the present invention.
[0059] Figure 5 It is a schematic diagram of the structure of the residual attention feature fusion module of the present invention.
[0060] Figure 6 It is a schematic diagram of the structure of the density map generation module of the present invention.
[0061] Figure 7 4 is a diagram of the crowd image counting result obtained in an embodiment of the present invention. DETAILED DESCRIPTION
[0062] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0063] To achieve the above objectives, the present invention proposes a crowd counting method based on a dual-path multi-scale fusion network, such as Figure 1 As shown, the specific steps are as follows:
[0064] S1. Obtain a crowd image dataset and preprocess it, divide the preprocessed dataset into a training set, a validation set, and a test set; use the labeled map to obtain a real crowd density map; the specific contents are as follows:
[0065] S101. Download part A of the ShanghaiTech dataset from the open source dataset website https: / / www.kaggle.com / tthien / shanghaitech. Use the resize function in the torchvision.transforms toolkit to scale the 300 training images and 182 test images in the dataset and their annotations so that the longest side of the image does not exceed 2048 pixels.
[0066] S102: putting the scaled training images and the labeled images into the training set, and putting the scaled test images and the labeled images into the validation set and the test set at the same time.
[0067] S103, using the Gaussian kernel function to generate a real crowd density map for the labeled map, the parameters of the Gaussian kernel function Take 10.
[0068] S2, such as Figure 2 As shown, a dual-path multi-scale fusion network is constructed using Python programming language and deep learning toolkit. The network includes a feature extraction network, a self-attention feature enhancement module, a spatial attention multi-scale feature fusion module, a residual attention feature fusion module and a density map generation module.
[0069] The feature extraction network includes the first 13 convolutional layers and the first 4 pooling layers of the VGG-16 network. The first 13 convolutional layers of the VGG-16 network are all 3×3 convolutions, and the number of channels of the convolutional layers are 64, 64, 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512, respectively; after the second, fourth, seventh and tenth convolutional layers, there is a maximum pooling layer with a stride of 2 and a pooling area size of 2.
[0070] The outputs of the 10th and 13th convolutional layers of the feature extraction network are transmitted to the self-attention feature enhancement module, the outputs of the 7th, 10th and 13th convolutional layers are transmitted to the spatial attention multi-scale feature fusion module, the outputs of the self-attention feature enhancement module and the spatial attention multi-scale feature fusion module are transmitted to the residual attention feature fusion module, and the output of the residual attention feature fusion module is transmitted to the density map generation module.
[0071] The specific contents are:
[0072] S201. Input the preprocessed crowd image into the first 13 convolutional layers and the first 4 pooling layers of the VGG-16 network for feature extraction, and obtain the first feature map F1 output by the 7th convolutional layer, the second feature map F2 output by the 10th convolutional layer, and the third feature map F3 output by the 13th convolutional layer.
[0073] S202, such as Figure 3 As shown in the figure, after the second feature map F2 is input into the self-attention feature enhancement module, it passes through the shallow feature self-attention branch, and uses a 3×3 convolution to reduce the number of channels of the second feature map F2 to obtain the seventh feature map T1. T1 is respectively subjected to three 1×1 convolutions to generate feature maps Q, K, and V. The feature map Q is transposed and dot-multiplied with the feature map K. After activation by the Softmax function, the attention weight is obtained. The attention weight is dot-multiplied with the feature map V, and then a 1×1 convolution is performed to obtain the eighth feature map T2. T1 and T2 are added to obtain the self-attention feature map.
[0074] After the third feature map F3 is input into the self-attention feature enhancement module, it is upsampled through the deep and shallow feature fusion branch and then concatenated with the second feature map F2. Then, a 1×1 convolution is performed to obtain the ninth feature map T3. T3 is dot-multiplied with the self-attention feature map to obtain the fourth feature map S.
[0075] like Figure 4As shown in the figure, after the first feature map F1 is input into the spatial attention multi-scale feature fusion module, F1 is average pooled and maximum pooled respectively after passing through the shallow spatial attention branch to obtain the tenth feature map T4 and the eleventh feature map T5. After adding T4 and T5, a 7×7 convolution is performed and then the Sigmoid function is used for activation to obtain the spatial attention weight. F1 is point-multiplied with the spatial attention weight and then down-sampled to obtain the twelfth feature map T6.
[0076] After the second feature map F2 and the third feature map F3 are input into the spatial attention multi-scale feature fusion module, F3 is upsampled and concatenated with F2 through the deep feature fusion branch, and then a 1×1 convolution is performed to obtain the thirteenth feature map T7; T6 and T7 are concatenated and concatenated, and then a 1×1 convolution is performed to obtain the fifth feature map M.
[0077] S203, such as Figure 5 As shown in the figure, after the fourth feature map S is input into the residual attention feature fusion module, it passes through the self-attention feature enhancement branch and uses global average pooling to capture the context information of S. The context information passes through a 1×1 convolution and a BN (Batch Normalization) to obtain the fourteenth feature map T8. T8 is activated by the Sigmoid function to obtain the weight, and the weight is multiplied by S to obtain the fifteenth feature map T9.
[0078] After the fifth feature map M is input into the residual attention feature fusion module, M and T9 are concatenated in series through the residual attention feature fusion branch, and then a 3×3 convolution is performed to obtain the sixteenth feature map T10. T10 is sequentially subjected to global average pooling, 1×1 convolution, Relu activation, 1×1 convolution and Sigmoid function activation to obtain the fused channel attention weight, which is dot-multiplied with T10 and then added to T10 to obtain the sixth feature map R.
[0079] S204, such as Figure 6 As shown in the figure, the sixth feature map R is input into the density map generation module, and the number of channels of R is reduced by three 3×3 convolutions with output channels of 256, 128, and 64 respectively, and then a 1×1 convolution with a channel number of 1 is used to obtain the predicted crowd density map of the same size as the input image.
[0080] S3, using the training set and the validation set in step S1 to train the dual-path multi-scale fusion network to obtain the trained dual-path multi-scale fusion network; the specific content is:
[0081] S301, set the initial training parameters, including the initial learning rate of 0.00001, select the Adam optimizer, the weight decay of 0.0001, the batch size of 4, and the number of training rounds of 500.
[0082] The input image is randomly cropped into 512×512 image patches and horizontally flipped with a probability of 0.5, the RGB channels of the color image are swapped with a probability of 0.2, and Gaussian noise with a mean of 0 and a standard deviation of 5 is added with a probability of 0.1; for datasets containing gray images, the color image is randomly converted to a grayscale image with a probability of 0.1.
[0083] Before training, the convolutional layers of the model all use normal distribution to initialize parameters, among which the parameters of the normal distribution are: mean ,variance .
[0084] S302, use SmoothL 1. Loss, L 2 Loss and Total Variation Loss L TV The combined loss function calculates the loss, and the specific formula is:
[0085] ;
[0086] in, L represents the loss function of the combination; N represents the total number of crowd images in the training set; y i Represents the output of the dual-path multi-scale fusion network i The difference between the predicted density map and the true density map of the crowd image; L TV represents the total variation loss function, , x i Indicates i images, represents the network weight, D represents the predicted crowd density map output by the dual-path multi-scale fusion network, Represents the real crowd density map; ; ; Represents the loss weight, which is set to 0.1.
[0087] The gradient is calculated according to the combined loss function, and the Adam optimizer is used to update the network model parameters.
[0088] S303, using the mean absolute error, mean square error and validation set to evaluate the updated dual-path multi-scale fusion network, the specific formulas of the mean absolute error and mean square error are:
[0089] ;
[0090] ;
[0091] in, MAE represents the mean absolute error, MSE represents the mean square error, C i represents the predicted count, Indicates the true count.
[0092] MAE and MSE The lower the value of , the better the crowd counting performance of the dual-path multi-scale fusion network.
[0093] S304, repeat steps S302-S303 until 500 training rounds are completed to obtain a trained dual-path multi-scale fusion network.
[0094] S4, input the test set in step S1 into the trained dual-path multi-scale fusion network to obtain the crowd image counting result; the specific content is:
[0095] S401, input the test set obtained in step S1 into the trained dual-path multi-scale fusion network to obtain a predicted crowd density map, and perform an integral operation on the crowd density map to obtain the predicted number of people.
[0096] S402: Based on the actual number of people and the predicted number of people in the crowd images in the test set, calculate the scores of the mean absolute error and the mean square error, and save the score results and the predicted number of people.
[0097] like Figure 7 As shown in the figure, five images of different crowd densities are randomly selected from Part A of the ShanghaiTech dataset as input. The three columns in the figure are, from left to right, the input image, the crowd density map predicted by the trained dual-path multi-scale fusion network, and the labeled map. The numbers marked on the map are the number of people corresponding to the image. It can be seen from the first and second rows that in relatively sparse crowd images, the counting effect obtained by the method proposed in the present invention is relatively accurate, and the predicted crowd density is basically consistent with the actual labeled crowd density; it can be seen from the third, fourth and fifth rows that in medium-dense crowd images, the counting error obtained by the method proposed in the present invention is controlled within an acceptable range.
[0098] The embodiment of the present invention also proposes a system for crowd counting based on a dual-path multi-scale fusion network, including an image preprocessing module, a dual-path multi-scale fusion network construction module, a network training module, a result acquisition module, and a computer program that can be run on a processor. It should be noted that each module in the above system corresponds to the specific steps of the method provided in the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided in the embodiment of the present invention.
[0099] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A crowd counting method based on a dual-path multi-scale fusion network, characterized in that: include: S1, obtain and preprocess a crowd image dataset, and divide the preprocessed dataset into a training set, a validation set, and a test set; Use the labeled map to obtain the real crowd density map; S2. Construct a dual-path multi-scale fusion network, which includes a feature extraction network, a self-attention feature enhancement module, a spatial attention multi-scale feature fusion module, a residual attention feature fusion module and a density map generation module; The feature extraction network includes the first 13 convolutional layers and the first 4 pooling layers of the VGG-16 network. The outputs of the 10th and 13th convolutional layers of the feature extraction network are transmitted to the self-attention feature enhancement module, and the outputs of the 7th, 10th and 13th convolutional layers are transmitted to the spatial attention multi-scale feature fusion module. The outputs of the self-attention feature enhancement module and the spatial attention multi-scale feature fusion module are transmitted to the residual attention feature fusion module, and the output of the residual attention feature fusion module is transmitted to the density map generation module. Among them, the self-attention feature enhancement module includes a shallow feature self-attention branch and a deep and shallow feature fusion branch, the spatial attention multi-scale feature fusion module includes a shallow spatial attention branch and a deep feature fusion branch, the residual attention feature fusion module includes a self-attention feature enhancement branch and a residual attention feature fusion branch, and the density map generation module includes three 3×3 convolutions and one 1×1 convolution connected in sequence; The specific contents are: S201, input the preprocessed crowd image into the first 13 convolutional layers and the first 4 pooling layers of the VGG-16 network for feature extraction, and obtain the first feature map F1 output by the 7th convolutional layer, the second feature map F2 output by the 10th convolutional layer, and the third feature map F3 output by the 13th convolutional layer; S202, inputting F2 and F3 obtained in step S201 into the self-attention feature enhancement module in sequence, and outputting the fourth feature map S; inputting the feature maps F1, F3 and F2 obtained in step S201 into the spatial attention multi-scale feature fusion module in sequence, and outputting the fifth feature map M; the specific contents are: After the second feature map F2 is input into the self-attention feature enhancement module, it passes through the shallow feature self-attention branch, uses 3×3 convolution to reduce the number of channels of the second feature map F2, and obtains the seventh feature map T1. T1 is respectively subjected to three 1×1 convolutions to generate feature maps Q, K, and V. The feature map Q is transposed and then multiplied with the feature map K. After activation by the Softmax function, the attention weight is obtained. The attention weight is multiplied with the feature map V, and then a 1×1 convolution is performed to obtain the eighth feature map T2. T1 and T2 are added to obtain the self-attention feature map. After the third feature map F3 is input into the self-attention feature enhancement module, it is upsampled through the deep and shallow feature fusion branch, and then concatenated with the second feature map F2 in series, and then through a 1×1 convolution to obtain the ninth feature map T3, and T3 is dot-multiplied with the self-attention feature map to obtain the fourth feature map S; After the first feature map F1 is input into the spatial attention multi-scale feature fusion module, F1 is average pooled and max pooled respectively through the shallow spatial attention branch to obtain the tenth feature map T4 and the eleventh feature map T5. After T4 and T5 are added, a 7×7 convolution is performed, and then the Sigmoid function is used for activation to obtain the spatial attention weight. F1 is dot-multiplied with the spatial attention weight, and then down-sampled to obtain the twelfth feature map T6. After the second feature map F2 and the third feature map F3 are input into the spatial attention multi-scale feature fusion module, F3 is upsampled and concatenated with F2 through the deep feature fusion branch, and then a 1×1 convolution is performed to obtain the thirteenth feature map T7; T6 and T7 are concatenated and concatenated, and then a 1×1 convolution is performed to obtain the fifth feature map M; S203, inputting S and M obtained in step S202 into the residual attention feature fusion module in sequence, and outputting the sixth feature map R; the specific content is: After the fourth feature map S is input into the residual attention feature fusion module, it passes through the self-attention feature enhancement branch and uses global average pooling to capture the context information of S. The context information passes through a 1×1 convolution and a normalization layer to obtain the fourteenth feature map T8. T8 is activated by the Sigmoid function to obtain the weight, which is multiplied by S to obtain the fifteenth feature map T9. After the fifth feature map M is input into the residual attention feature fusion module, M and T9 are concatenated in series through the residual attention feature fusion branch, and then a 3×3 convolution is performed to obtain the sixteenth feature map T10. T10 is sequentially subjected to global average pooling, 1×1 convolution, Relu activation, 1×1 convolution and Sigmoid function activation to obtain the fused channel attention weight, which is dot-multiplied with T10 and then added to T10 to obtain the sixth feature map R; S204, inputting R obtained in step S203 into a density map generation module to obtain a predicted crowd density map of the same size as the input image; the specific content is: The sixth feature map R is input into the density map generation module, and the number of channels of R is reduced by three 3×3 convolutions in sequence, and then a 1×1 convolution with a channel number of 1 is used to generate the predicted crowd density map; S3, using the training set and the validation set in step S1 to train the dual-path multi-scale fusion network to obtain a trained dual-path multi-scale fusion network; S4. Input the test set in step S1 into the trained dual-path multi-scale fusion network to obtain the crowd image counting result.
2. The crowd counting method based on dual-path multi-scale fusion network according to claim 1 is characterized in that: In step S1, image processing includes the following steps: S101, downloading a crowd image dataset from an open source dataset website or making a self-made one, and scaling the crowd images and their annotation images in the dataset; S102, dividing the scaled crowd image and its annotated image into a training set, a validation set, and a test set according to a set ratio; S103, using a Gaussian kernel function to generate a real crowd density map for the labeled map.
3. The crowd counting method based on dual-path multi-scale fusion network according to claim 1 is characterized in that: In step S3, obtaining the trained dual-path multi-scale fusion network includes the following steps: S301, setting initial training parameters, including initial learning rate, optimizer, weight decay, batch size and number of training rounds; S302, using SmoothL1 loss, L2 loss and total variation loss L TV The combined loss function calculates the loss, and the specific formula is: Where L represents the combined loss function; N represents the total number of crowd images in the training set; y i represents the difference between the predicted density map and the real density map of the i-th crowd image output by the dual-path multi-scale fusion network; λ represents the loss weight; x i represents the i-th image, η represents the network weight, D represents the predicted crowd density map output by the dual-path multi-scale fusion network, and D gt Represents the real crowd density map; Calculate the gradient based on the combined loss function and use the Adam optimizer to update the network model parameters; S303, using the mean absolute error, mean square error and validation set to evaluate the updated dual-path multi-scale fusion network, the specific formulas of the mean absolute error and mean square error are: Among them, MAE stands for mean absolute error, MSE stands for mean square error, C i represents the predicted count, represents the real count; S304, repeating steps S302-S303 until a preset number of training rounds is reached to obtain a trained dual-path multi-scale fusion network.
4. The crowd counting method based on dual-path multi-scale fusion network according to claim 1 is characterized in that: In step S4, obtaining the crowd image counting result includes the following steps: S401, inputting the test set obtained in step S1 into the trained dual-path multi-scale fusion network to obtain a predicted crowd density map, and performing an integral operation on the crowd density map to obtain a predicted number of people; S402: Based on the actual number of people and the predicted number of people in the crowd images in the test set, calculate the scores of the mean absolute error and the mean square error, and save the score results and the predicted number of people.
5. A system for the crowd counting method based on a dual-path multi-scale fusion network according to claim 1, characterized in that: include: An image preprocessing module is used to obtain and preprocess a crowd image dataset, and divide the preprocessed dataset into a training set, a validation set, and a test set; Use the labeled map to obtain the crowd density map; A dual-path multi-scale fusion network building module uses Python programming language and deep learning toolkit to build a dual-path multi-scale fusion network, which includes a feature extraction network, a self-attention feature enhancement module, a spatial attention multi-scale feature fusion module, a residual attention feature fusion module, and a density map generation module; A network training module is used to train the dual-path multi-scale fusion network using the training set and the verification set in the image preprocessing module to obtain a trained dual-path multi-scale fusion network; The result acquisition module is used to input the test set in the image preprocessing module into the trained dual-path multi-scale fusion network to obtain the crowd image counting result.
