Multi-scale feature fusion method based on WQCount model

Abstract

The invention discloses a multi-scale feature fusion method based on a WQCount model, and the method comprises the steps: S1, selecting a ShanghaiTeck data set, a UCF-QNRF data set and a JHU-Crowd + + data set, and dividing the data sets into a training set, a verification set and a test set; step S2, carrying out preprocessing on the ShanghaiTeck data set, the UCF-QNRF data set and the JHU-Crowd + + data set; step S3, a WQCount model is constructed; s4, training and verifying the WQCount model through the training set and the verification set, and testing the test set by using the trained WQCount model; and S5, predicting crowd density under each scale by using the tested WQCount model, and finally, optimizing feature representation through maximum density fusion. According to the method, the robustness of the model to domain migration is remarkably improved through a multi-scale feature decoupling and dynamic fusion mechanism.

Description

TECHNICAL FIELD

[0001] The application belongs to a multi-scale feature fusion method based on a WQCount model. BACKGROUND

[0002] In the field of computer vision, crowd counting aims to accurately count the number of pedestrians in an image or video through automated means. With the acceleration of global population growth and urbanization, dense crowd monitoring shows important value in public safety management, commercial behavior analysis, and other scenarios. Early research mainly relies on detection-based methods (such as head-shoulder detection or sliding window method), but such methods significantly decrease in performance in dense scenes with severe occlusion or complex background. To solve this problem, density map estimation-based methods have emerged, which predict the total number of people by regressing a continuous density map and accumulating pixel values. Compared with traditional detection methods, density map regression significantly improves the robustness in occluded scenes and becomes the current mainstream technology route.

[0003] Density map estimation-based crowd counting methods have achieved significant results in various application scenarios. However, this method still faces a series of challenges, including scale variation, complex background, and heterogeneous distribution of crowds. Most importantly, due to domain shift problems, most existing density map regression methods often perform unsatisfactorily when applied to new scenarios. To overcome these limitations, a multi-scale feature fusion method based on the WQCount (feature decoupling single-domain generalization crowd counting) model is provided. SUMMARY

[0004] To solve the above problems existing in the prior art, the application provides a multi-scale feature fusion method based on a WQCount model, which significantly improves the robustness of the model to domain shift through multi-scale feature decoupling and dynamic fusion mechanism.

[0005] The technical solution to achieve the above purpose is: A multi-scale feature fusion method based on a WQCount model, comprising: Step S1, selecting ShanghaiTech, UCF-QNRF, and JHU-Crowd++ datasets, and dividing the datasets into training sets, validation sets, and test sets; Step S2, preprocessing the ShanghaiTech, UCF-QNRF, and JHU-Crowd++ datasets; Step S3, constructing a WQCount model; Step S4, training and validating the WQCount model through the training set and the validation set, and testing the test set using the trained WQCount model; Step S5, predict the crowd density under each scale using the tested WQCount model, and finally optimize the feature representation through maximum density fusion.

[0006] Preferably, in step S2, the preprocessing operation includes data enhancement, and three photometric transformations are applied: color jitter, Gaussian blur, and sharpening; randomly crop image blocks with a size of 320x320, and apply random horizontal flipping to both the original image and the enhanced image.

[0007] Preferably, in step S3, the WQCount model includes: a feature processing module FPM for multi-scale feature decoupling, which uses multi-scale feature adjustment MFA and similar attention mechanism SAM to make each feature layer focus on its own strengths and non-overlapping content; a distribution loss DL for constraining the spatial distribution and representation difference of different feature layers to achieve feature decoupling; a maximum density fusion MDF for fusing information by selecting the maximum density value in the density map of different scale branches; a branch density loss BDL and a branch mask BM, which dynamically generate BM through the predicted density value, multiply the branch label with the label map to obtain the branch label, and then calculate the loss between the branch label and the branch density map.

[0008] Preferably, in step S3, the target of the WQCount model is to train a neural network with parameters , which receives an input image , where and are the height and width of the feature map respectively, 3 is the number of channels, and outputs a density map of the same size, which is an estimate of the total number of people , which is obtained by integrating all pixel values of the density map: ; The real density map is usually annotated based on the head coordinate points of the human body , where each represents the coordinates of the head position, and the real density map is calculated by applying a two-dimensional Gaussian model to each head coordinate: ; where is a discrete Dirac function, is a Gaussian kernel with a fixed variance ; The commonly used loss function for supervised neural networks is mean square error: ​​ .

[0009] Preferably, in the step S4, the WQCount model training comprises: performing photometric transformation on the original image to generate an enhanced image ; Subsequently, the feature extractor encodes and to generate features and respectively, wherein is a scale index, ; Based on the above features, a content error mask is calculated; At the same time, input and to the feature processing module FPM to obtain branch advantage mismatch features focusing on respective advantages and different regions and ; Based on the branch advantage mismatch features and , a distribution loss DL is calculated; The output results of are applied to and to obtain masked branch features; The above features are input to the attention memory bank after being adjusted in channel number by 1x1 convolution to reconstruct and ; The reconstructed features and are then input to the density head to predict the crowd density and obtain multi-scale branch density maps and ; The multi-scale branch density maps are input to the MDF to perform maximum density fusion and obtain density maps and ; At the same time, the highest level features encoded by the feature extractor and are input to the block-level classification head, and the output thereof is used to alleviate the pixel-level label ambiguity through binary classification supervision to generate predicted block-level classification maps and ; and binarized and resized as masks for filtering the estimated density maps and , and the fused density map and ; remove the area without head but with density value due to the wrong guidance of the label, and finally generate the branch density map filtered by the block-level classification mask and , and the final fused density map and .

[0010] Preferably, in the step S4, in the feature processing module FPM, in order to strengthen the dominant features of each level feature map and weaken the non-dominant features, a multi-scale feature adjustment MFA is designed, that is: For a given feature map input to the FPM: ; In the formula, is the number of channels; First, calculate the average value in the spatial dimension and the maximum value in the spatial dimension, so as to obtain the global average vector and the global maximum vector : ; ; In the formula, represents the maximum value of the feature map in the spatial dimension, i.e. height and width ; In order to measure the dominant features and the non-dominant features, a dominant threshold vector is defined: ; The features greater than the threshold are regarded as dominant features, and the features less than the threshold are regarded as non-dominant features; In order to achieve the goal of enhancing the dominant features and weakening the non-dominant features, the original feature map is divided by the threshold to calculate the feature dominant weight matrix , and the feature value greater than the threshold has a weight greater than 1 after being divided by the threshold, and the feature value less than the threshold has a weight less than 1 after being divided by the threshold: ; By multiplying the calculated weight matrix and the original feature map , the output feature map is obtained, wherein , and finally the adaptive enhancement of the dominant features is realized, and the non-dominant features are effectively suppressed. The similarity between feature maps is calculated by the Similarity Attention Mechanism (SAM), which encourages different feature layers to focus on non-overlapping regions, i.e.: For a given feature map: ; Optimization is performed in two dimensions: inter-level similarity calculation and intra-level fine-grained interaction; In inter-level similarity calculation, the similarity between different feature maps is calculated by multiplying the corresponding channel elements of the feature maps pixel by pixel and summing them up in the channel dimension to obtain their similarity scores : ; where is the sum in the channel dimension, and are the feature maps of the th and th feature map in the th channel, is the Hadamard product, is the scaling factor, ; Then, the comprehensive similarity score , of each feature map is calculated, which is the average of the similarity scores between adjacent feature maps: ; To guide different feature layers to focus on non-overlapping regions, the comprehensive similarity score is first negated, and then the negative similarity score is normalized by Softmax to generate the external weight , , which suppresses high similarity regions and thus enhances the complementarity between feature layers. The formula is defined as: ; In intra-level fine-grained interaction, the similarity between channels is measured by calculating the element-wise multiplication between channels, and more fine-grained adjustments are made to each feature map's internal channels. The calculated intra-level similarity score will generate an intra-level weight , for each channel through the Softmax operation: ; The external weight assigns the same weighting value to each feature map channel, ensuring balanced overall weight distribution; while the intra-level weight Then, more delicate adjustment is made in the channel dimension, and each channel has different weights, so as to strengthen the uniqueness of each channel; Because And The meaning of expression is consistent, when Let , ; Finally, each feature map is multiplied by the calculated external weight and internal weight , so as to guide different feature layers to pay attention to non-overlapping areas, and get the adjusted feature map , wherein .

[0011] Preferably, in the step S4, the distribution loss DL enhances the differential attention of the enhanced model to different feature map regions, and quantifies the dispersion degree of the probability distribution between different feature maps; Firstly, the weighted input feature map is converted into a probability distribution by Softmax: ; Then, the mean square error MSE is used to measure the probability distribution difference between different feature maps, and the negative value of MSE is taken to encourage greater distribution difference: ; In the formula, , is the distribution loss, and denote the index of the multi-scale feature layer, and the value range is ; The greater the value, the more significant the probability distribution difference between feature maps.

[0012] Preferably, in the step S4, the generation process of the branch mask BM is as follows: Firstly, the pixel-level confidence weight of each density map is calculated by the weighting operation: ; In the formula, , is a very small constant; The weight reflects the importance of each branch in the task, and a larger weight value will enhance the influence of the branch, thereby increasing the difference between different branch density maps; Subsequently, the confidence weight is applied to the original feature map to obtain the weighted density map: ; Then, the threshold matrix is calculated by averaging the three weighted density maps pixel by pixel: ; After the global threshold is calculated, the weighted branch density maps are compared in the spatial dimension. When the pixel value is greater than the threshold, it is considered as the dominant area, and the mask matrix corresponding position is marked as 1; if less than the threshold, it is a non-dominant area, and the mask matrix corresponding position is marked as 0: ; In the formula, represents the branch mask of the th scale branch at the pixel position , is the coordinate of the pixel in two-dimensional space; Finally, each branch mask is multiplied by the global label to generate the branch label corresponding to each branch, ensuring that the label is accurately aligned with the local branch features. The specific calculation is as follows: ; In the formula, is the local label of the th branch, is the global label.

[0013] Preferably, in step S4, the branch density loss BDL is composed of the density loss , , the block-level classification loss , , the attention consistency loss , the distribution loss and the branch density loss . The specific definitions are as follows: Density loss: The density losses of the original image and the enhanced image are denoted as and , respectively. The optimized density prediction result is obtained by minimizing the mean square error MSE between the predicted density map filtered by the block-level classification mask and the real density map : ; In the formula, , ; Block-level classification loss: The block-level classification losses of the original image and the enhanced image are denoted as and , respectively. The binary cross entropy BCE is used for supervision: ; where, is the binary probability value classification result before binarization, is the real block-level classification label, , ; Attention consistency loss: To constrain the attention distribution consistency of the original image and the enhanced image in the attention memory bank , the difference between their attention distributions is calculated using mean square error (MSE); Let , represent the attention distribution of the original image and the enhanced image in , respectively: ; Distribution loss: For the probability distribution corresponding to the weighted feature map , the difference between the probability distributions of different branches is maximized using mean square error (MSE), and the loss is defined as: ; Branch density loss: Let the predicted density map filtered by the block-level classification mask in the i-th branch be , and its corresponding local label be , which is supervised using mean square error (MSE): ; ; where, ; The final total loss function is the weighted sum of each loss: ; where, , , and are weight parameters that balance different loss terms.

[0014] ​Compared with the prior art, the beneficial effects of the present application are: the present application adopts a feature decoupling and post-fusion strategy, which first decouples the scale features, then lets them predict the crowd density under each scale in the independent feature space, and finally optimizes the feature representation through maximum density fusion; wherein, the similarity attention mechanism SAM, multi-scale feature adjustment MFA and distribution loss DL are introduced to realize feature decoupling; at the same time, in order to reduce the error that may be introduced by the fusion strategy and improve the performance of the multi-scale branch, the multi-scale branch mask BM is calculated, and the corresponding multi-scale branch loss BDL is introduced; the experimental results show that WQCount achieves the most advanced performance in processing diversified crowd distribution and most unseen scenes, and can significantly improve the counting accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, illustrate the present application together with the embodiments thereof, and explain the present application, but do not constitute a limitation of the present application. In the drawings: Figure 1 is a flowchart of a multi-scale feature fusion method based on a WQCount model. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0017] As shown in Figure 1 , a multi-scale feature fusion method based on a WQCount model comprises: Step S1, select ShanghaiTech, UCF-QNRF and JHU-Crowd++ datasets, and divide the datasets into training sets, validation sets and test sets.

[0018] In the embodiments, the ShanghaiTech dataset is divided into two parts: SHA(A) and SHB(B), SHA(A) contains 300 training images and 182 test images, and SHB(B) contains 400 training images and 316 test images; UCF-QNRF contains 1201 training images and 334 test images; JHU-Crowd++ contains 4372 images, which are divided into 2722 training sets, 500 validation sets and 1600 test sets.

[0019] Step S2: Preprocess the ShanghaiTech, UCF-QNRF, and JHU-Crowd++ datasets.

[0020] In the embodiment, the preprocessing operations include: data augmentation, applying three photometric transformations: color jittering, Gaussian blur and sharpening; randomly cropping image blocks of size 320×320, and applying random horizontal flipping to both the original image and the enhanced image.

[0021] Step S3: construct a WQCount model.

[0022] In the embodiment, the WQCount model includes: Feature Processing Module (FPM) is used for multi-scale feature decoupling. It uses multi-scale feature adjustment (MFA) and similarity attention mechanism (SAM) to enable each feature layer to focus on its own unique and non-overlapping content. Distribution loss DL is used to constrain the spatial distribution and representation differences of different feature layers to achieve feature decoupling; Maximum density fusion (MDF) is used to fuse information by selecting the maximum density value in branch density maps of different scales; The branch density loss BDL and the branch mask BM are dynamically generated by the predicted density value, multiplied by the label map to obtain the branch label, and the branch label is then calculated with the branch density map for loss.

[0023] In this embodiment, the goal of the WQCount model is to train a parameter Neural Network , neural network Receive input image ,in, and are the height and width of the feature map, 3 is the number of channels, and outputs a density map of the same size , an estimated total number of Obtained by integrating all pixel values ​​of the density map: ; The real density map is usually based on the human head coordinate point annotation Generate, where each Coordinates representing the head position, true density map Calculated by applying a 2D Gaussian model to each head coordinate: ; Where, is the discrete Dirac function, For fixed variance Gaussian kernel of Supervised Neural Networks The commonly used loss function of the WQCount model is mean square error: .

[0024] Step S4, training and verifying the WQCount model through the training set and the verification set, and testing the test set by using the trained WQCount model.

[0025] In the embodiment, the WQCount model training process includes: performing photometric transformation on the original image to generate an enhanced image ; Subsequently, the feature extractor encodes and to generate features and respectively, wherein is a scale index, ; Based on the above features, the content error mask is calculated; At the same time, input and to the feature processing module FPM to obtain branch advantage mismatch features and ; Based on the branch advantage mismatch features and , the distribution loss DL is calculated; The output result of is applied to and to obtain the masked branch features; The above features are input to the attention memory bank after being adjusted in channel number by 1x1 convolution to be reconstructed to generate and ; The reconstructed features and are then input to the density head to perform crowd density prediction to obtain multi-scale branch density maps and ; The multi-scale branch density maps are input to the MDF to perform maximum density fusion to obtain density maps and ; At the same time, the highest level features and are input to the block-level classification head, whose output alleviates the ambiguity of the pixel-level labels by binary classification supervision, generating a predicted block-level classification map and ; and binarized and resized as masks to filter the estimated density maps and , and the fused density maps and ; regions with no head but with density values due to mislabeling are removed, resulting in branch density maps filtered by block-level classification masks and , and the final fused density maps and .

[0026] In the embodiment, in the feature processing module FPM, in order to strengthen the advantageous features of each level of feature map and weaken the non-advantageous features, a multi-scale feature adjustment MFA is designed, that is: For a given feature map input to the FPM: ; In the formula, is the number of channels; First, the average value in the spatial dimension and the maximum value in the spatial dimension are calculated, so as to obtain the global average vector and the global maximum vector : ; ; In the formula, represents taking the maximum value of the feature map in the spatial dimension (height and width ); In order to measure the advantageous features and the non-advantageous features, an advantage threshold vector is defined: ; The features greater than the threshold are regarded as advantageous features, and the features less than the threshold are regarded as non-advantageous features; In order to achieve the goal of enhancing the advantageous features and weakening the non-advantageous features, the original feature map is divided by the threshold to calculate the feature advantage weight matrix , and the feature value greater than the threshold has a weight greater than 1 after being divided by the threshold, and the feature value less than the threshold has a weight less than 1 after being divided by the threshold: ; By multiplying the calculated weight matrix with the original feature map , the output feature map is obtained, wherein , the adaptive enhancement of dominant features is finally realized, and the non-dominant features are effectively suppressed; The similarity between feature maps is calculated by the similar attention mechanism SAM, which encourages different feature layers to focus on non-overlapping regions, that is: For a given feature map: ; Optimization is carried out from two dimensions: inter-level similarity calculation and intra-level fine-grained interaction; In inter-level similarity calculation, the similarity between different feature maps is calculated, the corresponding channel elements of the feature maps are multiplied pixel by pixel, and the sum is calculated in the channel dimension to obtain their similarity scores : ; In the formula, is the sum in the channel dimension, and are the feature maps of the th and the th feature map in the th channel, is the Hadamard product, is the scaling factor, ; Then, the comprehensive similarity score of each feature map is calculated, which is the average of the similarity scores between adjacent feature maps: ; To guide different feature layers to focus on non-overlapping regions, the comprehensive similarity score is first negated, and then the negative similarity score is normalized by Softmax to generate external weights , which enhances the complementarity between feature layers by suppressing high similarity regions, and the formula is defined as: ; In the intra-level fine-grained interaction, the similarity between channels is measured by calculating the element-wise multiplication between channels, and the internal channels of each feature map are adjusted more finely; The calculated intra-level similarity score will generate the intra-level weight of each channel through the Softmax operation : ; Outer weights Assign the same weight value to each feature map channel to ensure balanced overall weight distribution; while inner weights Make more subtle adjustments in the channel dimension, with each channel having a different weight to strengthen the uniqueness of each channel; Since And Express the same meaning, when Let , ; Finally, multiply each feature map By the calculated outer weight And the inner weight , so as to guide different feature layers to pay attention to non-overlapping areas, and get the adjusted feature map , wherein .

[0027] In the embodiment, the distribution loss DL enhances the differential attention of the enhanced model to different feature map regions, and quantifies the dispersion degree of the probability distribution between different feature maps; First, convert the weighted input feature map To a probability distribution by Softmax: ; Then, use the mean square error MSE to measure the probability distribution difference between different feature maps, and take the negative value of MSE to encourage greater distribution difference: ; In the formula, , The distribution loss is And Indicate the index of the multi-scale feature layer, the value range is ; The greater the value, the more significant the probability distribution difference between feature maps. In the embodiment, the generation process of the branch mask BM is as follows: First, calculate the pixel-level confidence weight of each density map through the weighting operation: ; In the formula, , Is a very small constant; The weight reflects the importance of each branch in the task, and a larger weight value will enhance the influence of the branch, thereby increasing the difference between different branch density maps; Subsequently, the confidence weight is applied to the original feature map to obtain the weighted density map: ; Then, the threshold matrix is calculated by averaging the three weighted density maps pixel by pixel: ; After the global threshold is calculated, the weighted branch density maps are compared in the spatial dimension. When the pixel value is greater than the threshold, it is considered as a dominant region, and the corresponding position of the mask matrix is marked as 1; if it is less than the threshold, it is a non-dominant region, and the corresponding position of the mask matrix is marked as 0: ; In the formula, represents the branch mask of the i-th scale branch at the pixel position , is the coordinate of the pixel in two-dimensional space; Finally, each branch mask is multiplied by the global label to generate the branch label corresponding to each branch, ensuring that the label is accurately aligned with the local branch features. The specific calculation is as follows: ; In the formula, is the local label of the i-th branch, and is the global label. In the embodiment, the branch density loss BDL is composed of the density loss

[0028] , the block-level classification loss , the attention consistency loss , the distribution loss and the branch density loss . The specific definitions are as follows: Density loss: The density losses of the original image and the enhanced image are denoted as and respectively. The optimized density prediction result is obtained by minimizing the mean square error MSE between the predicted density map filtered by the block-level classification mask and the real density map : ; In the formula, , ; Block-level classification loss: The block-level classification losses of the original image and the enhanced image are denoted as and respectively, and are supervised by binary cross entropy BCE: ; In the formula, is the binary probability value classification result before binarization, is the true block-level classification label, , ; Attention consistency loss: This loss promotes each branch to learn complementary feature representations by maximizing the difference between the probability distributions of different feature layers, and is used to constrain the attention distribution consistency of the original image and the enhanced image in the attention memory bank The difference between the attention distributions of the original image and the enhanced image in the attention memory bank is calculated using the mean square error (MSE). Let , represent the attention distribution of the original image and the enhanced image in the attention memory bank, respectively. ; Distribution loss: For the probability distribution corresponding to the weighted feature map , , the difference between the probability distributions of different branches is maximized using the mean square error (MSE), and the loss is defined as: ; Branch density loss: This loss is used to minimize the difference between the predicted density map filtered by the block-level classification mask and the corresponding local label in each branch, thereby alleviating the errors that may be introduced by maximum density fusion. Let the predicted density map filtered by the block-level classification mask in the i-th branch be , and the corresponding local label be , and the mean square error (MSE) is used for supervision: ; In the formula, ; The final total loss function is the weighted sum of each loss: ; In the formula, , , and are weight parameters for balancing different loss terms.

[0029] Step S5: Use the tested WQCount model to predict the crowd density at each scale, and finally optimize the feature representation through maximum density fusion.

[0030] ​​VGG16-BN is used as the feature extractor in the WQCount model. The density regression head consists of a single convolutional layer with a 1×1 filter, while the classification head consists of 3×3 and 1×1 convolutional layers. Batch normalization is applied after all convolutional layers except the last one. For data augmentation, three photometric transformations are applied: color jittering, Gaussian blur, and sharpening. Image patches of size 320 × 320 are also randomly cropped, and random horizontal flipping is applied to both the original image and the enhanced image. We use AdamW as the optimizer and OneCycleLR as the learning rate scheduler. The maximum learning rate is set to 1e-3 and the maximum training cycle is 300. The model adopts a hierarchical memory structure to reconstruct different feature layers. The number of memory vectors M is uniformly set to 512, and the number of channels of shallow, middle, and deep features are respectively 、 、 , the content error threshold of all feature layers Set to 0.5, loss weight and Both are 10, distribution loss weight 5, branch density loss is 1 / 3.

[0031] Comparisons with state-of-the-art methods on SHA(A), SHB(B), and UCF-QNRF(Q) demonstrate that WQCount achieves superior performance on most transfer tasks. For example, in the A→B transfer task, WQCount achieves a MAE of 9.5 and an MSE of 16.1, representing reductions of 16.7% and 18.8%, respectively, compared to the next-best method, MPCount. This demonstrates WQCount's superior generalization capabilities when handling large differences between the source and target domains. WQCount also performs well in the B→Q transfer task, achieving 6.5% and 5.2% reductions, respectively, compared to the next-best method. In the Q→A transfer task, WQCount achieves the best MAE and the next-best MSE. In the Q→B transfer task, due to the large number of high-density scenes in the Q dataset, shallow features may learn more local density information. During inference, since the B dataset is mostly low-density with a large amount of background, density estimation based on decoupled shallow features is easily affected by background noise, resulting in poor performance.

[0032] To further verify the performance of the WQCount model, the experimental results on the scene-specific (SD ↔ SR) and weather-specific (SN ↔ FH) domains of the JHU-Crowd++ dataset show that the WQCount achieves significant improvement in the SN→FH and FH→SN transfer tasks, and reduces in both indicators. In the FH→SN transfer task, it reduces by 42.4% and 42.5% compared with the suboptimal method, which shows that it has better adaptability when dealing with scene transfer under different weather conditions. In the SD→SR and SR→SD transfer tasks, the WQCount achieves suboptimal performance.

[0033] Overall, the WQCount exhibits excellent performance in some cases, providing convincing evidence of the effectiveness of our design.

[0034] Finally, it should be noted that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent replacements to some technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-scale feature fusion method based on a WQCount model, characterized in that, The application relates to a method for crowd counting, comprising the following steps: Step S1, selecting ShanghaiTech, UCF-QNRF and JHU-Crowd++ datasets, and dividing the datasets into training sets, validation sets and test sets; Step S2, preprocessing the ShanghaiTech, UCF-QNRF and JHU-Crowd++ datasets; Step S3, constructing a WQCount model; Step S4, training and verifying the WQCount model through the training set and the validation set, and testing the test set by using the trained WQCount model; Step S5, predicting the crowd density under each scale by using the tested WQCount model, and finally optimizing the feature representation through maximum density fusion.

2. The multi-scale feature fusion method based on the WQCount model according to claim 1, characterized in that, In the step S2, the preprocessing operation comprises data enhancement, application of three kinds of photometric transformations: color jittering, Gaussian blur and sharpening, random cropping of an image block with a size of 320*320, and application of random horizontal flipping to the original image and the enhanced image.

3. The multi-scale feature fusion method based on WQCount model according to claim 1, characterized in that, In the step S3, the WQCount model comprises: A feature processing module FPM for multi-scale feature decoupling, which uses a multi-scale feature adjustment MFA and a similar attention mechanism SAM to enable each feature layer to focus on respective good and non-overlapping contents; A distribution loss DL for constraining the spatial distribution and representation difference of different feature layers, so as to realize feature decoupling; A maximum density fusion MDF for fusing information by selecting the maximum density value in the density map of different scale branches; A branch density loss BDL and a branch mask BM, which are dynamically generated by the predicted density value, multiplied by the label map to obtain a branch label, and then the branch label is subjected to loss calculation with the branch density map.

4. The multi-scale feature fusion method based on WQCount model according to claim 3, characterized in that, In the step S3, the target of the WQCount model is to train a neural network with parameters The neural network receives an input image , where and are the height and width of the feature map respectively, and 3 is the number of channels, and outputs a density map of the same size , the estimated value of the total number of people is obtained by integrating all the pixel values of the density map. ； The real density map is usually based on human head coordinate point annotation generated, wherein each coordinates representing head positions, the real density map is calculated by applying a two-dimensional Gaussian model to each head coordinate: ； wherein is a discrete Dirac function, is a fixed variance Gaussian kernel; Supervised neural networks The common loss function for supervised neural networks is mean squared error: 。 5. The multi-scale feature fusion method based on WQCount model according to claim 4, characterized in that, In the step S4, the WQCount model training comprises: performing a photometric transformation on the original image to generate an enhanced image ; Subsequently, the feature extractor encodes the feature and to generate features and respectively, where is a scale index, ; Based on the above features, the content error mask is calculated ; At the same time, the and input to the feature processing module FPM results in features focusing on respective strengths and on non-overlapping features of the strengths of different regions and ; Based on the feature of branch dominance non-coincidence and The distribution loss DL is calculated. Applying the output result of to and , the masked branch features are obtained. The above features are input into the attention memory bank after being adjusted in channel number by 1x1 convolution reconstruction generation is performed and ; Reconstruction features And Subsequently input to the density head Carrying out crowd density prediction, obtaining multi-scale branch density maps And ; The multi-scale branch density map is input into the MDF for maximum density fusion to obtain a density map and ; Meanwhile, the highest level features encoded by the feature extractor and are input to a block-level classification head, whose output alleviates pixel-level label ambiguity by binary classification supervision, generating a predicted block-level classification map and ; and binarized and resized as a mask to filter the estimated density map and , and the fused density map and ; remove the regions without head but with density values due to the misdirection of the label, finally generate the branch density map filtered by the block-level classification mask and and the final fused density map and .

6. The multi-scale feature fusion method based on the WQCount model according to claim 5, characterized in that, In the step S4, in the feature processing module FPM, in order to strengthen the advantageous features of each level feature map and weaken the non-advantageous features, a multi-scale feature adjustment MFA is designed, that is: For a given feature map input into the FPM: ； In the formula, is the number of channels; First, the average over the spatial dimensions and the maximum over the spatial dimensions are computed, resulting in a global average vector and a global maximum vector respectively. ； ； wherein denotes the feature map the maximum value in the spatial dimension, i.e. height and width ; To measure the dominant and non-dominant features, a dominant threshold vector is defined : ； The features greater than the threshold value are regarded as advantageous features, and the features less than the threshold value are regarded as non-advantageous features; divide the original feature map by a threshold value divide the original feature map by a threshold value to calculate a feature advantage weight matrix a feature value greater than the threshold value has a weight greater than 1 after being divided by the threshold value, and a feature value less than the threshold value has a weight less than 1 after being divided by the threshold value ； The weight matrix obtained by With the original feature map Multiply to get the output feature map ,in, , ultimately achieving adaptive enhancement of dominant features and effectively suppressing non-dominant features; The similarity between the feature maps is calculated through a similar attention mechanism SAM, and different feature layers are encouraged to focus on non-overlapping regions, that is: For a given feature map: ； Optimization is performed from two dimensions: inter-layer similarity calculation and intra-layer fine-grained interaction; In the inter-level similarity calculation, the similarity between different feature maps is calculated, the corresponding channel elements of the feature maps are multiplied pixel by pixel, and summed in the channel dimension to obtain their similarity scores : ； wherein is the sum over the channel dimension, and are the first and second feature maps at the channel, is the Hadamard product, is a scaling factor, ; Then, a comprehensive similarity score for each feature map is calculated , which is the average of the similarity scores between adjacent feature maps: ； To guide different feature layers to focus on non-overlapping regions, first, the integrated similarity score is taken negative, and then the negative similarity score is normalized by Softmax to generate external weights , This operation strengthens the complementarity between feature layers by suppressing high similarity regions, and the formula is defined as: ； In the intra-layer fine-grained interaction, the similarity between the channels is measured by calculating the element-by-element multiplication of the channels, and each feature map is subjected to more fine adjustment of the internal channels; The computed inner similarity scores will be passed through a Softmax operation to generate inner weights for each channel , : ； Outer layer weight Assign the same weight value to each feature map channel to make the overall weight distribution balanced; Inner layer weight Fine adjustment in the channel dimension, each channel has a different weight, and the uniqueness of each channel is strengthened; Because and The meaning of the expression is consistent when is , ; Finally, each feature map is multiplied by the calculated external weight and internal weight , so as to guide different feature layers to pay attention to non-overlapping areas, to obtain an adjusted feature map , wherein, .

7. The multi-scale feature fusion method based on WQCount model according to claim 6, characterized in that, In the step S4, the distribution loss DL enhances the difference in attention of the model to different feature map regions, and quantifies the dispersion degree of the probability distribution between different feature maps; First, the weighted input feature maps are converted to a probability distribution by Softmax: ​ ； Then, the mean square error MSE is used to measure the probability distribution difference between different feature maps, and the negative value of the MSE is taken to encourage greater distribution difference: ； In the formula, , is a distribution loss, and indicates the index of the multi-scale feature layer, and the value range is ; The larger the value is, the more significant the probability distribution difference between the feature maps is.

8. The multi-scale feature fusion method based on WQCount model according to claim 7, characterized in that, In the step S4, the generation process of the branch mask BM is as follows: First, the pixel-level confidence weight of each density map is calculated through a weighting operation: ； wherein , is a very small constant; The weight reflects the importance of each branch in the task; Subsequently, the confidence weight is applied to the original feature map to obtain a weighted density map: ； Then, the threshold matrix is calculated by averaging the three weighted density maps pixel by pixel: ； After the global threshold is calculated, the spatial dimension of each weighted branch density map is compared; When the pixel value is greater than the threshold, it is considered as the dominant area, and the corresponding position of the mask matrix is marked as 1; if it is less than the threshold, it is considered as the non-dominant area, and the corresponding position of the mask matrix is marked as 0: ； wherein represents the branch mask of the th scale branch at pixel position , and is the coordinate of the pixel in two-dimensional space. Finally, each branch mask is multiplied by the global label to generate the branch label corresponding to each branch, ensuring that the label is accurately aligned with the local branch features. The specific calculation is as follows: ； wherein is a local tag for the first branch, is a global tag.

9. The multi-scale feature fusion method based on WQCount model according to claim 8, characterized in that, The branch density loss BDL in the step S4 is composed of the density loss , , the block-level classification loss , , the attention consistency loss , the distribution loss and the branch density loss . Density loss: The density loss of the original image and the enhanced image are denoted as and By minimizing the mean square error MSE of the predicted density map filtered by the block-level classification mask and the real density map The optimized density prediction result: ； In the formulae, , ; Block-level classification loss: The block-level classification loss of the original image and the enhanced image are denoted as and respectively, supervised by binary cross-entropy (BCE). ； In the formula, is the classification result of the probability value before binarization, is the real block-level classification label, , ; Attention consistency loss: To constrain the attention distribution consistency of the original image and the enhanced image in the attention memory bank , the mean square error (MSE) is used to calculate the difference between the attention distributions. with , represent the attention distribution of the original image and the enhanced image in , respectively: ； Distribution loss: For the weighted feature maps The corresponding probability distribution Maximizing the difference between the different branch probability distributions with mean square error, the loss is defined as: ； Branch density loss: Set the first branch, the prediction density map filtered by the block-level classification mask is , and the corresponding local label is , and the mean square error MSE is used for supervision: ； In the formulae, ; The final total loss function is the weighted sum of each loss: ； wherein, , , and are weight parameters balancing different loss terms.

Images