Crowd counting method and system based on attention fusion and dual-branch mutual supervision

Abstract

The present invention discloses a crowd counting method based on attention fusion and dual-branch mutual supervision, including extracting feature maps from input crowd images, encoding and splicing them in horizontal and vertical directions to obtain fusion features, generating attention maps and multiplying them with feature maps to obtain intermediate output features, and then obtaining two spatial attention maps from the intermediate output features and fusion with the feature maps to obtain attention fusion output features, and finally predicting through convolutional neural network branches and Transformer branches, and directly using the true value density map corresponding to the point annotation to supervise the entire network during the training process; and forming a supervision signal based on the prediction results of the two branches to perform detailed supervision on some suspected mislabeled data areas, and the remaining areas are learned based on the original real data. The present invention also discloses a crowd counting system that implements the method, which can effectively reduce the impact of noise annotation and improve accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a crowd counting method and system. Background Art

[0002] Crowd counting uses machine learning algorithms to automatically analyze the crowd in an image or video and provide an estimate of its number. Due to the scale change problem caused by the shooting angle of the surveillance camera, the heads of people at different distances in the same scene appear different sizes in the image. Generally speaking, the heads of people near the image appear larger in the image because they are closer to the camera, while the heads of people far away appear smaller in the image because they are farther away. This drastic scale change makes it difficult for traditional crowd counting algorithms to adapt to heads of various sizes, thereby reducing the accuracy and reliability of counting. In addition, the change in head size may also cause the counting algorithm to generate false positives or false negatives when detecting small-sized heads in the distance, especially when the head density is high and people occlude each other. At the same time, in the process of constructing crowd datasets, accurately annotating the location and size information of the target is crucial to training an effective counting model. Especially in scenes with dense crowds, mutual occlusion between people, complex human postures, and environmental factors often make the target boundaries unclear, making it extremely difficult to determine the exact location of individuals. In this scenario, labelers have to rely on personal experience and intuition to label, which can easily introduce subjective judgment errors, which directly affect the accuracy and generalization ability of model learning. In complex scenarios, counting errors caused by scale changes and noisy labels seriously interfere with the accuracy of counting methods, thus adversely affecting their performance.

[0003] In order to solve the scale change problem in crowd images, the existing technology adopts a crowd counting method based on CNN, which has some key problems that need to be solved. First, the continuous nature of scale changes in images means that it is difficult to completely solve the challenges brought by drastic scale changes, whether it is a multi-scale network structure or a multi-resolution processing strategy. Secondly, relying solely on a single convolutional layer to generate a crowd density map is prone to introduce counting errors and fails to fully mine the effective features in the image. Most current counting methods focus on fusing high-level semantic information, but often ignore the underlying details in the process, which may lead to the loss of some features and insufficient supplementation of detailed information in the density map, thereby limiting the improvement of counting performance in complex scenarios. Summary of the invention

[0004] In view of the above-mentioned defects of the prior art, the present invention provides a crowd counting method based on attention fusion and dual-branch mutual supervision, which solves the problem of low counting performance caused by drastic changes in crowd scale in the scene and noise annotation in the crowd data set. The present invention also provides a crowd counting system for implementing the method.

[0005] The technical solution of the present invention is as follows:

[0006] A crowd counting method based on attention fusion and dual-branch mutual supervision includes the following steps:

[0007] Step S1, preprocessing the acquired crowd image to obtain a true value density map;

[0008] Step S2: construct an attention fusion and dual-branch mutual supervision network;

[0009] Step S3, initializing the parameters of the attention fusion and dual-branch mutual supervision network;

[0010] Step S4: input the acquired crowd image and the true value density map obtained by preprocessing into the attention fusion and dual-branch mutual supervision network to complete the forward propagation;

[0011] Step S5, based on the mutual supervision strategy, the output obtained in step S4 is used to train the attention fusion and dual-branch mutual supervision network using a gradient descent algorithm;

[0012] Step S6, iterating step S4 and step S5 to a specified number of times;

[0013] Step S7, obtaining a crowd density map of the crowd image to be counted through iteratively updated attention fusion and dual-branch mutual supervision network, and obtaining an estimated number of people according to the crowd density map;

[0014] The attention fusion and dual-branch mutual supervision network extracts features from the input crowd image to obtain a feature map, the feature map is encoded and spliced ​​in the horizontal and vertical directions to obtain a fusion feature, the fusion feature generates an attention map and multiplies it with the feature map to obtain an intermediate output feature, the intermediate output feature is multiplied with the probability distribution map of the feature map, and the feature map is multiplied with the probability distribution map of the intermediate output feature to obtain two spatial attention maps, the two spatial attention maps are fused with the feature map to obtain an attention fusion output feature, and the attention fusion output feature enters the convolutional neural network branch to generate a convolution prediction result. , the attention fusion output features enter the Transformer branch to generate the Transformer prediction result ;

[0015] The mutual supervision strategy includes, in sequence, directly using the true density map corresponding to the point annotations to supervise the entire network; using the weighted combination of the Transformer prediction results and the true density map as the supervision signal of the convolutional neural network branch; using the weighted combination of the convolution prediction results and the true density map as the supervision signal of the Transformer branch, refining the supervision of some data areas suspected of being mislabeled, and learning the remaining areas based on the original real data.

[0016] Furthermore, the feature map obtained by extracting features from the input crowd image is extracted using a VGG-16 network with the fully connected layer removed.

[0017] Furthermore, the feature maps are encoded in the horizontal and vertical directions and spliced ​​to obtain the fused features by encoding the feature maps in the horizontal and vertical directions through global average pooling, and then splicing along the height of the image and passing Convolution obtains fused features.

[0018] Furthermore, the attention map is generated by the fusion feature and multiplied with the feature map to obtain the intermediate output feature. The intermediate output feature is multiplied with the probability distribution map of the feature map and the feature map is multiplied with the probability distribution map of the intermediate output feature to obtain two spatial attention maps. The fusion feature is generated by two Sigmoid functions. The two attention maps are then multiplied with the feature map to obtain the intermediate output feature. The intermediate output feature is converted by global average pooling and Softmax function to obtain the corresponding probability distribution map. The feature map is firstly After convolution, global average pooling and Softmax function conversion are performed to obtain the corresponding probability distribution map.

[0019] Furthermore, the two spatial attention maps are fused with the feature map to obtain the attention fusion output feature, which is to add the two spatial attention maps at corresponding positions, map them through the Sigmoid function, and then multiply the elements with the feature map to obtain the attention fusion output feature.

[0020] Furthermore, the convolutional neural network branch includes a hole convolution layer and a convolution layer, the Transformer branch includes a Transformer encoding layer and a convolution layer, and the convolutional neural network branch has the same convolutional layer structure as the Transformer branch.

[0021] Furthermore, the mutual supervision strategy includes an initial loss function and a mid-to-late loss function. The initial loss function is a function of the mean square error between the primary density map and the corresponding true value density map, the convolution prediction result, Mean square error with the true value density map and Transformer prediction results The mid- and late-stage loss function is composed of the convolution prediction result Mean squared error of the supervision signal with the convolutional neural network branch and the Transformer prediction results It is composed of the mean square error of the supervision signal of the Transformer branch.

[0022] Furthermore, the supervisory signal of the convolutional neural network branch is

[0023] ,

[0024] The supervision signal of the Transformer branch is

[0025] ,

[0026] in is the weighting coefficient, is the selection mask of the mislabeled areas of the crowd image of the Transformer branch, The selection mask of the mislabeled regions of the crowd image for the convolutional neural network branch, is the true value density map.

[0027] Further, is the set maximum value, t is the current number of iterations, and T is the set total number of iterations.

[0028] Another technical solution of the present invention is a crowd counting system based on attention fusion and dual-branch mutual supervision, comprising the following modules:

[0029] A preprocessing module, used for preprocessing the acquired crowd image to obtain a true value density map;

[0030] Network model module, used to build attention fusion and dual-branch mutual supervision networks;

[0031] Initialization module, used to initialize the parameters of attention fusion and dual-branch mutual supervision network;

[0032] The iterative training module is used to input the acquired crowd image and the true value density map obtained by preprocessing into the attention fusion and dual-branch mutual supervision network to complete the forward propagation; and the output is based on the mutual supervision strategy, and the gradient descent algorithm is used to iteratively train the attention fusion and dual-branch mutual supervision network to a specified number of times;

[0033] and an output module, for obtaining a crowd density map from the crowd image to be counted through iteratively updated attention fusion and a dual-branch mutual supervision network, and obtaining an estimated number of people based on the crowd density map;

[0034] The attention fusion and dual-branch mutual supervision network extracts features from the input crowd image to obtain a feature map, the feature map is encoded and spliced ​​in the horizontal and vertical directions to obtain a fusion feature, the fusion feature generates an attention map and multiplies it with the feature map to obtain an intermediate output feature, the intermediate output feature is multiplied with the probability distribution map of the feature map, and the feature map is multiplied with the probability distribution map of the intermediate output feature to obtain two spatial attention maps, the two spatial attention maps are fused with the feature map to obtain an attention fusion output feature, and the attention fusion output feature enters the convolutional neural network branch to generate a convolution prediction result. , the attention fusion output features enter the Transformer branch to generate the Transformer prediction result ;

[0035] The mutual supervision strategy includes, in sequence, directly using the true density map corresponding to the point annotations to supervise the entire network; using the weighted combination of the Transformer prediction results and the true density map as the supervision signal of the convolutional neural network branch; using the weighted combination of the convolution prediction results and the true density map as the supervision signal of the Transformer branch, refining the supervision of some data areas suspected of being mislabeled, and learning the remaining areas based on the original real data.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] The present invention integrates contextual information from different scales and spatial directions through a multi-scale cross-space attention mechanism, thereby enhancing the model's ability to recognize objects with large scale differences in the scene.

[0038] The present invention introduces a dual-branch mutual supervision network, and adopts a selective refinement supervision strategy to cross-supervise and correct the prediction of the other branch, thereby reducing the counting performance interference caused by background noise in the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the crowd counting method based on attention fusion and dual-branch mutual supervision of the present invention.

[0040] Figure 2 This is a structural diagram of the attention fusion and dual-branch mutual supervision network of the present invention.

[0041] Figure 3 This is a structural diagram of the multi-scale attention module of the present invention.

[0042] Figure 4 This is a schematic diagram of the visualization of the predicted density map of the present invention on ShanghaiTech.

[0043] Figure 5Schematic diagram of visualization of the predicted density map of the present invention on UCF-QNRF. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with the embodiments, but are not intended to limit the present invention.

[0045] The crowd counting system based on attention fusion and dual-branch mutual supervision of the embodiment of the present invention includes the following modules:

[0046] A preprocessing module, used for preprocessing the acquired crowd image to obtain a true value density map;

[0047] Network model module, used to build attention fusion and dual-branch mutual supervision networks;

[0048] Initialization module, used to initialize the parameters of attention fusion and dual-branch mutual supervision network;

[0049] The iterative training module is used to input the acquired crowd image and the true value density map obtained by preprocessing into the attention fusion and dual-branch mutual supervision network to complete the forward propagation; and the output is based on the mutual supervision strategy, and the gradient descent algorithm is used to iteratively train the attention fusion and dual-branch mutual supervision network to a specified number of times;

[0050] And an output module, which is used to obtain a crowd density map from the crowd image to be counted through iteratively updated attention fusion and a dual-branch mutual supervision network, and obtain the estimated number of people based on the crowd density map

[0051] Please combine Figure 1 As shown, the specific crowd counting method implemented by the crowd counting system includes the following steps:

[0052] Step S1: Preprocess the crowd images collected on the ShanghaiTech and UCF-QNRF datasets. The specific method is as follows:

[0053] Step S11: adjusting the collected crowd image according to a specified size range, and saving the corresponding head coordinate data.

[0054] Step S12: First, calculate the Euclidean distance between each point in the crowd image, then find the first three closest distances between each point and all other points, and calculate the average of these closest distances. Add the calculated average distance to the coordinate information of the head point, so that the information of each point is expanded from two dimensions to three dimensions.

[0055] Step S13: Create a matrix with all elements set to 0, which is the same as the resolution of the corresponding image, and set the elements of the corresponding positions of the matrix to 1 according to the coordinate information of the head point. Then perform Gaussian filtering on the position matrix to smooth the point distribution and simulate the distribution of crowd density in the real scene to obtain the true value density map. .

[0056] Step S14: Randomly enhance the crowd image and the true value density map by scaling, cropping, horizontal flipping and other operations, and the enhancement operations performed on a corresponding group of crowd images and true value density maps are the same.

[0057] Step S2: Construct an attention fusion and dual-branch mutual supervision network, which includes feature extraction, attention fusion, and dual-branch mutual supervision. Figure 2 and Figure 3 As shown, the processing procedures of each part are introduced below.

[0058] The feature extraction part uses the VGG-16 network with the fully connected layer removed for each crowd image. To extract preliminary features, four feature maps with different resolutions and channel numbers are generated, namely , , as well as ; Use the features of the first three stages of VGG-16 to predict the primary density maps respectively, and mark these primary density maps as , and , whose height and width are 1 / 8, 1 / 16 and 1 / 32 of the original input size respectively.

[0059] The multi-scale attention module in the attention fusion part combines each feature map, , divided along the channel dimension into M sub-feature graphs, each sub-feature graph The shape is , so the entire feature map Can be redefined as a feature map group .Will Each sub-feature map in Using two 1D global average pooling, the channels are encoded in the horizontal and vertical directions respectively to generate horizontal sub-feature maps. and vertical sub-feature map , ;Will and The images are stitched together along their height and through a Convolutional layer processing to generate fusion sub-features , thereby fusing information from different spatial dimensions. Fusion sub-features It is split and processed by two Sigmoid functions to generate two attention maps. These two attention maps are combined with the sub-feature map of the original group Multiply and reweight, adjust the weights of different channels and ensure the accurate preservation of spatial structure information, and finally get , while retaining the original feature information, the feature expression ability of the model is enhanced. pass After convolution, we get the sub-feature map ;Will and Perform global average pooling processing respectively and convert it into two two-dimensional probability distribution graphs through the Softmax function and ; The probability distribution diagram With through After convolution, we get the sub-feature map Perform matrix multiplication. Similarly, the probability distribution graph With feature map Matrix multiplication is also performed to obtain two spatial attention maps. The two spatial attention maps are added at corresponding positions and mapped by the Sigmoid function to the original input sub-feature map. Perform element multiplication to finally obtain the output sub-feature map that integrates spatial information features , the output feature map group is . The feature map The output feature map obtained as well as Upsample and then compare with the feature map The output feature map obtained Add together to get the attention fusion output feature .

[0060] The dual-branch mutual supervision part is the attention fusion output feature Fusion of local detail information and global information. The specific method is to output the feature of attention fusion , perform the following operations: first, the attention fusion output feature The convolutional neural network branch of the input is used to better obtain local information. A 4-layer dilated convolution structure with ReLU activation function is used, and the convolution kernel size of each layer is , and the expansion rate is set to 2, which enables the network to increase the receptive field without losing resolution and deeply extract more refined local features. Then it passes through 4 convolutional layers, of which the first 3 layers use Convolution, the last layer uses Convolution, finally generating convolution prediction results At the same time, the attention fusion output features Input Transformer branch to better obtain global information. Attention fusion output features After being processed by the four Transformer encoding layers of the Transformer branch, and then by the four convolutional layers with the same structure as the CNN branch, the Transformer prediction result is obtained. .

[0061] Step S3: Initialize the parameters of the attention fusion and dual-branch mutual supervision network. The Adam optimizer is used and the initial learning rate is set to , the weight decays to , the total number of training rounds is set to 2000. In the dual-branch mutual supervision learning, the SHA dataset Set to 0.1, the other datasets is 0.05; for the QNRF dataset, Set to 0.5. as well as For the meaning of , please refer to the mutual supervision strategy in step S5.

[0062] Step S4: Input the crowd image and crowd density map preprocessed in step S1 into the network to complete forward propagation.

[0063] Step S5: The forward output crowd density map and the randomly enhanced true value density map are trained using the gradient descent algorithm based on the loss function determined by the mutual supervision strategy to train the attention fusion and dual-branch mutual supervision network. The mutual supervision strategy performs detailed supervision on specific areas that may contain mislabeling, while keeping the remaining areas determined by the original real data annotation. The loss function is specifically determined in the following way.

[0064] Step S51: In the initial stage of training, since the predicted density map is inaccurate, the true value density map corresponding to the point annotation is directly used. The entire network is supervised, and on this basis, the mean square error is used to measure the primary density map , and The corresponding true value density map , and The error between As shown in formula (1):

[0065] (1)

[0066] Further combined with the convolution prediction results And Transformer prediction results , the total loss at the initial stage of training is expressed as formula (2), that is,

[0067] (2)

[0068] Step S52: Perform fine-grained supervision on special areas that may contain incorrect annotations. The weighted combination of the prediction results of the two branches and the true density map is used as a supervision signal for branch training. The weighting coefficient It will grow from zero to a maximum value as the training progresses , so that the attention fusion and dual-branch mutual supervision network gradually increase their reliance on their own prediction results in the middle and late stages of the training process, thereby optimizing the training effect. The definition of is shown in formula (3):

[0069] (3)

[0070] in, The value of is dynamically adjusted by formula (4):

[0071] (4)

[0072] In this formula, represents the current number of training iterations, and represents the set total number of training times;

[0073] Since the mutual supervision strategy is a selective refinement supervision strategy, only some suspected mislabeled data areas are refined and supervised, while the remaining areas are learned based on the original real data. First, the difference between the true value density map and the Transformer prediction result is calculated. , in order to evaluate the error size of each pixel. Further set a threshold It is a value that increases from 0 to the set value according to the training progress. The ratio value of , n is the number of pixels in the crowd image, and then sort them in descending order to select The minimum value among the largest differences is set as , to define those areas where the error exceeds the threshold. On this basis, a selection mask is defined for the Transformer branch , which is composed of the indicator function I, as shown in formula (5):

[0074] (5)

[0075] Based on this, the final supervision definition for the Transformer branch is shown in formula (6):

[0076] (6)

[0077] Where * means element-wise multiplication.

[0078] Fine-grained supervision value for CNN branches The definition of is shown in formula (7):

[0079] (7)

[0080] in, The value of is dynamically adjusted through formula (4).

[0081] Since the mutual supervision strategy is a selective refinement supervision strategy, only some suspected mislabeled data areas are refined and supervised, while the remaining areas are learned based on the original real data. First, the difference between the true value density map and the CNN prediction result is calculated. , in order to evaluate the error size of each pixel. Further set a threshold , It is a value that increases from 0 to the set value according to the training progress. The ratio value of , n is the number of pixels in the crowd image, and then sort them in descending order to select The minimum value among the largest differences is set as , to define those areas where the error exceeds the threshold. On this basis, a selection mask is defined for the CNN branch , which is indicated by the indicator function The composition is as shown in formula (8):

[0082] (8)

[0083] Based on this, the final supervision definition for the CNN branch is shown in formula (9):

[0084] (9)

[0085] Where * means element-wise multiplication.

[0086] Finally, the overall optimization loss function in the late stage of training is expressed as formula (10):

[0087] (10)

[0088] The attention fusion and dual-branch mutual supervision network are trained through the above different loss functions.

[0089] Step S6: Iterate steps S5 and S6 to a specified number of times.

[0090] Step S7: Obtain a crowd density map to estimate the number of people. The specific method is: generate a crowd density map by iteratively updating the crowd image through attention fusion and a dual-branch mutual supervision network, and sum the pixels of the crowd density map to determine the number of people in the image. The predicted density map of the ShanghaiTech dataset is visualized as follows: Figure 4 The prediction density map corresponding to the UCF-QNRF dataset is visualized as follows Figure 5 shown.

[0091]

[0092] The performance of the present invention on the commonly used crowd counting dataset ShanghaiTech is shown in the table above. It has excellent performance in both the mean absolute error (MAE) and mean squared error (MSE).

[0093]

[0094] The performance of the present invention on the commonly used crowd counting dataset UCF-QNRF is shown in the table above. It has excellent performance in both the mean absolute error (MAE) and mean squared error (MSE).

Claims

1. A crowd counting method based on attention fusion and dual-branch mutual supervision, characterized in that: The following steps are involved: Step S1, preprocessing the acquired crowd image to obtain a true value density map; Step S2: construct an attention fusion and dual-branch mutual supervision network; Step S3, initializing the parameters of the attention fusion and dual-branch mutual supervision network; Step S4: input the acquired crowd image and the true value density map obtained by preprocessing into the attention fusion and dual-branch mutual supervision network to complete the forward propagation; Step S5, based on the mutual supervision strategy, the output obtained in step S4 is used to train the attention fusion and dual-branch mutual supervision network using a gradient descent algorithm; Step S6, iterating step S4 and step S5 to a specified number of times; Step S7, obtaining a crowd density map of the crowd image to be counted through iteratively updated attention fusion and dual-branch mutual supervision network, and obtaining an estimated number of people according to the crowd density map; The attention fusion and dual-branch mutual supervision network extracts features from the input crowd image to obtain a feature map, the feature map is encoded and spliced ​​in the horizontal and vertical directions to obtain a fusion feature, the fusion feature generates an attention map and multiplies it with the feature map to obtain an intermediate output feature, the intermediate output feature is multiplied with the probability distribution map of the feature map, and the feature map is multiplied with the probability distribution map of the intermediate output feature to obtain two spatial attention maps, the two spatial attention maps are fused with the feature map to obtain an attention fusion output feature, and the attention fusion output feature enters the convolutional neural network branch to generate a convolution prediction result. , the attention fusion output features enter the Transformer branch to generate the Transformer prediction result ; The mutual supervision strategy includes, in sequence, directly using the true density map corresponding to the point annotations to supervise the entire network; using the weighted combination of the Transformer prediction results and the true density map as the supervision signal of the convolutional neural network branch; using the weighted combination of the convolution prediction results and the true density map as the supervision signal of the Transformer branch, refining the supervision of some data areas suspected of being mislabeled, and learning the remaining areas based on the original real data.

2. The crowd counting method based on attention fusion and dual-branch mutual supervision according to claim 1 is characterized in that: The feature map obtained by extracting features from the input crowd image is extracted using a VGG-16 network with the fully connected layer removed.

3. The crowd counting method based on attention fusion and dual-branch mutual supervision according to claim 1 is characterized in that: The fusion feature is obtained by encoding and splicing the feature map in the horizontal and vertical directions, and then splicing along the height of the image and passing through the global average pooling. Convolution obtains fused features.

4. The crowd counting method based on attention fusion and dual-branch mutual supervision according to claim 1 is characterized in that: The attention map is generated by the fusion feature and multiplied with the feature map to obtain the intermediate output feature. The intermediate output feature is multiplied with the probability distribution map of the feature map and the feature map is multiplied with the probability distribution map of the intermediate output feature to obtain two spatial attention maps. The fusion feature is generated by two Sigmoid functions. The two attention maps are then multiplied with the feature map to obtain the intermediate output feature. The intermediate output feature is converted by global average pooling and Softmax function to obtain the corresponding probability distribution map. The feature map is first passed through After convolution, global average pooling and Softmax function conversion are performed to obtain the corresponding probability distribution map.

5. The crowd counting method based on attention fusion and dual-branch mutual supervision according to claim 1 is characterized in that: The two spatial attention maps are fused with the feature map to obtain the attention fusion output feature, which is to add the two spatial attention maps at corresponding positions, map them through the Sigmoid function, and then multiply them element-wise with the feature map to obtain the attention fusion output feature.

6. The crowd counting method based on attention fusion and dual-branch mutual supervision according to claim 1 is characterized in that: The convolutional neural network branch includes a hole convolution layer and a convolution layer, the Transformer branch includes a Transformer encoding layer and a convolution layer, and the convolutional neural network branch has the same convolutional layer structure as the Transformer branch.

7. The crowd counting method based on attention fusion and dual-branch mutual supervision according to claim 1 is characterized in that: The mutual supervision strategy includes an initial loss function and a mid-to-late loss function. The initial loss function is a function of the mean square error between the primary density map and the corresponding true value density map, the convolution prediction result, and the Mean square error with the true value density map and Transformer prediction results The mid- and late-stage loss function is composed of the convolution prediction result Mean squared error of the supervision signal with the convolutional neural network branch and the Transformer prediction results It is composed of the mean square error of the supervision signal of the Transformer branch.

8. The crowd counting method based on attention fusion and dual-branch mutual supervision according to claim 7 is characterized in that: The supervisory signal of the convolutional neural network branch is ， The supervision signal of the Transformer branch is ， in is the weighting coefficient, is the selection mask of the mislabeled areas of the crowd image of the Transformer branch, The selection mask of the mislabeled regions of the crowd image for the convolutional neural network branch, is the true value density map.

9. The crowd counting method based on attention fusion and dual-branch mutual supervision according to claim 8 is characterized in that: is the set maximum value, t is the current number of iterations, and T is the set total number of iterations.

10. A crowd counting system based on attention fusion and dual-branch mutual supervision, characterized in that: Includes the following modules: A preprocessing module, used for preprocessing the acquired crowd image to obtain a true value density map; Network model module, used to build attention fusion and dual-branch mutual supervision networks; Initialization module, used to initialize the parameters of attention fusion and dual-branch mutual supervision network; Iterative training module, which is used to input the acquired crowd images and the pre-processed true value density map into the attention fusion and dual-branch mutual supervision network to complete the forward propagation; The output is based on the mutual supervision strategy, and the attention fusion and dual-branch mutual supervision network are iteratively trained to the specified number of times using the gradient descent algorithm; and an output module, for obtaining a crowd density map from the crowd image to be counted through iteratively updated attention fusion and a dual-branch mutual supervision network, and obtaining an estimated number of people based on the crowd density map; The attention fusion and dual-branch mutual supervision network extracts features from the input crowd image to obtain a feature map, the feature map is encoded and spliced ​​in the horizontal and vertical directions to obtain a fusion feature, the fusion feature generates an attention map and multiplies it with the feature map to obtain an intermediate output feature, the intermediate output feature is multiplied with the probability distribution map of the feature map, and the feature map is multiplied with the probability distribution map of the intermediate output feature to obtain two spatial attention maps, the two spatial attention maps are fused with the feature map to obtain an attention fusion output feature, and the attention fusion output feature enters the convolutional neural network branch to generate a convolution prediction result. , the attention fusion output features enter the Transformer branch to generate the Transformer prediction result ; The mutual supervision strategy includes, in sequence, directly using the true density map corresponding to the point annotations to supervise the entire network; using the weighted combination of the Transformer prediction results and the true density map as the supervision signal of the convolutional neural network branch; using the weighted combination of the convolution prediction results and the true density map as the supervision signal of the Transformer branch, refining the supervision of some data areas suspected of being mislabeled, and learning the remaining areas based on the original real data.

Images