Weakly supervised crowd counting method based on cross window self-attention mechanism

Abstract

The invention discloses a weakly supervised crowd counting method based on a cross-type window self-attention mechanism. The method comprises the following steps: 1) performing feature extraction on a target picture by adopting the cross-type window self-attention mechanism; (2) a CSWinCrowd structure based on global average pooling and regression Token is introduced; the method comprises the steps of (1) designing a CSWinwd-Token scheme based on a learnable position coding module, (2) designing a CSWinCrowd-Token scheme based on a learnable position coding module, and (4) providing a crowd counting model based on CSWinTransform for crowd counting of a target picture in combination with the steps (S1) to (S3). According to the method, the dependency relationship between different positions in a sequence can be well captured, and the problem that the counting effect is poor when the crowd density change of a current Transform model is large is solved.

Description

Technical Field

[0001] The present invention relates to the fields of signal processing for multimedia applications and occluded pedestrian perception technology, and in particular to a weakly supervised crowd counting method based on a cross-window self-attention mechanism. Background Art

[0002] Traditional crowd counting methods mainly predict the flow of crowds by observing surveillance videos with human eyes. However, in large-scale events, emergencies or disasters, it is difficult to judge the number and flow information of overly dense crowds by human eyes alone. Intelligent crowd counting methods can provide a large amount of data and statistical information, providing objective and quantitative basis for decision makers. Therefore, intelligent crowd counting methods are needed to help evaluate the evacuation of personnel, grasp the gathering areas and density of personnel, so as to better manage events and conduct emergency responses. Traditional dense crowd technology is mainly based on pedestrian detection and number regression. When the crowd density is large enough, the accuracy of traditional dense crowd counting methods cannot be guaranteed. Based on this, experts and scholars at home and abroad have proposed a crowd counting method based on density map estimation. Compared with previous methods, this method has greatly improved counting performance when dealing with complex scenes such as dense crowds, overlaps and occlusions. However, it also has some problems, such as accurate segmentation of dense crowd areas and the influence of noise. Therefore, in practical applications, it is usually necessary to combine other technologies and methods, such as deep learning and target tracking, to improve the performance of crowd counting models. Summary of the invention

[0003] The purpose of this invention is to solve the problem that the Transformer based on the attention mechanism architecture cannot effectively capture the spatial relationship and structural information between different positions in the crowd when the crowd density changes greatly, resulting in poor performance in real scenes with large changes in crowd density. A weakly supervised crowd counting method based on a cross-window self-attention mechanism is proposed.

[0004] To achieve the above object, the present invention adopts the following technical solution:

[0005] A weakly supervised crowd counting method based on a cross-window self-attention mechanism includes the following steps:

[0006] Step S1, using a cross-window self-attention mechanism to extract features from the target image;

[0007] Step S2: Introduce the CSWinCrowd structure based on global average pooling and regression Token;

[0008] Step S3, designing a CSWinCrowd-Token scheme based on a learnable position encoding module;

[0009] Step S4: Combining steps S1-S3, a crowd counting model based on CSWinTransformer is proposed for crowd counting of target images.

[0010] Specifically, the cross-window self-attention mechanism is used to extract features from the target image in step S1, including the following contents:

[0011] Input an image of size H×W×3, use overlapping convolutional tag embedding method, use 7×7 convolutional layer with stride 4, generate block markers, each of which has a dimension of C; in order to generate a hierarchical representation, the entire network consists of four stages, each of which consists of N consecutive CSWin networks. A convolution operation with a step size of 2 and a convolution kernel size of 3×3 is used between two adjacent stages to reduce the number of markers and double the channel dimension. Therefore, the feature map of the i-th stage has (H / 2 i+1 )×(W / 2 i+1 ) marks;

[0012] By dividing the input features into parallel stripes of equal width in the horizontal and vertical directions, a cross-shaped window is formed. For the self-attention of horizontal stripes, the input feature F is evenly divided into non-overlapping horizontal stripes [F1,...,F M ], the width of each stripe is λ, and each stripe contains λ×W tokens; assuming that the dimensions of the query vector, key, and value of the k-th head are all d, the output of the horizontal stripe self-attention of the k-th head is expressed as:

[0013] F=[F1,F2,...,F M ] (1)

[0014]

[0015] In the above formula, M=H / λ;i=1,…,M; Represents the query vector, key and value projection matrices of the kth head respectively; d k =C / K;

[0016] According to the formula of horizontal stripe self-attention, the output of its k-th head is expressed as V-Attn k (F'), the output of the self-attention of the vertical stripes is derived as:

[0017] F'=[F1',F2',...F' M ] (4)

[0018]

[0019] Assuming that the crowd image has no directional bias, the K heads are divided into two parallel groups, where K∈2N, N>0; horizontal stripe self-attention is performed on the first group of heads, and vertical stripe self-attention is performed on the second group of heads. The outputs of the two parallel groups are spliced ​​together, expressed as:

[0020] CSwin-Attn(F)=Concat(h1,h2,...,h k )W o (7)

[0021] h k =H-Attn k (F)(k=1,...,K / 2) (8)

[0022]

[0023] In the above formula, W o ∈R C×C is a commonly used projection matrix; the result of self-attention is projected to the target output dimension to obtain the attention feature map;

[0024] Multiple attention calculations are performed on the input sequence, and each head learns different attention weights. Finally, the outputs of each head are concatenated in a linear transformation layer to form the final multi-head attention feature map for crowd counting.

[0025] Specifically, the CSWinCrowd structure described in step S2 embeds the regression token into the input sequence and uses the global average pooling GAP to reduce the sequence length; for data sets with stable crowd density, CSWinCrowd-Token has better counting performance, and for data sets with large changes in crowd density, CSWinCrowd-GAP has better counting performance.

[0026] Specifically, the CSWinCrowd-Token scheme described in step S3 enables the model to more effectively capture the positional relationship of elements in the sequence by adding the LePE position encoding mechanism to the CSWinCrowd model. The specific contents are as follows:

[0027] Let a=(a1,...,a n ) represents the input sequence of length n, and b=(b1,...,b n ) represents the self-attention calculation output of the same length, where a i , b i ∈R c , then the self-attention calculation formula is expressed as:

[0028]

[0029] In the above formula, q i , i , η i is the query vector, key and value obtained by linearly transforming the input a, and dim is the feature dimension;

[0030] By introducing learnable element bias, the local enhanced position encoding module can be optimized according to the local characteristics of the data, expressed as:

[0031]

[0032] In the above formula, represents the kth element of vector b;

[0033] The CSWin Transformer block consisting of a cross-window self-attention mechanism feature extraction module and a local enhancement-based position encoding module is expressed as:

[0034] T l =CSwin-Attn(LN(T l-1 ))+T l-1 (12)

[0035] T l =MLP(LN(T l ))+T l (13)

[0036] In the above formula, T l Represents the output of the lth Transformer block or the output of the convolutional layer of the previous layer at each stage;

[0037] In order to further improve the counting performance of CSWinCrowd-Token, a learnable parameter λ is introduced in the LePE position encoding mechanism, which is expressed as:

[0038]

[0039] By introducing a learnable parameter λ, when the crowd density changes, the learnable parameter will adaptively adjust its size, thereby reducing the counting error of the CSWinCrowd-Token model.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. The method of the present invention can better capture the dependency between different positions in the sequence, and solves the problem that the current Transformer model has poor counting effect when the crowd density changes greatly.

[0042] 2. Aiming at the situations of uniform crowd density and large changes in crowd density, the method of the present invention proposes a CSWinCrowd structure based on global average pooling and regression Token; the CSWinCrowd Token structure uses additional learnable tokens to represent counts. This structure enables self-attention to propagate information between patch tokens and regression tokens, so that the regression token contains the overall crowd information, which can effectively improve the counting effect; the CSWinCrowd-GAP structure performs a GAP operation on all items in the Transformer encoder output sequence to obtain pooled visual markers, generate a pattern that can more accurately judge the semantics of the crowd, and can also greatly improve the crowd counting performance.

[0043] 3. To address the problem that CSWinCrowd-Token counting effect is weak when the crowd density changes greatly, the method of the present invention proposes a CSWinCrowd-Token scheme based on a learnable position encoding module; in the learnable position encoding module, the position encoding is added to the self-attention operation as a parallel module to focus on and utilize local features and context related to the crowd, helping the CSWinCrowd network to count better. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a crowd counting model based on CSWin Transformer in the present invention;

[0045] Figure 2 Schematic diagram of a cross-window self-attention mechanism in an embodiment of the present invention;

[0046] Figure 3 It is a processing flow chart of a feature extraction module based on a cross-window self-attention mechanism in an embodiment of the present invention;

[0047] Figure 4 Schematic diagram of two regression head input methods in an embodiment of the present invention;

[0048] Figure 5 Schematic diagram of a learnable position encoding module in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0050] Example

[0051] A weakly supervised crowd counting method based on a cross-window self-attention mechanism includes the following steps:

[0052] Step S1, using a cross-window self-attention mechanism to extract features from the target image;

[0053] Step S2: Introduce the CSWinCrowd structure based on global average pooling and regression Token;

[0054] Step S3, designing a CSWinCrowd-Token scheme based on a learnable position encoding module;

[0055] Step S4: Combined with steps S1-S3, a crowd counting model based on CSWinTransformer is proposed. Figure 1 As shown, crowd counting is used for target images.

[0056] Specifically, the cross-window self-attention mechanism is used to extract features from the target image in step S1, including the following contents:

[0057] like Figure 2 As shown in Figure 1, we input an image of size H×W×3, use the overlapping convolutional tag embedding method, and use a 7×7 convolutional layer with a step size of 4 to generate block markers, each of which has a dimension of C; in order to generate a hierarchical representation, the entire network consists of four stages, each of which consists of N consecutive CSWin networks. A convolution operation with a step size of 2 and a convolution kernel size of 3×3 is used between two adjacent stages to reduce the number of markers and double the channel dimension. Therefore, the feature map of the i-th stage has (H / 2 i+1 )×(W / 2 i+1 ) marks;

[0058] By dividing the input features into parallel stripes of equal width in the horizontal and vertical directions, a cross-shaped window is formed. For the self-attention of horizontal stripes, the input feature F is evenly divided into non-overlapping horizontal stripes [F1,...,F M ], the width of each stripe is λ, and each stripe contains λ×W tokens; assuming that the dimensions of the query vector, key, and value of the k-th head are all d, the output of the horizontal stripe self-attention of the k-th head is expressed as:

[0059] F=[F1,F2,...,F M ] (1)

[0060]

[0061] In the above formula, M=H / λ;i=1,…,M; Represents the query vector, key and value projection matrices of the kth head respectively; d k =C / K;

[0062] According to the formula of horizontal stripe self-attention, the output of its k-th head is expressed as V-Attn k (F'), the output of the self-attention of the vertical stripes is derived as:

[0063] F'=[F1',F2',...F' M ] (4)

[0064]

[0065] Assuming that the crowd image has no directional bias, the K heads are divided into two parallel groups, where K∈2N, N>0; horizontal stripe self-attention is performed on the first group of heads, and vertical stripe self-attention is performed on the second group of heads. The outputs of the two parallel groups are spliced ​​together, expressed as:

[0066] CSwin-Attn(F)=Concat(h1,h2,...,h k )W o (7)

[0067] h k =H-Attn k (F)(k=1,...,K / 2) (8)

[0068]

[0069] In the above formula, W o ∈R C×C is a commonly used projection matrix; the result of self-attention is projected to the target output dimension to obtain the attention feature map;

[0070] like Figure 3 As shown in the figure, the key to the design of the cross-window self-attention mechanism is to divide the multi-head attention into different groups and apply vertical and horizontal self-attention operations accordingly; through multi-head grouping, the attention area of ​​each tag in the Transformer block is expanded. By expanding the attention area of ​​each tag, the model can better capture the details and correlation relationships in crowded areas. In addition, it can also improve the processing ability of crowds of different scales and capture the correlation relationships in the crowd, thereby improving the accuracy and robustness of counting.

[0071] Multiple attention calculations are performed on the input sequence, and each head learns different attention weights to obtain multiple different representation subspaces. Each head can focus on different parts of the input sequence to capture different relationships and feature representations. The output of each head is obtained by linearly transforming the input sequence with the weight matrix corresponding to the head, calculating the attention score, and then weighted summing the score with the value of the input sequence. By introducing multiple attention heads, the CSWinCrowd network can learn multiple different relationships and feature representations in parallel, thereby enriching the expressive power of the model. Each head can focus on different positions and different semantics in the image, thereby capturing richer crowd information. Finally, the outputs of each head are spliced ​​in the last linear transformation layer to form the final multi-head attention representation. The introduction of the multi-head attention mechanism enables the CSWinCrowd network to better handle long-distance dependencies and global contextual information, reduce the impact of changes in crowd density on the model counting performance, and help crowd counting.

[0072] Specifically, the CSWinCrowd structure based on global average pooling and regression token is introduced in step S2; the goal of the regression head is to generate the predicted number of people. Similar to Bert and ViT, the method of the present invention embeds the regression token into the input sequence, such as Figure 4 As shown in (a), the architecture enables self-attention to propagate information between the patch token and the regression token, so that the regression token contains the overall crowd information. The regression head is mainly a multi-layer perceptron layer containing two linear layers. The CSWinCrowd with additional tokens is called CSWinCrowd-Token. The method of the present invention uses global average pooling (GAP) to reduce the sequence length, as shown in Figure 4 As shown in (b), similar to CSWinCrowd-Token, the regression head includes a multi-layer perceptron layer; the method of the present invention refers to CSWinCrowd with GAP as CSWinCrowd-GAP.

[0073] Among them, CSWinCrowd-Token uses an additional learnable token to represent the count, and the CSWinCrowd-GAP structure uses global average pooling (GAP) for all items in the Transformer encoder output sequence to obtain pooled visual tags; the counting performance of CSWinCrowd-GAP and CSWinCrowd-Token is verified on public datasets. Experimental data show that on the ShanghaiTech_Part A dataset with relatively stable crowd density, CSWinCrowd-Token has better counting performance, but on the ShanghaiTech_Part B and UCF-QNRF datasets with large changes in crowd density, CSWinCrowd-GAP uses GAP to generate a more accurate pattern for judging crowd semantics, which is better than using token counting performance; experimental results show that when the crowd density changes greatly, CSWinCrowd-GAP can produce more reasonable attention weights, and compared with CSWinCrowd-Token, it converges faster and has stronger counting performance.

[0074] Specifically, in the CSWinCrowd-Token scheme described in step S3, when the crowd density varies greatly, the order and position of the crowd have an important impact on the counting result. Since the self-attention operation cannot distinguish elements at different positions, it cannot effectively capture the time dependency or sequential pattern in the sequence, thereby ignoring the important position information in the crowd image. In order to add such information to improve the counting performance of CSWinCrowd-Token, Figure 5 As shown in Figure 2, the LePE position encoding mechanism is added to the CSWinCrowd model, which enables the model to more effectively capture the position relationship of elements in the sequence. The specific contents are as follows:

[0075] Let a=(a1,...,a n ) represents the input sequence of length n, and b=(b1,...,b n ) represents the self-attention calculation output of the same length, where a i 、b i ∈R c , then the self-attention calculation formula is expressed as:

[0076]

[0077] In the above formula, q i , i , η i is the query vector, key and value obtained by linearly transforming the input a, and dim is the feature dimension;

[0078] By introducing learnable element bias, the local enhanced position encoding module can be optimized according to the local characteristics of the data, expressed as:

[0079]

[0080] In the above formula, represents the kth element of vector b;

[0081] The CSWin Transformer block consisting of a cross-window self-attention mechanism feature extraction module and a local enhancement-based position encoding module is expressed as:

[0082] T l =CSwin-Attn(LN(T l-1 ))+T l-1 (12)

[0083] T l =MLP(LN(Tl))+T l (13)

[0084] In the above formula, T l Represents the output of the lth Transformer block or the output of the convolutional layer of the previous layer at each stage;

[0085] In order to further improve the counting performance of CSWinCrowd-Token, a learnable parameter λ is introduced in the LePE position encoding mechanism, which is expressed as:

[0086]

[0087] By introducing a learnable parameter λ, when the crowd density changes, the learnable parameter will adaptively adjust its size, thereby reducing the counting error of the CSWinCrowd-Token model.

[0088] The method of the present invention is further illustrated below by a specific example.

[0089] In this example, the crowd counting model (CSWinCrowd) based on the cross-window self-attention mechanism is experimented based on the Ubuntu 18.04.5 operating system and Pytorch1.7.1. The processor used is the Intel Xeon Gold 6240R with a main frequency of 2.4GHz, and the graphics card uses NVIDIA's GeForce RTX

[0090] 3090, with 24GB video memory.

[0091] As shown in Table 1 below, the method CSwinCrowd proposed in the present invention achieves better counting performance than existing methods on the public crowd datasets: ShanghaiTechPartA and B.

[0092] Table 1. Performance comparison of CSwinCrowd and other weakly supervised counting methods on public datasets

[0093]

[0094] From the data in the table above, we can see that in the ShanghaiTech A part of the data set, compared with the MATT method, the MAE of CSwinCrowd-Token is reduced by 16.9, and the MSE is reduced by 25.8. Compared with the Method in method, the MAE is reduced by 41.4, and the MSE is reduced by 41.6. Compared with the Transcrowd-Token method, the MAE of CSwinCrowd-Token method is reduced by 5.8, and the MSE is reduced by 12.9. In addition, compared with the MATT, Method in method and Transcrowd-GAP method, the MAE and MSE values ​​of the CSwinCrowd-GAP proposed in the present invention are also greatly reduced.

[0095] In the ShanghaiTech dataset, part B, CSwinCrowd-GAP reduces MAE by 3 and MSE by 1.6 compared with MATT; MAE is reduced by 3.6 and MSE is reduced by 5.3 compared with Method in; MAE is reduced by 0.6 and MSE is reduced by 0.2 compared with Transcrowd-GAP. In addition, compared with MATT, Method in and Transcrowd-Token, the CSwinCrowd-Token proposed in this invention also achieves significant reduction in MAE and MSE.

[0096] In summary, compared with the existing crowd counting methods and models based on weak supervision, whether CSWinCrowd uses the Token or GAP method, its MAE value and MSE value are smaller, indicating that the average error between the prediction results of the CSWinCrowd model proposed in the present invention and the actual crowd count is smaller. This shows that compared with the previous model, the model proposed in the present invention predicts the number of people more accurately. At the same time, the decrease in the MAE value also shows that the generalization ability of the CSWinCrowd model on new data is stronger than that of the existing crowd counting methods based on weak supervision.

[0097] Table 2. Test results of different models based on weak supervision on the UCF-QNRF dataset

[0098]

[0099] Table 2 above lists the test results of four models based on weak supervision: CCTrans, Crowdformer, TransCrowd and CSwinCrowd proposed in the present invention on the UCF-QNRF dataset.

[0100] As can be seen from the above table, the CSwinCrowd network model proposed in the present invention has greatly improved the crowd counting effect. Among the four models in the table, the MAE and MSE values ​​of the CSwinCrowd network model are the lowest, indicating that compared with the three existing crowd counting methods based on weak supervision, the CSwinCrowd network model has a better counting effect for changes in crowd density. Among them: the MAE value of CSwinCrowd-GAP is reduced by 1.7 compared with CCTrans, and the MSE value is reduced by 8; the MAE value of CSwinCrowd-GAP is reduced by 2.9 compared with Crowdformer, and the MSE value is reduced by 10; the MAE value of CSwinCrowd-GAP is reduced by 6.8 compared with TransCrowd-GAP, and the MSE value is reduced by 17.6; it shows that compared with the TransCrowd model based on VIT, the CSwinCrowd based on CSwin-transformer proposed in the present invention has better effect in crowd counting; in addition, compared with CCTrans, Crowdformer, and Transcrowd-Token, the MAE value of CSwinCrowd-Token proposed in the present invention has been greatly reduced.

[0101] Ablation experiment analysis

[0102] Ablation experiment of Lepe module

[0103] In order to verify the role of the Lepe module in the CSWinCrowd model in processing crowd counting tasks, this example conducts ablation experiments on three public datasets. The experimental parameters are set as follows:

[0104] baseline1:

[0105] Under the Gap structure, baseline1 represents the CSWinCrowd-GAP model without the Lepe module; baseline1+Lepe represents the original CSWinCrowd-GAP model; baseline1+learnable Lepe represents the CSWinCrowd-GAP model with a learnable position encoding module.

[0106] The ablation results are shown in Tables 3 and 4 below.

[0107] Table 3. Ablation verification of Lepe module on ShanghaiTech_A under GAP structure

[0108]

[0109] As can be seen from Table 3 above, on the ShanghaiTech_A dataset, the MAE value of CSWinCrowd-GAP dropped by 6.5 compared to the model without the LePe module. These data show that when the CSWinCrowd-GAP model processes crowd counting tasks, the LePe module can help enhance the counting performance of the visual Transformer. As can be seen from Table 3, the MAE values ​​of CSWinCrowd-GAP with the LePe module and the learnable position encoding module are very close, indicating that for the CSWinCrowd-GAP structure, on the ShanghaiTech_A dataset where the crowd density does not change much, the LePe module and the learnable position encoding module have similar auxiliary effects on the CSWinCrowd-GAP structure.

[0110] As can be seen from Table 4 below, on the ShanghaiTech_B dataset, the MAE value of CSWinCrowd-GAP only dropped by 0.5 compared to the model without the LePe module. This is because although the density of the ShanghaiTech_B dataset varies greatly, the crowd density is small, so the performance of the model is not much different. On the UCF-QNRF dataset, where the crowd density varies greatly, the MAE value of CSWinCrowd-GAP dropped by 14.6 compared to the model without the LePe module. These data show that when the CSWinCrowd-GAP model processes crowd counting tasks, the LePe module can help enhance the counting performance of the visual Transformer.

[0111] Table 4. Ablation verification of Lepe module on ShanghaiTech_B and UCF-QNRF under Gap structure

[0112]

[0113] baseline2:

[0114] Under the Token structure, baseline2 represents the CSWinCrowd-Token model without the Lepe module, baseline2+Lepe represents the original CSWinCrowd-Token model, and baseline2+learnable Lepe represents the CSWinCrowd-Token model with learnable parameters Lepe.

[0115] The ablation results are shown in Tables 5 and 6 below.

[0116] Table 5. Ablation verification of Lepe module on ShanghaiTech_A dataset under Token structure

[0117]

[0118]

[0119] Table 6. Ablation verification of Lepe module on ShanghaiTech_B and UCF-QNRF under Token structure

[0120]

[0121] As can be seen from Tables 5 and 6, on the ShanghaiTech_A dataset, the MAE value of the CSWinCrowd-Token model decreased by 9.5 compared with the model without the LePe module; and the MAE value of the CSWinCrowd-GAP decreased by 6.5 compared with the model without the LePe module. This shows that the performance of the CSWinCrowd-Token model will be better when processing datasets such as ShanghaiTech_A, where the crowd density is dense but the crowd density does not change much. Whether it is CSWinCrowd-GAP or CSWinCrowd-Token, the MAE value of CSWinCrowd with the Lepe module is lower, indicating that adding the Lepe module to assist the CSWinCrowd model in processing crowd counting tasks can help enhance the counting performance of the visual Transformer.

[0122] Performance comparison of multi-head attention grouping methods

[0123] In the CSWinCrowd network, by default, K multi-head attentions are evenly divided into two groups (each group has K / 2 heads, K is an even number). In order to verify the effectiveness of the equal division, the K multi-head attentions are divided into 3 and 4 parts, respectively, denoted as experimental group A (k-3) and experimental group B (k-4), and then the two groups A and B are allocated respectively, divided into 1 / 3, 2 / 3 and 1 / 4, 3 / 4 allocation methods, respectively denoted as CSWinCrowd-GAP / Token-1 / 3, CSWinCrowd-GAP / Token-2 / 3 and CSWinCrowd-GAP / Token-1 / 4, CSWinCrowd-GAP / Token-3 / 4, and verified on the datasets ShanghaiTech Part_A and ShanghaiTech Part_B. The ablation verification results are shown in Tables 7 and 8 below.

[0124] Table 7. Ablation verification results of 1:2 split of multi-head attention

[0125]

[0126] Table 8. Ablation verification results of multi-head attention 1:3 split

[0127]

[0128]

[0129] As can be seen from Tables 7 and 8, CSWinCrowd-GAP and CSWinCrowd-Token, compared with dividing K multi-head attention into 1, 3 points or 1, 2 points, divide K multi-head attention equally into two groups. On the ShanghaiTech_Part A and ShanghaiTech_Part B datasets, their MAE and MSE values ​​are the smallest. The smaller the MAE value, the higher the crowd counting accuracy. This shows that dividing K multi-head attention equally into two groups has the best effect.

[0130] From the ablation experiment results in Tables 3, 4, 5, and 6, we can see that CSWinCrowd-Token has better counting performance on ShanghaiTech_Part A where the crowd density does not change much, but CSWinCrowd-GAP has better counting performance on datasets such as ShanghaiTech_PartB and UCF-QNRF where the crowd density changes greatly. The experimental results show that when the crowd density changes greatly, CSWinCrowd-GAP can generate more reasonable attention weights, and it converges faster and has stronger counting performance than CSWinCrowd-Token.

[0131] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A weakly supervised crowd counting method based on a cross-window self-attention mechanism, characterized in that: The following steps are involved: Step S1, using a cross-window self-attention mechanism to extract features from the target image; Step S2: Introduce the CSWinCrowd structure based on global average pooling and regression Token; Step S3, designing a CSWinCrowd-Token scheme based on a learnable position encoding module; Step S4: Combining steps S1-S3, a crowd counting model based on CSWinTransformer is proposed for crowd counting of target images.

2. According to claim 1, a weakly supervised crowd counting method based on a cross-window self-attention mechanism is characterized in that: In step S1, the cross-window self-attention mechanism is used to extract features from the target image. Includes the following: Input an image of size H×W×3, use overlapping convolutional tag embedding method, use 7×7 convolutional layer with stride 4, generate block markers, each of which has a dimension of C; in order to generate a hierarchical representation, the entire network consists of four stages, each of which consists of N consecutive CSWin networks. A convolution operation with a step size of 2 and a convolution kernel size of 3×3 is used between two adjacent stages to reduce the number of markers and double the channel dimension. Therefore, the feature map of the i-th stage has (H / 2 i+1 )×(W / 2 i+1 ) marks; By dividing the input features into parallel stripes of equal width in the horizontal and vertical directions, a cross-shaped window is formed. For the self-attention of horizontal stripes, the input feature F is evenly divided into non-overlapping horizontal stripes [F1,...,F M ], the width of each stripe is λ, and each stripe contains λ×W tokens; assuming that the dimensions of the query vector, key, and value of the k-th head are all d, the output of the horizontal stripe self-attention of the k-th head is expressed as: F=[F1,F2,...,F M ] (1) In the above formula, M=H / λ;i=1,…,M; Represents the query vector, key and value projection matrices of the kth head respectively; d k =C / K; According to the formula of horizontal stripe self-attention, the output of its k-th head is expressed as V-Attn k (F'), the output of the self-attention of the vertical stripes is derived as: F'=[F1',F2',...F' M ] (4) Assuming that the crowd image has no directional bias, the K heads are divided into two parallel groups, where K∈2N, N>0; horizontal stripe self-attention is performed on the first group of heads, and vertical stripe self-attention is performed on the second group of heads. The outputs of the two parallel groups are spliced ​​together, expressed as: CSwin-Attn(F)=Concat(h1,h2,...,h k )W o (7) h k =H-Attn k (F)(k=1,...,K / 2) (8) In the above formula, W o ∈R C×C is a commonly used projection matrix; the result of self-attention is projected to the target output dimension to obtain the attention feature map; Multiple attention calculations are performed on the input sequence, and each head learns different attention weights. Finally, the outputs of each head are concatenated in a linear transformation layer to form the final multi-head attention feature map for crowd counting.

3. According to claim 1, a weakly supervised crowd counting method based on a cross-window self-attention mechanism is characterized in that: The CSWinCrowd structure described in step S2 embeds the regression token into the input sequence and uses the global average pooling GAP to reduce the sequence length; for data sets with stable crowd density, CSWinCrowd-Token has better counting performance, and for data sets with large changes in crowd density, CSWinCrowd-GAP has better counting performance.

4. According to claim 1, a weakly supervised crowd counting method based on a cross-window self-attention mechanism is characterized in that: The CSWinCrowd-Token scheme described in step S3 enables the model to more effectively capture the positional relationship of elements in the sequence by adding the LePE position encoding mechanism to the CSWinCrowd model. The specific contents are as follows: Let a=(a1,...,a n ) represents the input sequence of length n, and b=(b1,...,b n ) represents the self-attention calculation output of the same length, where a i 、b i ∈R c , then the self-attention calculation formula is expressed as: In the above formula, q i , i , η i is the query vector, key and value obtained by linearly transforming the input a, and dim is the feature dimension; By introducing learnable element bias, the local enhanced position encoding module can be optimized according to the local characteristics of the data, expressed as: In the above formula, represents the kth element of vector b; The CSWin Transformer block consisting of a cross-window self-attention mechanism feature extraction module and a local enhancement-based position encoding module is expressed as: T l =CSwin-Attn(LN(T l-1 ))+T l-1 (12) T l =MLP(LN(T l ))+T l (13) In the above formula, T l Represents the output of the lth Transformer block or the output of the convolutional layer of the previous layer at each stage; In order to further improve the counting performance of CSWinCrowd-Token, a learnable parameter λ is introduced in the LePE position encoding mechanism, which is expressed as: By introducing a learnable parameter λ, when the crowd density changes, the learnable parameter will be adaptively adjusted to reduce the counting error of the CSWinCrowd-Token model.

Images