A transformer-based cross-modal crowd counting system and method

Abstract

The application discloses a cross-modal crowd counting system and method based on a Transformer, and steps are as follows: inputting an infrared thermal image and a visible light image to a first-order feature extraction module to obtain first-order features; obtaining first-order cross-modal features and attention weights through a first-order modal mixer; obtaining first-order enhanced features through element-by-element multiplication and addition operations; inputting the enhanced visible light and infrared thermal features into a second-order feature extraction module to obtain second-order features; inputting the second-order features into a second-order modal mixer to obtain second-order cross-modal features and attention weights; obtaining second-order enhanced features through element-by-element multiplication and addition operations; inputting the enhanced visible light and infrared thermal features into a third-order feature extraction module to obtain third-order features; inputting the enhanced visible light and infrared thermal features into a third-order modal mixer to obtain third-order cross-modal features; and inputting the cross-modal features of the three stages into a regression head to obtain an estimated density map.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a Transformer-based cross-modal crowd counting system and method. Background Technology

[0002] The accelerated pace of urbanization has led to a significant increase in the population of central cities. This massive population growth has not only brought opportunities for rapid urban development but also posed serious challenges to urban housing, transportation, and management. The increasing number of large-scale cultural and commercial events has resulted in frequent instances of dense crowds, placing immense pressure on urban traffic and public safety management. To address this pressure, various crowd counting methods have emerged.

[0003] The mainstream crowd counting method currently available is the density map regression method based on CNNs. To extract more features, CNN-based methods typically employ various multi-scale structures to expand the receptive field. However, due to the limited receptive field, the performance of the model is still limited, especially in dense regions. To overcome this problem, in addition to the widely used CNN-based model design, Transformer has become a popular approach for computer vision tasks. Transformer has a global receptive field, which can improve the feature extraction capability of CNNs. However, even when using Transformer as a feature extraction module, RGB image crowd counting methods still struggle to accurately extract features under adverse lighting conditions because the pixel values ​​are less distinguishable. In contrast, thermal images can highlight objects with different temperatures and exhibit good robustness under low light and adverse weather conditions. Therefore, thermal images can serve as an important supplementary information to RGB images in crowd counting, providing a representative supplementary data source for counting scenarios. This crowd counting method that combines visible light images and infrared thermal images is often called the RGB-T crowd counting method. Current Transformer-based RGB-T crowd counting methods typically utilize mutual attention to calculate cross-modal attention. However, mutual attention typically involves computing the attention of the current modality using a query from another modality, making it impossible to model the global context of each modality individually. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a cross-modal crowd counting method based on Transformer, in order to solve the negative impact of low lighting conditions, the limitations of traditional convolutional neural network methods due to the limited receptive field, and the fact that mutual attention uses the query of another modality to calculate the attention of the current modality, which leads to the inability to extract the global context of each modality, on crowd counting performance.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: comprising a first-order feature extraction module, a first-order cross-modal mixer, a second-order feature extraction module, a second-order cross-modal mixer, a third-order feature extraction module, a third-order cross-modal mixer, and a regression head connected in sequence.

[0006] A further improvement to the technical solution of this invention lies in the following: the specific structure of each module is as follows:

[0007] The first-order feature extraction module consists of an overlapping patch embedding module, an average pooling module, and a feedforward module. The overlapping patch embedding module includes a convolutional layer with a kernel size of 7×7, 128 output channels, and a stride of 4, and a group normalization regularization operation. The average pooling module includes an average pooling layer with a pooling window size of 3×3 and a stride of 1, a subtraction operation, and an addition operation. The feedforward module includes a convolutional layer with a kernel size of 1×1, 512 output channels, and a stride of 1, a GELU activation function, and a convolutional layer with a kernel size of 1×1, 128 output channels, and a stride of 1.

[0008] The second-order feature extraction module consists of an overlapping patch embedding module, an average pooling module, and a feedforward module. The overlapping patch embedding module includes a convolutional layer with a kernel size of 3×3, 320 output channels, and a stride of 2, and a group normalization regularization operation. The average pooling module includes an average pooling layer with a pooling window size of 3×3 and a stride of 1, a subtraction operation, and an addition operation. The feedforward module includes a convolutional layer with a kernel size of 1×1, 1280 output channels, and a stride of 1, a GELU activation function, and a convolutional layer with a kernel size of 1×1, 320 output channels, and a stride of 1.

[0009] The third-order feature extraction module consists of an overlapping patch embedding module, an average pooling module, and a feedforward module. The overlapping patch embedding module includes, in sequence, a convolutional layer with a kernel size of 3×3, 512 output channels, and a stride of 2, and a group normalization regularization operation. The average pooling module includes, in sequence, an average pooling layer with a pooling window size of 3×3 and a stride of 1, a subtraction operation, and an addition operation. The feedforward module includes, in sequence, a convolutional layer with a kernel size of 1×1, 2048 output channels, and a stride of 1, a GELU activation function, and a convolutional layer with a kernel size of 1×1, 512 output channels, and a stride of 1.

[0010] First-order cross-modal mixer: block embedding layer, four identical Transformer encoder layers and reshaping module, wherein the block embedding layer includes an addition operation, a convolutional layer with a kernel size of 3×3, 128 output channels and stride of 1, a flattening operation and a transpose operation.

[0011] The Transformer encoder layer consists of a multi-head self-attention layer with eight heads, a feedforward layer, and a normalization operation.

[0012] The multi-head self-attention layer includes a query vector calculation module, a key vector and value vector calculation module, and a multi-head self-attention calculation module. The query vector calculation module includes a fully connected layer and a reshaping operation. The key vector and value vector calculation module includes a fully connected layer and a reshaping operation.

[0013] The feedforward layer consists of a fully connected layer with 512 neurons, a GELU activation operation, and a fully connected layer with 128 neurons.

[0014] The reshaping module includes a reshaping operation, a multiplication operation, and an addition operation.

[0015] The second-order cross-modal mixer consists of a block embedding layer, four identical Transformer encoder layers, and a reshaping module. The block embedding layer includes, in sequence, an addition operation, a convolutional layer with a kernel size of 3×3, 320 output channels, and a stride of 1, a flattening operation, and a transpose operation.

[0016] The Transformer encoder layer consists of a multi-head self-attention layer with eight heads, a feedforward layer, and a normalization operation.

[0017] The multi-head self-attention layer includes a query vector calculation module, a key vector and value vector calculation module, and a multi-head self-attention calculation module. The query vector calculation module includes a fully connected layer and a reshaping operation. The key vector and value vector calculation module includes a fully connected layer and a reshaping operation.

[0018] The feedforward layer consists of a fully connected layer with 1280 neurons, a GELU activation function, and a fully connected layer with 320 neurons.

[0019] The reshaping module includes a reshaping operation, a multiplication operation, and an addition operation.

[0020] The third-order cross-modal mixer consists of a block embedding layer, four identical Transformer encoder layers, and a reshaping module. The block embedding layer includes, in sequence, an addition operation, a convolutional layer with a kernel size of 3×3, 512 output channels, and a stride of 1, a flattening operation, and a transpose operation.

[0021] The Transformer encoder layer consists of a multi-head self-attention layer with eight heads, a feedforward layer, and a normalization operation.

[0022] The multi-head self-attention layer includes a query vector calculation module, a key vector and value vector calculation module, and a multi-head self-attention calculation module. The query vector calculation module includes a fully connected layer and a reshaping operation. The key vector and value vector calculation module includes a fully connected layer and a reshaping operation.

[0023] The feedforward layer consists of a fully connected layer with 2048 neurons, a GELU activation function, and a fully connected layer with 512 neurons.

[0024] The reshaping module includes a reshaping operation, a multiplication operation, and an addition operation.

[0025] Regression head: upsampling operation, summation operation, convolutional layer with kernel size of 3×3 and stride of 1 and convolutional layer with kernel size of 1×1 and stride of 1.

[0026] A cross-modal crowd counting method based on Transformer, the steps of which are as follows:

[0027] Step 1: Input the infrared thermal image and the visible light image into the first-order feature extraction module to obtain the first-order visible light features and the first-order infrared thermal features, respectively;

[0028] Step 2: Input the first-order visible light feature and the first-order infrared thermal feature into the first-order modal mixer to obtain the first-order cross-modal feature and the first-order attention weight map;

[0029] Step 3: Multiply the first-order attention weight map with the first-order visible light feature and the first-order infrared thermal feature by pixel position and add them by pixel position respectively to obtain the first-order enhanced visible light feature and the first-order enhanced infrared thermal feature.

[0030] Step 4: Input the first-order enhanced visible light feature and the first-order enhanced infrared thermal feature into the second-order feature extraction module to obtain the second-order visible light feature and the second-order infrared thermal feature;

[0031] Step 5: Input the second-order visible light features and second-order infrared thermal features into the second-order modal mixer to obtain the second-order cross-modal features and the second-order attention weight map;

[0032] Step 6: Multiply the second-order attention weight map with the second-order visible light feature and the infrared thermal feature by pixel position, and then add them by pixel position to obtain the second-order enhanced visible light feature and the second-order enhanced infrared thermal feature.

[0033] Step 7: Input the second-order enhanced visible light feature and the second-order enhanced infrared thermal feature into the third-order feature extraction module to obtain the third-order visible light feature and the third-order infrared thermal feature;

[0034] Step 8: Input the third-order visible light feature and the third-order infrared thermal feature into the third-order mode mixer to obtain the third-order cross-modal feature;

[0035] Step 9: Input the first-order cross-modal features, second-order cross-modal features, and third-order cross-modal features into the regression head to obtain the estimated density map, and sum the pixel values ​​of the estimated density map to obtain the population size.

[0036] A further improvement to the technical solution of the present invention is as follows: Step 1 is specifically performed as follows:

[0037] Step 11: Input the visible light image and the infrared thermal image into the overlap patch embedding module to obtain the first-order regularized block visible light feature map and the first-order regularized block infrared thermal feature map;

[0038] Step 111: Input the visible light image and the infrared thermal image into a convolutional layer with a kernel size of 7×7, an output channel number of 128, and a stride of 4 to obtain block-based first-order visible light feature maps and block-based first-order infrared thermal feature maps.

[0039] Step 112: Perform group normalization regularization on the block-based first-order visible light features and block-based first-order infrared thermal features to obtain first-order regularized block-based visible light feature map and first-order regularized block-based infrared thermal feature map.

[0040] Step 12: Input the first-order regularized block visible light feature map and the first-order regularized block infrared thermal feature map into the average pooling module to obtain the first-order pooling block visible light feature map and the first-order pooling block infrared thermal feature map.

[0041] Step 121: Input the first-order regularized block visible light feature map and the first-order regularized block infrared thermal feature map into the average pooling layer with a pooling window size of 3×3 and a step number of 1 to obtain the first-order regularized pooling block visible light feature map and the first-order regularized pooling block infrared thermal feature map.

[0042] Step 122: Subtract the corresponding pixels of the first-order regularized pooling block visible light feature map and the first-order regularized pooling block infrared thermal feature map from the corresponding pixels of the first-order regularized block visible light feature map and the first-order regularized block infrared thermal feature map, respectively, to obtain the first-order residual pooling visible light block feature map and the first-order residual pooling infrared thermal block feature map.

[0043] Step 123: Add the first-order residual pooling visible light block feature map and the first-order residual pooling infrared thermal block feature map to the corresponding pixels of the block first-order visible light feature map and the block first-order infrared thermal feature map, respectively, to obtain the first-order pooling block visible light feature map and the first-order pooling block infrared thermal feature map.

[0044] Step 13: Input the first-order pooling block visible light feature map and the first-order pooling block infrared thermal feature map into the feedforward layer to obtain the first-order visible light feature and the first-order infrared thermal feature.

[0045] Step 131: Input the first-order pooling block visible light feature map and the first-order pooling block infrared thermal feature map into a convolutional layer with a kernel size of 1×1, an output channel number of 512, and a stride of 1 to obtain the connected first-order pooling block visible light feature map and the connected first-order pooling block infrared thermal feature map.

[0046] Step 132: Input the concatenated first-order pooling block visible light feature map and the concatenated first-order pooling block infrared thermal feature map into the GELU activation function to obtain the concatenated activated first-order pooling block visible light feature map and the concatenated activated first-order pooling block infrared thermal feature map.

[0047] Step 133: Input the concatenated activated first-order pooling block visible light feature map and the concatenated activated first-order pooling block infrared thermal feature map into a convolutional layer with a kernel size of 1×1, an output channel number of 128, and a stride of 1 to obtain the first-order visible light feature and the first-order infrared thermal feature.

[0048] A further improvement to the technical solution of the present invention is as follows: Step 2 is specifically as follows:

[0049] Step 21: Input the first-order visible light feature and the first-order infrared thermal feature into the block embedding layer to obtain the flattened first-order cross-modal feature;

[0050] Step 211: Add the first-order visible light feature and the first-order infrared thermal feature at corresponding pixel positions to obtain the first-order fused feature;

[0051] Step 212: Input the first-order fusion feature into a convolutional layer with a kernel size of 3×3, an output channel number of 128, and a stride of 1 to obtain the first-order block fusion feature;

[0052] Step 213: One-dimensionalize and transpose the first-order block fusion features to obtain flattened first-order cross-modal features;

[0053] Step 22: Input the flattened first-order cross-modal features into four Transformer encoder layers consecutively to obtain a first-order attention weight map;

[0054] Step 221: Input the flattened first-order cross-modal features into the query vector module of the multi-head self-attention layer to obtain the query vector;

[0055] Step 222: Input the flattened first-order cross-modal features into the key and value vector calculation module of the multi-head self-attention layer to obtain the key and value vectors.

[0056] The formulas for calculating the query vector, key vector, and value vector are as follows:

[0057]

[0058] Where Q, K, and V are the output vectors representing the first-order query vector, the first-order key vector, and the first-order value vector, respectively. It is the corresponding projection matrix, Z i It is the input first-order, one-dimensional vector;

[0059] Step 223: Input the query vector, key vector, and value vector into the multi-head self-attention calculation module to obtain the self-attention result; the formula for calculating the self-attention result is:

[0060]

[0061] MHSA = Concat(head1,…,head) h W o

[0062] head i L is the attention value of the i-th head. i W is the length of the input one-dimensional vector. o It is the projection matrix, Concat(·) represents the concatenation operation, and MHSA is the result of self-attention;

[0063] Step 224: Input the self-attention result into a fully connected layer with 512 neurons in the feedforward layer to obtain the connected self-attention result; then input it into the GELU activation function in the feedforward layer, and input the result into a fully connected layer with 128 neurons in the feedforward layer, and normalize it to obtain the first-order attention weight map calculated once.

[0064] Step 225: Input the first-order attention weight map calculated once into the Transformer encoder layer to obtain the first-order attention weight map calculated twice; input the first-order attention weight map calculated twice into the Transformer encoder layer to obtain the first-order attention weight map calculated three times; input the first-order attention weight map calculated three times into the Transformer encoder layer to obtain the first-order attention weight map.

[0065] Step 23: Input the first-order attention weight map and the first-order fusion feature into the reshaping module to obtain the first-order attention weight map and the first-order cross-modal feature map;

[0066] Step 231: Reshape the first-order attention weight map to obtain a reshaped first-order attention weight map;

[0067] Step 232: Multiply the reshaped first-order attention weight map with the first-order cross-modal fusion feature by pixel position to obtain the weighted first-order cross-modal fusion feature;

[0068] Step 233: Add the weighted first-order cross-modal fusion features and the first-order cross-modal fusion features according to pixel position to obtain the first-order cross-modal features.

[0069] A further improvement to the technical solution of the present invention is as follows: Step 3 is specifically as follows:

[0070] Step 31: Multiply the first-order attention weight map output in Step 2 element by element on the corresponding channel with the first-order visible light feature and the first-order infrared thermal feature output in Step 1 to obtain the multiplied first-order visible light feature map and the multiplied first-order infrared thermal feature map.

[0071] Step 32: Add the multiplied first-order visible light feature map and the multiplied first-order infrared thermal feature map to the first-order visible light feature map and the first-order infrared thermal feature map respectively on the corresponding channels to obtain the first-order enhanced visible light feature map and the first-order enhanced infrared thermal feature map.

[0072] A further improvement to the technical solution of this invention is as follows: Step 4 involves inputting the first-order enhanced visible light feature and the first-order enhanced infrared thermal feature into the second-order feature extraction module to obtain the second-order visible light feature and the second-order infrared thermal feature. The calculation process in the second-order feature extraction module is the same as the calculation process in the first-order feature extraction module in Step 1. Step 7 involves inputting the second-order enhanced visible light feature and the second-order enhanced infrared thermal feature into the third-order feature extraction module to obtain the third-order visible light feature and the third-order infrared thermal feature. The calculation process in the third-order feature extraction module is the same as the calculation process in the first-order feature extraction module in Step 1. The calculation process in step 1 is the same as that in step 2; step 5 involves inputting the second-order visible light features and second-order infrared thermal features into the second-order modal mixer to obtain the second-order cross-modal features and the second-order attention weight map; the calculation process in the second-order modal mixer is the same as that in the first-order modal mixer in step 2; step 8 involves inputting the third-order visible light features and third-order infrared thermal features into the third-order modal mixer to obtain the third-order cross-modal features; the calculation process in the third-order modal mixer is the same as that in the first-order modal mixer in step 2; the calculation process of multiplication and addition in step 6 is the same as that in step 3.

[0073] A further improvement to the technical solution of the present invention is as follows: Step 9 is specifically as follows:

[0074] Step 91: Upsample the third-order transmodal features obtained in Step 8 by a factor of 2 using bilinear interpolation, and then connect them with the second-order transmodal features obtained in Step 5 to obtain the feature map after the first connection.

[0075] Step 92: Smooth the feature map after the first connection, and then reduce the number of channels by using a convolutional layer with a kernel of 3×3 and a stride of 1 to obtain the convolutional feature map.

[0076] Step 93: Upsample the convolutional feature map by a factor of 2 using bilinear interpolation, and then concatenate it with the first-order cross-modal feature obtained in Step 2 to obtain the second concatenated feature map.

[0077] Step 94: Dimensionality reduction is performed on the feature map after the second connection using a convolutional layer with a kernel of 3×3 and a stride of 1 and a convolutional layer with a kernel of 1×1 and a stride of 1, finally obtaining an estimated density map with 1 output channel.

[0078] Step 95: Sum the pixel values ​​of the density map to obtain the population size.

[0079] The technological advancements achieved by this invention, due to the adoption of the above technical solutions, are as follows: by replacing multi-head attention with average pooling as the core sub-module of the feature extraction module in each stage, the number of model parameters is reduced on the basis of the traditional Transformer structure, thereby improving the training speed of the model and reducing the requirements of training the model on hardware devices.

[0080] 2. By adding the complementary information of the two modalities at corresponding pixel positions in the modal mixer; calculating the attention weight map using multi-head self-attention; and then fully fusing the above addition results with the attention weight map bit by bit, this method avoids the problem of missing global context information caused by existing Transformer-based RGB-T methods using the query of another modality to calculate the attention of this modality during modal fusion, thus achieving the goal of improving counting performance under poor lighting conditions.

[0081] 3. By upsampling the cross-modal feature maps output in the third stage and the cross-modal feature maps output in the second stage to the size of the cross-modal feature maps output in the first stage during the regression process, the distortion problem caused by upsampling is weakened, thereby improving the accuracy of the final population counting results. Attached Figure Description

[0082] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0083] Figure 1 This is a flowchart of the present invention;

[0084] Figure 2 This is a schematic diagram of the overall network structure of the present invention;

[0085] Figure 3 A schematic diagram of the modal mixer module designed for this invention;

[0086] Figure 4 This is a schematic diagram of the regression head designed for this invention;

[0087] Figure 5 This is a schematic diagram illustrating the use of the present invention for population counting density estimation. Detailed Implementation

[0088] The present invention will be further described in detail below with reference to embodiments:

[0089] like Figure 2 The diagram shows a schematic of a Transformer-based cross-modal crowd counting system, which includes a first-order feature extraction module, a first-order cross-modal mixer, a second-order feature extraction module, a second-order cross-modal mixer, a third-order feature extraction module, a third-order cross-modal mixer, and a regression head connected in sequence.

[0090] The first-order feature extraction module consists of an overlapping patch embedding module, an average pooling module, and a feedforward module. The overlapping patch embedding module includes, in sequence, a convolutional layer with a kernel size of 7×7, 128 output channels, and a stride of 4, and a group normalization regularization operation. The average pooling module includes, in sequence, an average pooling layer with a pooling window size of 3×3 and a stride of 1, a subtraction operation, and an addition operation. The feedforward module includes, in sequence, a convolutional layer with a kernel size of 1×1, 512 output channels, and a stride of 1, a GELU activation function, and a convolutional layer with a kernel size of 1×1, 128 output channels, and a stride of 1.

[0091] The second-order feature extraction module consists of an overlapping patch embedding module, an average pooling module, and a feedforward module. The overlapping patch embedding module includes, in sequence, a convolutional layer with a kernel size of 3×3, 320 output channels, and a stride of 2, and a group normalization regularization operation. The average pooling module includes, in sequence, an average pooling layer with a pooling window size of 3×3 and a stride of 1, a subtraction operation, and an addition operation. The feedforward module includes, in sequence, a convolutional layer with a kernel size of 1×1, 1280 output channels, and a stride of 1, a GELU activation function, and a convolutional layer with a kernel size of 1×1, 320 output channels, and a stride of 1.

[0092] The third-order feature extraction module consists of an overlapping patch embedding module, an average pooling module, and a feedforward module. The overlapping patch embedding module includes, in sequence, a convolutional layer with a kernel size of 3×3, 512 output channels, and a stride of 2, and a group normalization regularization operation. The average pooling module includes, in sequence, an average pooling layer with a pooling window size of 3×3 and a stride of 1, a subtraction operation, and an addition operation. The feedforward module includes, in sequence, a convolutional layer with a kernel size of 1×1, 2048 output channels, and a stride of 1, a GELU activation function, and a convolutional layer with a kernel size of 1×1, 512 output channels, and a stride of 1.

[0093] The structures of the first-order transmodal mixer, second-order transmodal mixer, and third-order transmodal mixer are as follows: Figure 3 As shown.

[0094] The first-order cross-modal mixer consists of a block embedding layer, four identical Transformer encoder layers, and a reshaping module. The block embedding layer includes, in sequence, an addition operation, a convolutional layer with a kernel size of 3×3, 128 output channels, and a stride of 1, a flattening operation, and a transpose operation. The Transformer encoder layers include, in sequence, a multi-head self-attention layer with 8 heads, a feedforward layer, and a normalization operation.

[0095] The multi-head self-attention layer includes a query vector calculation module, a key vector and value vector calculation module, and a multi-head self-attention calculation module. The query vector calculation module includes a fully connected layer and a reshaping operation. The key vector and value vector calculation module includes a fully connected layer and a reshaping operation.

[0096] The feedforward layer consists of a fully connected layer with 512 neurons, a GELU activation function, and a fully connected layer with 128 neurons.

[0097] The reshaping module includes a reshaping operation, a multiplication operation, and an addition operation.

[0098] The second-order cross-modal mixer consists of a block embedding layer, four identical Transformer encoder layers, and a reshaping module. The block embedding layer includes, in sequence, an addition operation, a convolutional layer with a kernel size of 3×3, 320 output channels, and a stride of 1, a flattening operation, and a transpose operation.

[0099] The Transformer encoder layer consists of a multi-head self-attention layer with eight heads, a feedforward layer, and a normalization operation.

[0100] The multi-head self-attention layer includes a query vector calculation module, a key vector and value vector calculation module, and a multi-head self-attention calculation module. The query vector calculation module includes a fully connected layer and a reshaping operation. The key vector and value vector calculation module includes a fully connected layer and a reshaping operation.

[0101] The feedforward layer consists of a fully connected layer with 1280 neurons, a GELU activation function, and a fully connected layer with 320 neurons.

[0102] The reshaping module includes a reshaping operation, a multiplication operation, and an addition operation.

[0103] The third-order cross-modal mixer consists of a block embedding layer, four identical Transformer encoder layers, and a reshaping module. The block embedding layer includes, in sequence, an addition operation, a convolutional layer with a kernel size of 3×3, 512 output channels, and a stride of 1, a flattening operation, and a transpose operation.

[0104] The Transformer encoder layer consists of a multi-head self-attention layer with eight heads, a feedforward layer, and a normalization operation.

[0105] The multi-head self-attention layer includes a query vector calculation module, a key vector and value vector calculation module, and a multi-head self-attention calculation module. The query vector calculation module includes a fully connected layer and a reshaping operation. The key vector and value vector calculation module includes a fully connected layer and a reshaping operation.

[0106] The feedforward layer consists of a fully connected layer with 2048 neurons, a GELU activation function, and a fully connected layer with 512 neurons.

[0107] The reshaping module includes a reshaping operation, a multiplication operation, and an addition operation.

[0108] Regression head structure such as Figure 4 As shown.

[0109] Regression head: upsampling operation, summation operation, convolutional layer with kernel size of 3×3 and stride of 1 and convolutional layer with kernel size of 1×1 and stride of 1.

[0110] Using the above-mentioned technical system, a cross-modal crowd counting method based on Transformer can be implemented, as shown in the flowchart below. Figure 1 As shown, the specific steps are as follows:

[0111] Step 1: Input the infrared thermal image and the visible light image into the first-order feature extraction module to obtain the first-order visible light feature and the first-order infrared thermal feature, respectively.

[0112] Step 11: Input the visible light image and the infrared thermal image into the overlap patch embedding module to obtain the first-order regularized block visible light feature map and the first-order regularized block infrared thermal feature map.

[0113] Step 111: Input the visible light image and the infrared thermal image into a convolutional layer with a kernel size of 7×7, an output channel number of 128, and a stride of 4 to obtain block-based first-order visible light feature maps and block-based first-order infrared thermal feature maps.

[0114] Step 112: Perform group normalization regularization on the block-based first-order visible light features and block-based first-order infrared thermal features to obtain first-order regularized block-based visible light feature map and first-order regularized block-based infrared thermal feature map.

[0115] Step 12: Input the first-order regularized block visible light feature map and the first-order regularized block infrared thermal feature map into the average pooling module to obtain the first-order pooling block visible light feature map and the first-order pooling block infrared thermal feature map.

[0116] Step 121: Input the first-order regularized block visible light feature map and the first-order regularized block infrared thermal feature map into an average pooling layer with a pooling window size of 3×3 and a step number of 1 to obtain the first-order regularized pooling block visible light feature map and the first-order regularized pooling block infrared thermal feature map.

[0117] Step 122: Subtract the corresponding pixels of the first-order regularized pooling block visible light feature map and the first-order regularized pooling block infrared thermal feature map from the corresponding pixels of the first-order regularized block visible light feature map and the first-order regularized block infrared thermal feature map, respectively, to obtain the first-order residual pooling visible light feature map and the first-order residual pooling infrared thermal feature map.

[0118] Step 123: Add the first-order residual pooling visible light block feature map and the first-order residual pooling infrared thermal block feature map to the corresponding pixels of the block first-order visible light feature map and the block first-order infrared thermal feature map, respectively, to obtain the first-order pooling block visible light feature map and the first-order pooling block infrared thermal feature map.

[0119] Step 13: Input the first-order pooling block visible light feature map and the first-order pooling block infrared thermal feature map into the feedforward layer to obtain the first-order visible light feature and the first-order infrared thermal feature.

[0120] Step 131: Input the first-order pooling block visible light feature map and the first-order pooling block infrared thermal feature map into a convolutional layer with a kernel size of 1×1, an output channel number of 512, and a stride of 1 to obtain the concatenated first-order pooling block visible light feature map and the concatenated first-order pooling block infrared thermal feature map.

[0121] Step 132: Input the concatenated first-order pooling block visible light feature map and the concatenated first-order pooling block infrared thermal feature map into the GELU activation function to obtain the concatenated activated first-order pooling block visible light feature map and the concatenated activated first-order pooling block infrared thermal feature map.

[0122] Step 133: Input the concatenated activated first-order pooling block visible light feature map and the concatenated activated first-order pooling block infrared thermal feature map into a convolutional layer with a kernel size of 1×1, an output channel number of 128, and a stride of 1 to obtain the first-order visible light feature and the first-order infrared thermal feature.

[0123] Step 2: Input the first-order visible light feature and the first-order infrared thermal feature into the first-order modal mixer to obtain the first-order cross-modal feature and the first-order attention weight map.

[0124] Step 21: Input the first-order visible light feature and the first-order infrared thermal feature into the block embedding layer to obtain the flattened first-order cross-modal feature.

[0125] Step 211: Add the first-order visible light feature and the first-order infrared thermal feature at the corresponding pixel positions to obtain the first-order fused feature.

[0126] Step 212: Input the first-order fusion feature into a convolutional layer with a kernel size of 3×3, an output channel number of 128, and a stride of 1 to obtain the first-order block fusion feature.

[0127] Step 213: One-dimensionalize and transpose the first-order block fusion features to obtain flattened first-order cross-modal features.

[0128] Step 22: Input the flattened first-order cross-modal features into four Transformer encoder layers to obtain a first-order attention weight map.

[0129] Step 221: Input the flattened first-order cross-modal features into the key and value vector calculation module of the multi-head self-attention layer to obtain the key and value vectors.

[0130] The formulas for calculating the query vector, key vector, and value vector are as follows:

[0131]

[0132] Where Q, K, and V are the output vectors representing the first-order query vector, the first-order key vector, and the first-order value vector, respectively. It is the corresponding projection matrix, Z i It is the input first-order, one-dimensional vector.

[0133] Step 222: Input the query vector, key vector, and value vector into the multi-head self-attention calculation module to obtain the self-attention result; the formula for calculating the self-attention result is:

[0134]

[0135] MHSA = Concat(head1,…,head) h W o

[0136] head i L is the attention value of the i-th head. i W is the length of the input one-dimensional vector. o It is the projection matrix, Concat(·) represents the concatenation operation, and MHSA is the result of self-attention.

[0137] Step 223: Input the self-attention result into the fully connected layer containing 512 neurons in the feedforward layer to obtain the connected self-attention result; then input it into the GELU activation function in the feedforward layer, and input the result into the fully connected layer containing 512 neurons in the feedforward layer, and normalize it to obtain the first-order attention weight map calculated once.

[0138] Step 224: Input the first-order attention weight map calculated once into the Transformer encoder layer to obtain the first-order attention weight map calculated twice; input the first-order attention weight map calculated twice into the Transformer encoder layer to obtain the first-order attention weight map calculated three times; input the first-order attention weight map calculated three times into the Transformer encoder layer to obtain the first-order attention weight map.

[0139] Step 23: Input the first-order attention weight map and the first-order fusion feature into the reshaping module to obtain the first-order attention weight map and the first-order cross-modal feature map.

[0140] Step 231: Reshape the first-order attention weight map to obtain the reshaped first-order attention weight map.

[0141] Step 232: Multiply the reshaped first-order attention weight map with the first-order cross-modal fusion feature by pixel position to obtain the weighted first-order cross-modal fusion feature.

[0142] Step 233: Add the weighted first-order cross-modal fusion features and the first-order cross-modal fusion features according to pixel position to obtain the first-order cross-modal features.

[0143] Step 3: Multiply the first-order attention weight map by the first-order visible light feature and the first-order infrared thermal feature by pixel position, and then add them by pixel position to obtain the first-order enhanced visible light feature and the first-order enhanced infrared thermal feature.

[0144] Step 31: Multiply the first-order attention weight map output in Step 2 element by element on the corresponding channel with the first-order visible light feature and the first-order infrared thermal feature output in Step 1 to obtain the multiplied first-order visible light feature map and the multiplied first-order infrared thermal feature map.

[0145] Step 32: Add the multiplied first-order visible light feature map and the multiplied first-order infrared thermal feature map to the first-order visible light feature map and the first-order infrared thermal feature map respectively on the corresponding channels to obtain the first-order enhanced visible light feature map and the first-order enhanced infrared thermal feature map.

[0146] Step 4: Input the first-order enhanced visible light feature and the first-order enhanced infrared thermal feature into the second-order feature extraction module to obtain the second-order visible light feature and the second-order infrared thermal feature.

[0147] Step 41: Input the first-order enhanced visible light feature map and the first-order enhanced infrared thermal feature map into the overlapping patch embedding module to obtain the second-order regularized block visible light feature map and the second-order regularized block infrared thermal feature map.

[0148] Step 411: Input the visible light image and infrared thermal image into a convolutional layer with a kernel size of 3×3, an output channel number of 320, and a stride of 2 to obtain block-based second-order visible light features and block-based second-order infrared thermal features.

[0149] Step 412: Perform group normalization regularization on the block second-order visible light features and block second-order infrared thermal features to obtain second-order regularized block visible light feature map and second-order regularized block infrared thermal feature map.

[0150] Step 42: Input the second-order regularized block visible light feature map and the second-order regularized block infrared thermal feature map into the average pooling module to obtain the second-order pooling block visible light feature map and the second-order pooling block infrared thermal feature map.

[0151] Step 421: Input the second-order regularized block visible light feature map and the second-order regularized block infrared thermal feature map into an average pooling layer with a pooling window size of 3×3 and a step number of 1 to obtain the second-order regularized pooling block visible light feature map and the second-order regularized pooling block infrared thermal feature map.

[0152] Step 422: Subtract the corresponding pixels of the second-order regularized pooling block visible light feature map and the second-order regularized pooling block infrared thermal feature map from the corresponding pixels of the second-order regularized block visible light feature map and the second-order regularized block infrared thermal feature map, respectively, to obtain the second-order residual pooling visible light feature map and the second-order residual pooling infrared thermal feature map.

[0153] Step 423: Add the second-order residual pooling visible light block feature map and the second-order residual pooling infrared thermal block feature map to the corresponding pixel positions of the block second-order visible light feature map and the block second-order infrared thermal feature map, respectively, to obtain the second-order pooling block visible light feature map and the second-order pooling block infrared thermal feature map.

[0154] Step 43: Input the second-order pooling block visible light feature map and the second-order pooling block infrared thermal feature map into the feedforward layer to obtain the second-order visible light feature and the second-order infrared thermal feature.

[0155] Step 431: Input the second-order pooling block visible light feature map and the second-order pooling block infrared thermal feature map into a convolutional layer with a kernel size of 1×1, an output channel number of 1280, and a stride of 1 to obtain the connected second-order pooling block visible light feature map and the connected second-order pooling block infrared thermal feature map.

[0156] Step 432: Input the concatenated second-order pooling block visible light feature map and the concatenated second-order pooling block infrared thermal feature map into the GELU activation function to obtain the concatenated activated second-order pooling block visible light feature map and the concatenated activated second-order pooling block infrared thermal feature map.

[0157] Step 433: Input the concatenated activated second-order pooling block visible light feature map and the concatenated activated second-order pooling block infrared thermal feature map into a convolutional layer with a kernel size of 1×1, an output channel number of 320, and a stride of 1 to obtain second-order visible light features and second-order infrared thermal features.

[0158] Step 5: Input the second-order visible light features and second-order infrared thermal features into the second-order modal mixer to obtain the second-order cross-modal features and the second-order attention weight map.

[0159] Step 51: Input the second-order visible light features and second-order infrared thermal features into the block embedding layer to obtain the flattened second-order cross-modal features.

[0160] Step 511: Add the second-order visible light feature and the second-order infrared thermal feature at the corresponding pixel positions to obtain the second-order fused feature.

[0161] Step 512: Input the second-order fusion feature into a convolutional layer with a kernel size of 3×3, an output channel number of 320, and a stride of 1 to obtain the second-order block fusion feature.

[0162] Step 513: Flatten the second-order block fusion feature and transpose it to obtain the flattened second-order cross-modal feature.

[0163] Step 52: Input the flattened second-order cross-modal features into four Transformer encoder layers consecutively to obtain a second-order attention weight map.

[0164] Step 521: Input the flattened second-order cross-modal features into the query vector module of the multi-head self-attention layer to obtain the query vector.

[0165] Step 522: Input the flattened second-order cross-modal features into the key vector and value vector calculation module of the multi-head self-attention layer to obtain the key vector and value vector.

[0166] The formulas for calculating the query vector, key vector, and value vector are as follows:

[0167]

[0168] Where Q, K, and V are the output vectors representing the second-order query vector, the second-order key vector, and the second-order value vector, respectively. It is the corresponding projection matrix, Z i It is the input second-order one-dimensional vector.

[0169] Step 523: Input the query vector, key vector, and value vector into the multi-head self-attention calculation module to obtain the self-attention result; the formula for calculating the self-attention result is:

[0170]

[0171] MHSA = Concat(head1,…,head) h W o

[0172] head i L is the attention value of the i-th head. i W is the length of the input one-dimensional vector. o It is the projection matrix, Concat(·) represents the concatenation operation, and MHSA is the result of self-attention.

[0173] Step 524: Input the self-attention result into the fully connected layer containing 1280 neurons in the feedforward layer to obtain the connected self-attention result; then input it into the GELU activation function in the feedforward layer, and input the result into the second fully connected layer containing 320 neurons in the feedforward layer, and normalize it to obtain the second-order attention weight map calculated once.

[0174] Step 525: Input the second-order attention weight map calculated once into the Transformer encoder layer to obtain the second-order attention weight map calculated twice; input the second-order attention weight map calculated twice into the Transformer encoder layer to obtain the second-order attention weight map calculated three times; input the second-order attention weight map calculated three times into the Transformer encoder layer to obtain the second-order attention weight map.

[0175] Step 53: Input the second-order attention weight map and the second-order fusion feature into the reshaping module to obtain the second-order attention weight map and the second-order cross-modal feature map.

[0176] Step 531: Reshape the second-order attention weight map to obtain a reshaped second-order attention weight map.

[0177] Step 532: Multiply the reshaped second-order attention weight map with the second-order cross-modal fusion feature by pixel position to obtain the weighted second-order cross-modal fusion feature.

[0178] Step 533: Add the weighted second-order cross-modal fusion features and the second-order cross-modal fusion features according to pixel position to obtain the second-order cross-modal feature map.

[0179] Step 6: Multiply the second-order attention weight map with the second-order visible light feature and the infrared thermal feature by pixel position, and then add them by pixel position to obtain the second-order enhanced visible light feature and the second-order enhanced infrared thermal feature.

[0180] Step 61: Multiply the second-order attention weight map output in Step 5 element by element on the corresponding channel with the second-order visible light feature and the second-order infrared thermal feature output in Step 4 to obtain the multiplied second-order visible light feature map and the multiplied second-order infrared thermal feature map.

[0181] Step 62: Add the multiplied second-order visible light feature map and the multiplied second-order infrared thermal feature map to the second-order visible light feature map and the second-order infrared thermal feature map respectively on the corresponding channels to obtain the second-order enhanced visible light feature map and the second-order enhanced infrared thermal feature map.

[0182] Step 7: Input the second-order enhanced visible light feature and the second-order enhanced infrared thermal feature into the third-order feature extraction module to obtain the third-order visible light feature and the third-order infrared thermal feature.

[0183] Step 71: Input the second-order enhanced visible light feature map and the second-order enhanced infrared thermal feature map into the overlapping patch embedding module to obtain the third-order regularized block visible light feature map and the third-order regularized block infrared thermal feature map.

[0184] Step 711: Input the visible light image and infrared thermal image into a convolutional layer with a kernel size of 3×3, an output channel number of 512, and a stride of 2 to obtain block-based third-order visible light features and block-based third-order infrared thermal features.

[0185] Step 712: Perform group normalization regularization on the block-based third-order visible light features and block-based third-order infrared thermal features to obtain third-order regularized block-based visible light feature map and third-order regularized block-based infrared thermal feature map.

[0186] Step 72: Input the third-order regularized block visible light feature map and the third-order regularized block infrared thermal feature map into the average pooling module to obtain the third-order pooling block visible light feature map and the third-order pooling block infrared thermal feature map.

[0187] Step 721: Input the third-order regularized block visible light feature map and the third-order regularized block infrared thermal feature map into an average pooling layer with a pooling window size of 3×3 and a step number of 1 to obtain the third-order regularized pooling block visible light feature map and the third-order regularized pooling block infrared thermal feature map.

[0188] Step 722: Subtract the corresponding pixels of the third-order regularized pooling block visible light feature map and the third-order regularized pooling block infrared thermal feature map from the corresponding pixels of the third-order regularized block visible light feature map and the third-order regularized block infrared thermal feature map, respectively, to obtain the third-order residual pooling visible light feature map and the third-order residual pooling infrared thermal feature map.

[0189] Step 723: Add the third-order residual pooling visible light block feature map and the third-order residual pooling infrared thermal block feature map to the corresponding pixel positions of the block third-order visible light feature map and the block third-order infrared thermal feature map, respectively, to obtain the third-order pooling block visible light feature map and the third-order pooling block infrared thermal feature map.

[0190] Step 73: Input the third-order pooling block visible light feature map and the third-order pooling block infrared thermal feature map into the feedforward layer to obtain the third-order visible light feature and the third-order infrared thermal feature.

[0191] Step 731: Input the third-order pooling block visible light feature map and the third-order pooling block infrared thermal feature map into a convolutional layer with a kernel size of 1×1, an output channel number of 2048, and a stride of 1 to obtain the concatenated third-order pooling block visible light feature map and the concatenated third-order pooling block infrared thermal feature map.

[0192] Step 732: Input the concatenated third-order pooling block visible light feature map and the concatenated third-order pooling block infrared thermal feature map into the GELU activation function to obtain the concatenated activated third-order pooling block visible light feature map and the concatenated activated third-order pooling block infrared thermal feature map.

[0193] Step 733: Input the concatenated activated third-order pooling block visible light feature map and the concatenated activated third-order pooling block infrared thermal feature map into a convolutional layer with a kernel size of 1×1, an output channel number of 512, and a stride of 1 to obtain the third-order visible light feature and the third-order infrared thermal feature.

[0194] Step 8: Input the third-order visible light feature and the third-order infrared thermal feature into the third-order mode mixer to obtain the third-order cross-modal feature.

[0195] Step 81: Input the third-order visible light features and the third-order infrared thermal features into the block embedding layer to obtain the flattened third-order cross-modal features.

[0196] Step 811: Add the third-order visible light feature and the third-order infrared thermal feature at the corresponding pixel positions to obtain the third-order fused feature.

[0197] Step 812: Input the third-order fusion feature into a convolutional layer with a kernel size of 3×3, an output channel number of 512, and a stride of 1 to obtain the third-order block fusion feature.

[0198] Step 813: Flatten the third-order block fusion feature and transpose it to obtain the flattened third-order cross-modal feature.

[0199] Step 82: Input the flattened third-order cross-modal features into the Transformer encoder layer four times to obtain the third-order attention weight map.

[0200] Step 821: Input the flattened third-order cross-modal features into the query vector module of the multi-head self-attention layer to obtain the query vector.

[0201] Step 822: Input the flattened third-order cross-modal features into the key vector and value vector calculation module of the multi-head self-attention layer to obtain the key vector and value vector.

[0202] The formulas for calculating the query vector, key vector, and value vector are as follows:

[0203]

[0204] Where Q, K, and V are the output vectors representing the third-order query vector, the third-order key vector, and the third-order value vector, respectively. It is the corresponding projection matrix, Z i It is the input third-order one-dimensional vector;

[0205] Step 823: Input the query vector, key vector, and value vector into the multi-head self-attention calculation module to obtain the self-attention result; the formula for calculating the self-attention result is:

[0206]

[0207] MHSA = Concat(head1,…,head) h W o

[0208] head i L is the attention value of the i-th head. i W is the length of the input one-dimensional vector. o It is the projection matrix, Concat(·) represents the concatenation operation, and MHSA is the result of self-attention.

[0209] Step 824: Input the self-attention result into the fully connected layer containing 2048 neurons in the feedforward layer to obtain the connected self-attention result; then input it into the GELU activation function in the feedforward layer, and input the result into the fully connected layer containing 2048 neurons in the feedforward layer, and normalize it to obtain the third-order attention weight map calculated once.

[0210] Step 825: Input the third-order attention weight map calculated once into the Transformer encoder layer to obtain the third-order attention weight map calculated twice; input the third-order attention weight map calculated twice into the Transformer encoder layer to obtain the third-order attention weight map calculated three times; input the third-order attention weight map calculated three times into the Transformer encoder layer to obtain the third-order attention weight map.

[0211] Step 83: Input the third-order attention weight map and the third-order fusion feature into the reshaping module to obtain the third-order attention weight map and the third-order cross-modal feature map.

[0212] Step 831: Reshape the third-order attention weight map to obtain the reshaped third-order attention weight map.

[0213] Step 832: Multiply the reshaped third-order attention weight map with the third-order cross-modal fusion feature by pixel position to obtain the weighted third-order cross-modal fusion feature.

[0214] Step 833: Add the weighted three-order cross-modal fusion features and the three-order cross-modal fusion features according to pixel position to obtain the three-order cross-modal feature map.

[0215] Step 9: Input the first-order cross-modal features, second-order cross-modal features, and third-order cross-modal features into the regression head to obtain the estimated density map, and sum the pixel values ​​of the estimated density map to obtain the population size.

[0216] Step 91: Upsample the third-order transmodal features obtained in Step 8 by a factor of 2 using bilinear interpolation, and then connect them with the second-order transmodal features obtained in Step 5 to obtain the feature map after the first connection.

[0217] Step 92: Smooth the feature map after the first connection, and then reduce the number of channels by using a convolutional layer with a kernel of 3×3 and a stride of 1 to obtain the convolutional feature map.

[0218] Step 93: Upsample the convolutional feature map by a factor of 2 using bilinear interpolation, and then perform channel concatenation with the first-order cross-modal feature obtained in Step 2 to obtain the second concatenated feature map.

[0219] Step 94: Dimensionality reduction is performed on the feature map after the second connection using convolutional layers with kernel size of 3×3 and stride of 1 and convolutional layers with kernel size of 1×1 and stride of 1, finally obtaining an estimated density map with 1 output channel.

[0220] Step 95: Sum the pixel values ​​of the density map to obtain the population size.

[0221] A Transformer-based cross-modal crowd counting model is trained using a crowd density estimation map and Bayesian loss as the loss function, resulting in a trained deep neural network model. After generating crowd density maps from visible light and infrared thermal images of the number of people to be estimated, the model's iteration error is determined by comparing the actual and generated crowd density maps based on the total loss function. This error is then backpropagated to adjust the parameters of the crowd counting model, thus optimizing the Transformer-based cross-modal crowd counting model. The Adam optimization algorithm is used to update the parameters in each optimization iteration until the loss function converges to a small value. The parameters and the trained model are then saved.

[0222] By inputting any pair of visible light images and infrared thermal images of the number of people to be estimated into the trained Transformer-based cross-modal crowd counting model, a crowd density map can be obtained. The summation of the pixel values ​​in the crowd density map yields the crowd count result. Figure 5 This is a schematic diagram illustrating the use of the present invention to count crowds using a visible light image and an infrared thermal image of the number of people to be estimated.

[0223] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A cross-modal crowd counting method based on Transformer, characterized in that, The steps are as follows: Step 1: Input the infrared thermal image and the visible light image into the first-order feature extraction module to obtain the first-order visible light features and the first-order infrared thermal features, respectively; Step 2: Input the first-order visible light features and the first-order infrared thermal features into the first-order modal mixer to obtain the first-order cross-modal features and the first-order attention weight map; The specific steps are as follows: Step 21: Input the first-order visible light feature and the first-order infrared thermal feature into the block embedding layer to obtain the flattened first-order cross-modal feature; Step 211: Add the first-order visible light feature and the first-order infrared thermal feature at corresponding pixel positions to obtain the first-order fused feature; Step 212: Input the first-order fusion feature into a convolutional layer with a kernel size of 3×3, an output channel number of 128, and a stride of 1 to obtain the first-order block fusion feature; Step 213: One-dimensionalize and transpose the first-order block fusion features to obtain flattened first-order cross-modal features; Step 22: Input the flattened first-order cross-modal features into four Transformer encoder layers consecutively to obtain a first-order attention weight map; Step 221: Input the flattened first-order cross-modal features into the query vector module of the multi-head self-attention layer to obtain the query vector; Step 222: Input the flattened first-order cross-modal features into the key and value vector calculation module of the multi-head self-attention layer to obtain the key and value vectors. The formulas for calculating the query vector, key vector, and value vector are as follows: in, The output vectors represent the first-order query vector, the first-order key vector, and the first-order value vector, respectively. It is the corresponding projection matrix. It is the input first-order, one-dimensional vector; Step 223: Input the query vector, key vector, and value vector into the multi-head self-attention calculation module to obtain the self-attention result; the formula for calculating the self-attention result is: in, It is the attention value of the i-th head. It is the length of the input one-dimensional vector. It is a projection matrix. This indicates a splicing operation. It is a result of self-attention; Step 224: Input the self-attention result into the fully connected layer containing 512 neurons in the feedforward layer to obtain the connected self-attention result; then input it into the GELU activation function in the feedforward layer, and input the result into the fully connected layer containing 128 neurons in the feedforward layer, and normalize it to obtain the first-order attention weight map calculated once. Step 225: Input the first-order attention weight map calculated once into the Transformer encoder layer to obtain the first-order attention weight map calculated twice; input the first-order attention weight map calculated twice into the Transformer encoder layer to obtain the first-order attention weight map calculated three times; input the first-order attention weight map calculated three times into the Transformer encoder layer to obtain the first-order attention weight map. Step 23: Input the first-order attention weight map and the first-order fusion feature into the reshaping module to obtain the first-order attention weight map and the first-order cross-modal feature map; Step 231: Reshape the first-order attention weight map to obtain a reshaped first-order attention weight map; Step 232: Multiply the reshaped first-order attention weight map with the first-order cross-modal fusion feature by pixel position to obtain the weighted first-order cross-modal fusion feature; Step 233: Add the weighted first-order cross-modal fusion features and the first-order cross-modal fusion features according to pixel position to obtain the first-order cross-modal features; Step 3: Multiply the first-order attention weight map with the first-order visible light feature and the first-order infrared thermal feature by pixel position and add them by pixel position respectively to obtain the first-order enhanced visible light feature and the first-order enhanced infrared thermal feature. The specific steps are as follows: Step 31: Multiply the first-order attention weight map output in Step 2 element by element on the corresponding channel with the first-order visible light feature and the first-order infrared thermal feature output in Step 1 to obtain the multiplied first-order visible light feature map and the multiplied first-order infrared thermal feature map. Step 32: Add the multiplied first-order visible light feature map and the multiplied first-order infrared thermal feature map to the first-order visible light feature map and the first-order infrared thermal feature map respectively on the corresponding channels to obtain the first-order enhanced visible light feature map and the first-order enhanced infrared thermal feature map. Step 4: Input the first-order enhanced visible light feature and the first-order enhanced infrared thermal feature into the second-order feature extraction module to obtain the second-order visible light feature and the second-order infrared thermal feature; Step 5: Input the second-order visible light features and second-order infrared thermal features into the second-order modal mixer to obtain the second-order cross-modal features and the second-order attention weight map; Step 6: Multiply the second-order attention weight map with the second-order visible light feature and the infrared thermal feature by pixel position, and then add them by pixel position to obtain the second-order enhanced visible light feature and the second-order enhanced infrared thermal feature; the calculation process of multiplication and addition in step 6 is the same as in step 3. Step 7: Input the second-order enhanced visible light feature and the second-order enhanced infrared thermal feature into the third-order feature extraction module to obtain the third-order visible light feature and the third-order infrared thermal feature; Step 8: Input the third-order visible light feature and the third-order infrared thermal feature into the third-order mode mixer to obtain the third-order cross-modal feature; Step 9: Input the first-order cross-modal features, second-order cross-modal features, and third-order cross-modal features into the regression head to obtain the estimated density map, and sum the pixel values ​​of the estimated density map to obtain the population size.

2. The method for cross-modal crowd counting based on Transformer according to claim 1, characterized in that, The specific steps of step 1 are as follows: Step 11: Input the visible light image and the infrared thermal image into the overlap patch embedding module to obtain the first-order regularized block visible light feature map and the first-order regularized block infrared thermal feature map; Step 111: Input the visible light image and the infrared thermal image into a convolutional layer with a kernel size of 7×7, an output channel number of 128, and a stride of 4 to obtain block-based first-order visible light feature maps and block-based first-order infrared thermal feature maps. Step 112: Perform group normalization regularization on the block-based first-order visible light features and block-based first-order infrared thermal features to obtain first-order regularized block-based visible light feature map and first-order regularized block-based infrared thermal feature map. Step 12: Input the first-order regularized block visible light feature map and the first-order regularized block infrared thermal feature map into the average pooling module to obtain the first-order pooling block visible light feature map and the first-order pooling block infrared thermal feature map. Step 121: Input the first-order regularized block visible light feature map and the first-order regularized block infrared thermal feature map into the average pooling layer with a pooling window size of 3×3 and a step number of 1 to obtain the first-order regularized pooling block visible light feature map and the first-order regularized pooling block infrared thermal feature map. Step 122: Subtract the corresponding pixels of the first-order regularized pooling block visible light feature map and the first-order regularized pooling block infrared thermal feature map from the corresponding pixels of the first-order regularized block visible light feature map and the first-order regularized block infrared thermal feature map, respectively, to obtain the first-order residual pooling visible light block feature map and the first-order residual pooling infrared thermal block feature map. Step 123: Add the first-order residual pooling visible light block feature map and the first-order residual pooling infrared thermal block feature map to the corresponding pixels of the block first-order visible light feature map and the block first-order infrared thermal feature map, respectively, to obtain the first-order pooling block visible light feature map and the first-order pooling block infrared thermal feature map. Step 13: Input the first-order pooling block visible light feature map and the first-order pooling block infrared thermal feature map into the feedforward layer to obtain the first-order visible light feature and the first-order infrared thermal feature. Step 131: Input the first-order pooling block visible light feature map and the first-order pooling block infrared thermal feature map into a convolutional layer with a kernel size of 1×1, an output channel number of 512, and a stride of 1 to obtain the connected first-order pooling block visible light feature map and the connected first-order pooling block infrared thermal feature map. Step 132: Input the concatenated first-order pooling block visible light feature map and the concatenated first-order pooling block infrared thermal feature map into the GELU activation function to obtain the concatenated activated first-order pooling block visible light feature map and the concatenated activated first-order pooling block infrared thermal feature map. Step 133: Input the concatenated activated first-order pooling block visible light feature map and the concatenated activated first-order pooling block infrared thermal feature map into the convolution kernel with a size of 1.

1. A convolutional layer with 128 output channels and a stride of 1 produces first-order visible light features and first-order infrared thermal features.

3. The method for cross-modal crowd counting based on Transformer according to claim 1, characterized in that, Step 4 involves inputting the first-order enhanced visible light features and the first-order enhanced infrared thermal features into the second-order feature extraction module to obtain second-order visible light features and second-order infrared thermal features. The calculation process in the second-order feature extraction module is the same as that in the first-order feature extraction module in Step 1. Step 7 involves inputting the second-order enhanced visible light features and the second-order enhanced infrared thermal features into the third-order feature extraction module to obtain third-order visible light features and third-order infrared thermal features. The calculation process in the third-order feature extraction module is the same as that in the first-order feature extraction module in Step 1. Step 5 involves inputting the second-order visible light features and the second-order infrared thermal features into the second-order modal mixer to obtain second-order cross-modal features and a second-order attention weight map. The calculation process in the second-order modal mixer is the same as that in the first-order modal mixer in Step 2. Step 8 involves inputting the third-order visible light features and the third-order infrared thermal features into the third-order modal mixer to obtain third-order cross-modal features. The calculation process in the third-order modal mixer is the same as that in the first-order modal mixer in Step 2.

4. The method for cross-modal crowd counting based on Transformer according to claim 1, characterized in that, Step 9 is detailed below: Step 91: Upsample the third-order transmodal features obtained in Step 8 by a factor of 2 using bilinear interpolation, and then connect them with the second-order transmodal features obtained in Step 5 to obtain the feature map after the first connection. Step 92: Smooth the feature map after the first connection, and then reduce the number of channels by using a convolutional layer with a kernel of 3×3 and a stride of 1 to obtain the convolutional feature map. Step 93: Upsample the convolutional feature map by a factor of 2 using bilinear interpolation, and then concatenate it with the first-order cross-modal feature obtained in Step 2 to obtain the second concatenated feature map. Step 94: The feature map after the second connection is reduced in dimensionality by using a convolutional layer with a kernel size of 3×3 and a stride of 1 and a convolutional layer with a kernel size of 1×1 and a stride of 1, and finally an estimated density map with 1 output channel is obtained. Step 95: Sum the pixel values ​​of the density map to obtain the population size.

Images