Cross-modal crowd counting method based on CNN and transformer
By combining deep neural network models of CNN and transformer, and utilizing the complementary information of RGB images and thermal images, cross-modal global features are extracted, solving the problem that CNN cannot extract global information in existing methods and achieving more accurate crowd counting.
Patent Information
- Application Number
- CN202211208011.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-30
- Publication Date
- 2026-03-06
- Estimated Expiration
- 2042-09-30
AI Technical Summary
Existing cross-modal crowd counting methods only use CNNs as the backbone network, which cannot effectively extract global information from the image, resulting in inaccurate counting.
A cross-modal crowd counting method based on CNN and transformer is adopted. By constructing a deep neural network model that combines CNN and transformer, the complementary information of RGB images and thermal images is used to extract cross-modal global features, fuse feature maps and perform attention module processing, and finally generate an accurate density map.
It improves the accuracy of crowd counting, better reflects the real crowd distribution in complex scenarios, and generates more accurate crowd density maps.
Smart Images
Figure CN115526862B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a cross-modal crowd counting method based on CNN and transformer, belonging to the field of computer vision. Background Technology
[0002] Crowd counting involves analyzing images in unconstrained scenarios to estimate the number of pedestrians. It has wide applications in real-life situations. For example, during emergency evacuations of large crowds, accurately estimating crowd density and implementing appropriate security measures can effectively reduce or prevent stampedes. In the current COVID-19 pandemic, crowd counting provides technical support, enabling relevant departments to conduct their work more accurately, efficiently, and scientifically.
[0003] Traditional crowd counting methods include detection-based and regression-based approaches. With the development of deep learning, numerous deep learning methods based on density map estimation have emerged. Most early research utilized optical information obtained from RGB images to generate crowd density maps for crowd counting. However, due to variations in ambient lighting conditions and interference from pedestrian pose information, relying solely on RGB images as the information source for crowd counting tasks is unreliable. With the rapid development of thermal imagers and depth cameras, multimodal information can be acquired within the same scene, providing more information sources for crowd counting tasks. Therefore, recent research has begun to focus on cross-modal crowd counting methods.
[0004] Existing cross-modal crowd counting methods can be divided into RGB-D crowd counting methods and RGB-T crowd counting methods. RGB-D or RGB-T crowd counting methods utilize the complementary advantages of RGB and depth images, or RGB and thermal images, to improve the accuracy of the final counting result. Due to the relative scarcity of publicly available large-scale cross-modal crowd datasets, cross-modal crowd counting methods are also relatively few. Lian et al. combined depth information extracted from depth images with optical information extracted from RGB images in the head location detection process, naming it RDNet. Liu et al. proposed a cross-modal collaborative representation learning framework that combines thermal features and RGB features three times to generate an estimated density map. Existing methods only use CNNs as the backbone network to extract cross-modal features, but due to the limited receptive field of CNN convolutional kernels, they cannot extract global information from the image (global information plays an important role in crowd counting), resulting in inaccurate crowd counting methods. Summary of the Invention
[0005] To address the shortcomings of current cross-modal crowd counting methods that only use CNNs as the backbone network to extract cross-modal features and cannot extract global information from images, this invention proposes a cross-modal crowd counting method based on CNNs and transformers. By building a deep neural network model that combines CNNs and transformers, the complementary information of RGB images and thermal images can be fully utilized to improve counting accuracy.
[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:
[0007] A cross-modal crowd counting method based on CNN and transformer includes the following steps:
[0008] (1) Input the RGB image and the heat image into each branch of the dual-branch network structure composed of CNN to obtain the RGB feature map and the heat feature map respectively. Each branch of the dual-branch network structure includes 12 convolutional layers and 3 max pooling layers.
[0009] (2) Using the RGB feature map and heat map obtained in step (1) as input, the feature map is converted into a sequence vector through dimensional transformation. The sequence vector is then input into a novel cross-modal transformer to learn the cross-modal global features of different modal feature maps, resulting in RGB sequence vectors and heat sequence vectors with cross-modal global features extracted by the novel cross-modal transformer. The dimensions of the RGB feature map and the heat map are both C×H×W, where C, H, and W are the number of channels, height, and width, respectively. The shape of the sequence vector is K×E, where K and E are the sequence number and the length of the one-dimensional vector, respectively.
[0010] (3) Reshape the RGB sequence vector and heat sequence vector obtained in step (2) into cross-modal RGB feature map and cross-modal heat feature map respectively. Fuse the cross-modal RGB feature map with the RGB feature map extracted by the dual-branch structure composed of CNN, and fuse the cross-modal heat feature map with the heat feature map extracted by the dual-branch structure composed of CNN to obtain the fused RGB feature map and the fused heat feature map respectively.
[0011] (4) The RGB feature maps and heat maps after fusion of different layers of the network are connected through a cross-layer connection structure to obtain the output RGB feature map and the output heat map.
[0012] (5) Input the output RGB feature map and output thermal feature map of the cross-layer connection structure into the cross-modal attention module, and fuse their feature maps to obtain a new feature map;
[0013] (6) Input the new feature map into the tail network and obtain the estimated density map through a series of calculations. The tail network includes 4 convolutional layers.
[0014] (7) Add the density maps pixel by pixel to get the estimated number of people.
[0015] A further improvement of the technical solution of the present invention is that, in step (1), the number of feature map channels generated by each convolutional layer in each branch is 16, 16, 32, 32, 64, 64, 64, 64, 128, 128, 128, 128 in the order from input to output, and the pooling layer stride in each branch of the dual-branch network structure composed of CNN is 2.
[0016] A further improvement to the technical solution of the present invention is that the detailed process of converting the feature map into a sequence vector in step (2) is as follows:
[0017] If a feature map of height H and width W is divided into 4×4 image blocks, then all image blocks at the same position in C channels are sequentially unfolded into one-dimensional vectors and concatenated end-to-end to obtain a complete one-dimensional vector of length E = C × 4 × 4, and there are a total of We obtain two one-dimensional vectors of the same length, namely E and K.
[0018] A further improvement to the technical solution of the present invention is that the specific steps for obtaining the RGB sequence vector and the heat sequence vector with cross-modal global features in step (2) are as follows:
[0019] First, the RGB sequence vector and the popularity sequence vector are multiplied by their weight matrix to obtain the Value' matrix. Then, the RGB sequence vector and the popularity sequence vector are layer-normalized to maintain the stability of the data feature distribution and accelerate the model convergence. Finally, the layer-normalized RGB sequence vector and the popularity sequence vector are multiplied by their weight matrix to obtain the Query matrix.
[0020] Then, spatial pyramid downsampling is performed on the layer-normalized RGB sequence vector and heat sequence vector to further extract the semantic information of the sequence vector and heat sequence vector;
[0021] The RGB sequence vector and the heat sequence vector, which have been downsampled by the spatial pyramid, are then concatenated to fuse the features of the two modalities. These features are then multiplied with the corresponding weight matrices to obtain the Key matrix and Value matrix.
[0022] Finally, the Query matrix and Key matrix are multiplied together and then logistic regression is performed. The result is multiplied together with Value and then added to Value' matrix to obtain RGB sequence vectors and heat sequence vectors with cross-modal global features.
[0023] A further improvement to the technical solution of this invention is that the novel cross-modal transformer processing procedure is shown in the following formula:
[0024] Q = LN(E) RGB W Q (6),
[0025] K = Concat(DW(LN(E) RGB )),DW(LN(E Thermal )))W K (7),
[0026] V = Concat(DW(LN(E) RGB )),DW(LN(E Thermal )))W V (8),
[0027] V′=LN(E RGB W V' (9),
[0028]
[0029] Where LN(·) denotes layer normalization; Concat(·) denotes concatenation operation; W denotes the weight matrix; DW(·) denotes spatial pyramid downsampling operation; V' denotes the original sequence vector that retains more original modal information through residual concatenation; Q, K, and V denote the transformer's Query matrix, Key matrix, and Value matrix, respectively; E RGB Represents an RGB sequence vector; MHA(Q,K,V,V') represents the multi-head attention operation of a novel cross-modal transformer; d k This represents the scaling factor.
[0030] A further improvement to the technical solution of the present invention is that the detailed process of step (3) is as follows:
[0031] The RGB sequence vector with cross-modal global features extracted by the novel cross-modal transformer is upsampled to align its spatial dimension with that of the RGB feature map. Then, a 1×1 convolution operation is used to align E with C of the RGB feature map, resulting in the cross-modal RGB feature map. The cross-modal RGB feature map is then multiplied pixel-wise with the RGB feature map to obtain the fused RGB feature map.
[0032] Similarly, the heat sequence vector with cross-modal global features extracted by the novel cross-modal transformer is upsampled to align with the spatial dimension of the heat feature map, and then E is aligned with C of the heat feature map through a 1×1 convolution operation to obtain the cross-modal heat feature map. The cross-modal heat feature map and the heat feature map are multiplied at the pixel level to obtain the fused heat feature map;
[0033] A further improvement to the technical solution of the present invention is that the specific process of step (4) is as follows:
[0034] The fused feature maps obtained from the dual-branch CNN network and the novel cross-modal transformer are input into a cross-layer connection structure. The cross-layer connection structure adjusts the number of channels through convolution operations with a kernel of 1 and adjusts the spatial size of the map through max pooling operations, resulting in fused feature maps of each layer with the same number of channels and spatial size. The channel information of the fused RGB feature maps and the fused heatmaps of different layers of the network is enhanced by a branch attention module. The number of channels and spatial size of these fused feature maps of each layer are first adjusted, and then the channel information of the fused RGB feature maps and the fused heatmaps of different layers of the network is enhanced by a branch attention module. Finally, the feature maps with enhanced channel information are added pixel by pixel to obtain the output RGB feature map and the output heatmap.
[0035] A further improvement to the technical solution of this invention lies in that the cross-layer connection structure enhances the channel information of the RGB feature map and the fused heat feature map after fusion of different layers of the network through a branch attention module, specifically including:
[0036] First, the fused RGB feature map and the fused heat map are convolved through three convolutional layers: a 5×5 kernel convolutional layer, a 7×7 kernel convolutional layer with a dilation rate of 3, and a 1×1 kernel convolutional layer. The output of each convolutional layer is activated using the ReLU activation function. Then, it passes through a 1×1 kernel convolutional layer and is activated using the Sigmoid activation function. Finally, the fused RGB feature map or heat map with enhanced overchannel information is obtained.
[0037] The branch attention module processing procedure is shown in the following formula:
[0038]
[0039]
[0040]
[0041] in, Represents long-range information of the feature map; D-Conv7,3(·) represents a dilated convolution operation with a 7×7 kernel and a dilation rate of 3; Conv5(·) represents a convolution operation with a 5×5 kernel; F f i The fused feature map is represented by sigm(·); Sigm(·) represents the sigmoid function; Conv1(·) represents a convolution operation with a 1×1 kernel; Relu(·) represents the ReLU activation function; W represents the weight matrix; and Output represents the fused and enhanced feature map.
[0042] A further improvement to the technical solution of the present invention is that step (5) specifically includes:
[0043] First, the output thermal feature maps of the cross-layer connection structure are subjected to max pooling with strides of 2 and 4, respectively, and then concatenated. This concatenation is then passed through a 1×1 convolutional layer to obtain thermal feature maps with spatial context information. Next, these thermal feature maps with spatial context information are subjected to channel-dimensional average pooling and max pooling, respectively. The outputs of the two pooling operations are concatenated, passed through a 5×5 convolutional layer, and activated using a sigmoid activation function to obtain learnable weight parameters for enhancing the RGB feature maps. Then, the output RGB feature maps of the cross-layer connection structure are subjected to channel-dimensional average pooling and max pooling, and the outputs of the two pooling operations are concatenated. This concatenation is then passed through a 5×5 convolutional layer and activated using a sigmoid activation function. The activation function is used to activate the feature map and obtain learnable weight parameters for enhancing the heat map. Finally, the heat map with spatial context information is multiplied with the learnable weight parameters for enhancing the heat map, and the output RGB feature map of the cross-layer connection structure is multiplied with the learnable weight parameters for enhancing the RGB feature map. The two multiplication results are then added pixel by pixel to obtain a new feature map.
[0044] A further improvement to the technical solution of the present invention is that the cross-modal attention module processing procedure is shown in the following formula:
[0045]
[0046]
[0047]
[0048]
[0049] Where Sigm(·) represents the sigmoid function; Conv5(·) represents a convolution operation with a 5×5 kernel; Conv1(·) represents a convolution operation with a 1×1 kernel; max_pool(·) represents max pooling along the channel dimension; avg_pool(·) represents average pooling along the channel dimension; and Concat(·) represents concatenation operation. Represents contextual information about the heatmap; Represents the weight information of the RGB image; This represents the weight information of the heatmap; max_pool2(·) represents max pooling with a kernel of 2×2 and a stride of 2; max_pool4(·) represents max pooling with a kernel of 4×4 and a stride of 4. This indicates that two images are multiplied pixel by pixel. This indicates that two images are added pixel by pixel.
[0050] The technological advancements achieved by this invention due to the adoption of the above technical solutions are as follows:
[0051] (1) A cross-layer connection structure is proposed to connect the feature maps fused at the front end of the network and the feature maps fused at the back end of the network, and the channel information of the fused feature maps is enhanced by a branch attention module.
[0052] (2) A cross-modal attention module is proposed, which can fuse complementary information from RGB images and thermal images to enhance cross-modal feature representation.
[0053] (3) A network framework combining CNN and novel cross-modal transformer is proposed to fully learn the complementary features of cross-modal images, thereby modeling the distribution of crowds in complex scenes and generating a density map that reflects the real distribution of crowds, and performing more accurate crowd counting. Attached Figure Description
[0054] Figure 1 This is a flowchart illustrating the population counting method proposed in this invention;
[0055] Figure 2 This is a schematic diagram of the overall structure of the deep neural network model of the present invention;
[0056] Figure 3 This is a detailed schematic diagram of the novel transformer of this invention;
[0057] Figure 4 This is a detailed schematic diagram of the branch attention module of the present invention;
[0058] Figure 5 This is a detailed schematic diagram of the cross-modal attention module of the present invention;
[0059] Figure 6 This is a schematic diagram illustrating a specific embodiment of the present invention. Detailed Implementation
[0060] The present invention will be further described in detail below with reference to embodiments:
[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0062] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the following description is provided in conjunction with... Figure 1 The present invention will be further described in detail below with reference to specific embodiments.
[0063] (1) The RGB image and the heat map image are respectively input into the RGB feature extraction network branch and the heat map feature extraction network branch of the dual-branch network structure composed of CNN, to obtain the RGB feature map and the heat map feature map respectively. The RGB feature extraction network branch and the heat map feature extraction network branch of the dual-branch network composed of CNN have the same structure, both including 12 convolutional layers and 3 max pooling layers. The specific structure of each network branch is: 4 convolutional layers, 1 max pooling layer, 4 convolutional layers, 1 max pooling layer, 4 convolutional layers, 1 max pooling layer. The convolutional kernel size of the convolutional layers is 3×3, and the number of channels of the generated feature maps are 16, 16, 32, 32, 64, 64, 64, 64, 128, 128, 128, 128, and the stride of the 3 max pooling layers is 2.
[0064] (2) Using the RGB feature maps and heatmaps extracted by the dual-branch CNN structure as input, the RGB feature maps and heatmaps (both with dimensions C×H×W, where C, H, and W are the number of channels, height, and width, respectively) are first transformed into RGB sequence vectors and heatmap sequence vectors (the shape of the sequence vector is K×E, where K and E are the sequence number and the length of the one-dimensional vector, respectively). Taking the RGB feature map as an example, the specific transformation process is as follows: the feature map with height H and width W is divided into 4×4 image blocks. Then, all image blocks at the same position of C channels are unfolded into one-dimensional vectors in sequence and concatenated end to end to obtain a complete one-dimensional vector with a length of E = C×4×4, which has a total of We obtain E and K of the RGB sequence vectors from two one-dimensional vectors of the same length. Similarly, we can transform the heatmap into a heat sequence vector. Then, we input the sequence vector into a novel cross-modal transformer to learn the cross-modal global features of different modal feature maps, and obtain the RGB sequence vector and heat sequence vector with cross-modal global features extracted by the transformer.
[0065] The following is combined Figure 3 The novel cross-modal transformer used in this invention will be described as follows:
[0066] like Figure 3 As shown in the detailed diagram of the novel cross-modal transformer, we aim to obtain RGB sequence vectors and heat sequence vectors with cross-modal global features extracted by the transformer (taking the RGB vector with cross-modal global features as an example, the process for obtaining the heat sequence vector with cross-modal global features is similar). First, the RGB sequence vector is multiplied by its weight matrix to obtain the Value' matrix. The RGB sequence vector and heat sequence vector are then subjected to layer normalization, which maintains the stability of the data feature distribution and accelerates the model convergence. The layer-normalized RGB sequence vector is multiplied by its weight matrix to obtain the Query matrix. Then, spatial pyramid downsampling is performed on the layer-normalized RGB sequence vector to further extract the semantic information of the sequence vector. Next, the RGB sequence vector and heat sequence vector after spatial pyramid downsampling are concatenated to fuse the features of the two modalities. This is then multiplied by the corresponding weight matrices to obtain the Key matrix and Value matrix. Finally, the Query matrix and Key matrix are multiplied together and logistic regression is performed. The result is then multiplied by the Value matrix and added to the Value' matrix to finally obtain the RGB sequence vector with cross-modal global features.
[0067] (3) The RGB sequence vector with cross-modal global features extracted by the novel cross-modal transformer is upsampled to align with the spatial dimension of the RGB feature map. Then, a 1×1 convolution operation is used to align E with C of the RGB feature map, reshaping it into a cross-modal RGB feature map. Similarly, the cross-modal heat map can be obtained. The cross-modal RGB feature map is then multiplied pixel-wise with the RGB feature map extracted by the dual-branch structure composed of CNN to obtain the fused RGB feature map. Similarly, the fused heat map can be obtained.
[0068] (4) The fused feature maps obtained from the dual-branch network composed of CNN and the novel cross-modal transformer are adjusted by convolution operation with kernel 1, and the spatial size of the fused feature maps is adjusted by max pooling operation to obtain fused feature maps of each layer with the same number of channels and the same spatial size. The channel information of the fused RGB feature maps and the fused heat feature maps of different layers of the network is enhanced by the branch attention module. The number of channels and spatial size of these fused feature maps of each layer are adjusted first, and then the channel information of the fused RGB feature maps and the fused heat feature maps of different layers of the network is enhanced by the branch attention module. Finally, the feature maps with enhanced channel information are added pixel by pixel to obtain the output RGB feature map and the output heat feature map.
[0069] The following is combined Figure 4 The branch attention enhancement module used in this invention will be described as follows:
[0070] like Figure 4 As shown in the detailed diagram of the branch attention enhancement module, we want to enhance the channel information of the RGB feature map and the fused heat map after different layers of the network are fused. First, the fused feature map (i.e., the fused RGB feature map or the fused heat map) is convolved through three convolutional layers. These three convolutional layers are: a 5×5 kernel convolutional layer, a 7×7 kernel convolutional layer with a dilation rate of 3, and a 1×1 kernel convolutional layer. The output of the convolutional layer is activated by the ReLU activation function. Then, it goes through a 1×1 kernel convolutional layer and is activated by the Sigmoid activation function. Finally, we obtain the fused RGB feature map or heat map with enhanced channel information.
[0071] (5) Input the output RGB feature map and output heat feature map of the cross-layer connection structure into the cross-modal attention module, and fuse their feature maps to obtain a new feature map.
[0072] The following is combined Figure 5 The cross-modal attention module used in this invention will be described as follows:
[0073] like Figure 5As shown in the detailed diagram of the cross-modal attention module, we aim to fuse RGB feature maps and heatmaps to obtain a new feature map. First, the output heatmaps from the cross-layer connection structure are subjected to max pooling with strides of 2 and 4, respectively, and then concatenated. This concatenation is then passed through a 1×1 convolutional layer to obtain a heatmap with spatial context information. Next, this heatmap with spatial context information is subjected to channel-dimensional average pooling and max pooling, respectively. The outputs of the two pooling operations are concatenated, passed through a 5×5 convolutional layer, and activated using the Sigmoid activation function to obtain the feature map used to enhance the RGB features. The process involves several steps: First, the learnable weight parameters for the feature map are obtained. Then, the output RGB feature map of the cross-layer connection structure is processed through channel-dimensional average pooling and max pooling. The outputs of the two pooling operations are concatenated, passed through a 5×5 convolutional layer, and activated using the Sigmoid activation function to obtain the learnable weight parameters for enhancing the heat map. Finally, the heat map with spatial context information is multiplied by the learnable weight parameters for enhancing the heat map, and the output RGB feature map of the cross-layer connection structure is multiplied by the learnable weight parameters for enhancing the RGB feature map. The two multiplication results are then added pixel by pixel to obtain a new feature map.
[0074] (6) Input the new feature map into the tail network. The tail network consists of 4 convolutional layers with a kernel size of 3×3. The number of channels in the generated feature maps are 126, 64, 32 and 1 respectively. The estimated density map is obtained through a series of calculations.
[0075] (7) Add the density maps pixel by pixel to get the estimated number of people.
[0076] The following describes specific embodiments of the present invention and... Figure 4 The present invention will be further explained below.
[0077] Step 1: Establish a deep neural network model for crowd density estimation, with the overall structure as follows: Figure 2 As shown, Figure 2 This is a schematic diagram of the overall structure of the deep neural network model of this invention. The process of establishing the deep neural network model is as follows:
[0078] 1.1) Establish a two-branch CNN network. The two branches accept the input RGB image and the heatmap image respectively. After feature extraction, RGB feature maps and heatmap feature maps are obtained. This stage includes 12 convolutions and 3 max pooling operations. For example... Figure 2As shown, the sequence is 4 convolutions, 1 pooling, 4 convolutions, 1 pooling, 4 convolutions, 1 pooling, 4 convolutions, 1 pooling. The size of each convolutional layer is 3×3. The number of channels in the generated channel maps are 16, 16, 32, 32, 64, 64, 64, 64, 128, 128, 128, 128 respectively. The stride of the pooling layer is 2. The RGB feature map and the heat map after each pooling are input into the novel cross-modal transformer in section 1.2) and the cross-layer connection structure in section 1.3) respectively.
[0079] 1.2) Establish a novel cross-modal transformer that receives the RGB feature map and heat map extracted by the dual-branch network composed of CNNs in 1.1). The processing procedure in this stage is shown in the following formula:
[0080] Q = LN(E) RGB W Q (13)
[0081] K = Concat(DW(LN(E) RGB )),DW(LN(E Thermal )))W K (14)
[0082] V = Concat(DW(LN(E) RGB )),DW(LN(E Thermal )))W V (15)
[0083] V′=LN(E RGB W V' (16)
[0084]
[0085] The detailed processing procedure for this stage is as follows:
[0086] First, the RGB sequence vector is multiplied by its weight matrix to obtain the Value' matrix. The RGB sequence vector and the popularity sequence vector are then subjected to layer normalization. The layer-normalized RGB sequence vector is multiplied by its weight matrix to obtain the Query matrix. Next, spatial pyramid downsampling is performed on the layer-normalized RGB sequence vector. Then, the RGB sequence vector and the popularity sequence vector after spatial pyramid downsampling are concatenated to fuse the features of the two modalities. This concatenation is then multiplied by the corresponding weight matrices to obtain the Key matrix and the Value matrix. Finally, the Query matrix and the Key matrix are multiplied together and logistic regression is performed. The result is then multiplied by the Value matrix and added to the Value' matrix to obtain the RGB sequence vector with cross-modal global features, which is then input into the cross-layer connection structure in section 1.3). Similarly, the popularity sequence vector with cross-modal global features is input into the cross-layer connection structure in section 1.3.
[0087] 1.3) Establish a cross-layer connection structure, receiving the RGB feature maps and heatmaps extracted by the dual-branch network composed of CNNs in 1.1), and the RGB sequence vectors of cross-modal global features extracted by the novel cross-modal transformer in 1.2), and the heatmap sequence vectors with cross-modal global features. The detailed processing of this stage is as follows:
[0088] The fused feature maps obtained from the dual-branch network composed of CNNs and the novel cross-modal transformer are adjusted in terms of channel number through convolution operations with kernel number 1, and the spatial size of the fused feature maps is adjusted through max pooling operations to obtain fused feature maps of each layer with the same number of channels and spatial size. The channel information of the fused RGB feature maps and the fused heat feature maps of different layers of the network is enhanced by the branch attention module. The channel number and spatial size of these fused feature maps of each layer are first adjusted, and then the channel information of the fused RGB feature maps and the fused heat feature maps of different layers of the network are enhanced by the branch attention module. Finally, the feature maps with enhanced channel information are added pixel by pixel to obtain the output RGB feature map and the output heat feature map. The output RGB feature map and the output heat feature map are input into the cross-modal attention module (1.4).
[0089] The processing procedure of the branch attention module is shown in the following formula:
[0090]
[0091]
[0092]
[0093] The detailed processing procedure of the branch attention module is shown below:
[0094] First, the fused feature map (i.e., the fused RGB feature map or the fused heat map) is convolved through three convolutional layers: a 5×5 kernel convolutional layer, a 7×7 kernel convolutional layer with a dilation rate of 3, and a 1×1 kernel convolutional layer. The output of the convolutional layers is activated by the ReLU activation function. Then, it passes through a 1×1 kernel convolutional layer and is activated by the Sigmoid activation function. Finally, the fused RGB feature map or heat map with enhanced overchannel information is obtained.
[0095] 1.4) Establish a cross-modal attention module to receive the output RGB feature map and output heat map from the cross-layer connection structure in 1.3). The processing procedure in this stage is shown in the following formula:
[0096]
[0097]
[0098]
[0099]
[0100] The detailed processing procedure for this stage is as follows:
[0101] First, the output thermal feature maps of the cross-layer connection structure are subjected to max pooling with strides of 2 and 4, respectively, and then concatenated. This concatenation is then passed through a 1×1 convolutional layer to obtain thermal feature maps with spatial context information. Next, these thermal feature maps with spatial context information are subjected to channel-dimensional average pooling and max pooling, respectively. The outputs of the two pooling operations are concatenated, passed through a 5×5 convolutional layer, and activated using a sigmoid activation function to obtain learnable weight parameters for enhancing the RGB feature maps. Then, the output RGB feature maps of the cross-layer connection structure are subjected to channel-dimensional average pooling and max pooling, and the outputs of the two pooling operations are concatenated. This concatenation is then passed through a 5×5 convolutional layer and activated using a sigmoid activation function. The activation function is used to activate the feature map and obtain the learnable weight parameters for enhancing the heat map. Finally, the heat map with spatial context information is multiplied with the learnable weight parameters for enhancing the heat map, and the output RGB feature map of the cross-layer connection structure is multiplied with the learnable weight parameters for enhancing the RGB feature map. The two multiplication results are added pixel by pixel to obtain a new feature map, which is then input into the tail network established in 1.5).
[0102] 1.5) Establish the tail network to receive the new feature map output by the cross-modal attention module in 1.4). This stage includes 4 convolutional layers with a kernel size of 3×3. The number of channels in the generated feature maps are 126, 64, 32, and 1, respectively. The estimated density map is obtained after 4 convolution operations.
[0103] Step 2: Input the RGB image and heat map of the target image into the deep neural network dual branch established in Step 1 to obtain the density map.
[0104] Step 3: Add the density maps obtained in Step 2 pixel by pixel to obtain the estimated number of people.
[0105] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0106] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of these embodiments are merely for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.
Claims
1. A cross-modal crowd counting method based on CNN and transformer, characterized in that , comprising the following steps: (1) inputting the RGB image and the heat image into an RGB feature extraction network branch and a heat feature extraction network branch in a double-branch network structure composed of CNN respectively to obtain an RGB feature map and a heat feature map, each branch in the double-branch network structure comprising 12 convolutional layers and 3 maximum pooling layers; (2) taking the RGB feature map and the heat feature map obtained in step (1) as input, converting the above feature maps into sequence vectors through dimension transformation, inputting the sequence vectors into a cross-modal transformer to learn cross-modal global features of different modal feature maps, and obtaining an RGB sequence vector and a heat sequence vector with cross-modal global features extracted by the cross-modal transformer; the dimensions of the RGB feature map and the heat feature map are both CxHxW, wherein C, H and W are respectively the number of channels, height and width; the shape of the sequence vector is KxE, wherein K and E are respectively the number of sequence vectors and the length of one-dimensional vectors; the detailed process of converting the feature map into the sequence vector in step (2) is as follows: If a feature map of height H and width W is divided into 4×4 image blocks, then all image blocks at the same position in C channels are sequentially unfolded into one-dimensional vectors and concatenated end-to-end to obtain a complete one-dimensional vector of length E = C × 4 × 4, and there are a total of We obtain two one-dimensional vectors of the same length, namely E and K; the specific steps in step (2) to obtain the RGB sequence vector and the heat sequence vector with cross-modal global features are as follows: First, multiply the RGB sequence vector and the heat sequence vector by their weight matrices to obtain a Value' matrix, perform layer normalization on the RGB sequence vector and the heat sequence vector to maintain the stability of the data feature distribution and accelerate the model convergence effect, multiply the layer-normalized RGB sequence vector and the heat sequence vector by their weight matrices to obtain a Query matrix; Then, perform spatial pyramid down-sampling operation on the layer-normalized RGB sequence vector and the heat sequence vector to further extract the semantic information of the sequence vector and the heat sequence vector; Then, the RGB sequence vector and the heat sequence vector subjected to spatial pyramid down-sampling are spliced to fuse the features of the two modalities, and multiplied by the corresponding two weight matrices to obtain a Key matrix and a Value matrix; Finally, perform matrix multiplication on the Query matrix and the Key matrix, and perform logistic regression, multiply the result by the Value matrix, and finally obtain the RGB sequence vector and the heat sequence vector with cross-modal global features; the cross-modal transformer processing process is shown in the following formula: Q RGB = LN(E RGB )W Q (1), Q Thermal = LN(E Thermal )W Q K = Concat(DW(LN(E RGB )), DW(LN(E Thermal )))W K (2), V = Concat(DW(LN(E RGB )), DW(LN(E Thermal )))W V (3), V' RGB = E RGB W V' (4), V' Thermal = E Thermal W V' where LN(·) denotes layer normalization; Concat(·) denotes concatenation operation; W denotes weight matrix; DW(·) denotes spatial pyramid down-sampling operation; V' denotes more original modality information reserved by residual connection for the original sequence vector; Q, K, V denote Query matrix, Key matrix and Value matrix of the transformer respectively; E RGB denotes RGB sequence vector; MHA(Q, K, V, V') denotes multi-head attention operation of the cross-modal transformer; d k denotes scaling factor; (3) remolding the RGB sequence vector and the heat sequence vector obtained in step (2) into a cross-modal RGB feature map and a cross-modal heat feature map, fusing the cross-modal RGB feature map with the RGB feature map extracted by the double-branch structure composed of CNN, and fusing the cross-modal heat feature map with the heat feature map extracted by the double-branch structure composed of CNN, to obtain a fused RGB feature map and a fused heat feature map respectively; the detailed process of step (3) is as follows: The RGB sequence vector with cross-modal global features extracted by the cross-modal transformer is aligned with the spatial dimension of the RGB feature map through upsampling operation, and then the E is aligned with the C of the RGB feature map through 1*1 convolution operation, to obtain a cross-modal RGB feature map, and the cross-modal RGB feature map is pixel-level multiplied with the RGB feature map to obtain a fused RGB feature map; Similarly, the heat sequence vector with cross-modal global features extracted by the cross-modal transformer is aligned with the spatial dimension of the heat feature map through upsampling operation, and then the E is aligned with the C of the heat feature map through 1*1 convolution operation, to obtain a cross-modal heat feature map, and the cross-modal heat feature map is pixel-level multiplied with the heat feature map to obtain a fused RGB feature map; (4) The fused RGB feature map and the fused heat feature map are connected through the cross-layer connection structure to obtain an output RGB feature map and an output heat feature map; (5) The output RGB feature map and the output heat feature map of the cross-layer connection structure are input into the cross-modal attention module to fuse the feature maps of each other and obtain a new feature map; (6) The new feature map is input into the tail network to obtain an estimated density map through a series of calculations, and the tail network includes four convolution layers; (7) The obtained density map is added pixel by pixel to obtain an estimated number of people.
2. The CNN and transformer based cross-modal crowd counting method according to claim 1, wherein, The channel number of the feature map generated by each convolution layer in each branch in step (1) is 16, 16, 32, 32, 64, 64, 64, 64, 128, 128, 128, 128 from input to output direction, and the step of the pooling layer in each branch of the double-branch network structure composed of CNN is 2.
3. The CNN and transformer based cross-modal crowd counting method according to claim 1, wherein, The specific process of step (4) is: The fused feature map obtained from the double-branch network composed of CNN and the cross-modal transformer is input into the cross-layer connection structure, the cross-layer connection structure adjusts the channel number through 1 convolution operation and adjusts the spatial size of the map through maximum pooling operation to obtain each layer of fused feature map with the same channel number and spatial size, and the branch attention module enhances the channel information of the fused RGB feature map and the fused heat feature map of different layers of the network, adjusts the channel number and spatial size of these each layer of fused feature map first, then the branch attention module enhances the channel information of the fused RGB feature map and the fused heat feature map of different layers of the network, and finally the feature map with enhanced channel information is pixel-level added to obtain an output RGB feature map and an output heat feature map.
4. The CNN and transformer based cross-modal crowd counting method according to claim 3, wherein, The branch attention module in the cross-layer connection structure enhances the channel information of the fused RGB feature map and the fused heat feature map of different layers of the network, which specifically includes: Firstly, the fused RGB feature map and the fused heat feature map are convolved through three convolutional layers, the three convolutional layers are respectively: a convolutional layer with a convolution kernel of 5*5, a dilated convolutional layer with a convolution kernel of 7*7 and a dilated rate of 3, and a convolutional layer with a convolution kernel of 1*1, and the output of the convolutional layer is activated through a ReLU activation function; then a convolutional layer with a convolution kernel of 1*1 is passed through a Sigmoid activation function; finally, the RGB feature map or the heat feature map after fusion and enhancement of channel information is obtained; The branch attention module processing process is as shown in the following formula: wherein, represents long-distance information of the feature map; D-Conv 7，3 (·) represents a convolution kernel of 7x7 and a dilated convolution operation with a dilated rate of 3; Conv5(·) represents a convolution kernel of 5x5; F f i represents a fused feature map; Sigm(·) represents a sigmoid function; Conv1(·) represents a convolution kernel of 1x1; Relu(·) represents a ReLu activation function; W represents a weight matrix; and Output represents a feature map after fusion and enhancement.
5. The CNN and transformer based cross-modal crowd counting method according to claim 1, wherein, The step (5) specifically comprises: Firstly, the output heat feature map of the cross-layer connection structure is respectively subjected to maximum pooling operation with a step of 2 and 4, spliced, subjected to a convolutional layer with a convolution kernel of 1*1 to obtain a heat feature map with spatial context information, and then the heat feature map with spatial context information is respectively subjected to channel dimension average pooling and maximum pooling, the output results of the two poolings are spliced, subjected to a convolutional layer with a convolution kernel of 5*5, and activated through a Sigmoid activation function to obtain a learnable weight parameter for enhancing the RGB feature map; then, the output RGB feature map of the cross-layer connection structure is subjected to channel dimension average pooling and maximum pooling, the output results of the two poolings are spliced, subjected to a convolutional layer with a convolution kernel of 5*5, and activated through a Sigmoid activation function to obtain a learnable weight parameter for enhancing the heat feature map; finally, the heat feature map with spatial context information and the learnable weight parameter for enhancing the heat feature map are multiplied, the output RGB feature map of the cross-layer connection structure and the learnable weight parameter for enhancing the RGB feature map are multiplied, and the two multiplication results are added pixel by pixel to obtain a new feature map.
6. The CNN and transformer based cross-modal crowd counting method according to claim 5, wherein, The cross-modal attention module processing process is as shown in the following formula: wherein Sigm(·) represents a sigmoid function; Conv5(·) represents a convolution operation with a convolution kernel of 5x5; Conv1(·) represents a convolution operation with a convolution kernel of 1x1; max_pool(·) represents a maximum pooling in a channel dimension; avg_pool(·) represents an average pooling in a channel dimension; Concat(·) represents a concatenation operation; represents context information of a heat map image; represents weight information of an RGB image; represents weight information of a heat map; max_pool2(·) represents a maximum pooling with a kernel of 2x2 and a step of 2; max_pool4(·) represents a maximum pooling with a kernel of 4x4 and a step of 4; represents pixel-by-pixel multiplication of two images; represents pixel-by-pixel addition of two images.