RGB-T crowd counting method based on cross-modal and multi-scale perception
By using Swin Transformer to extract features in RGB-T population counting and combining cross-modal feature fusion and multi-scale feature aggregation modules, the problem of insufficient noise suppression and spatial distribution information is solved, and the counting accuracy and robustness are improved.
Patent Information
- Application Number
- CN202411031906.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-08
AI Technical Summary
The existing RGB-T population counting method ignores the suppression and removal of noise information during the cross-modal feature fusion process, and does not fully utilize the population spatial distribution information, resulting in limited counting accuracy.
The dual-stream network based on Swin Transformer extracts RGB and thermal image features, and the cross-modal feature fusion module realizes feature sharing and noise removal between each mode, and uses the multi-scale feature aggregation module to model through the spatial context-aware graph convolution unit. Crowd space structure, integrating feature maps at different scales.
Effectively remove noise data and capture the spatial distribution information of the crowd, improving the counting accuracy and robustness of RGB-T crowd counting, especially in scenarios where the crowd distribution is uneven.
Smart Images

Figure CN118918531A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to an RGB-T crowd counting method based on cross-modal and multi-scale perception. Background Art
[0002] In today's rapidly developing urban environment, the increasing population density makes crowd counting technology play an increasingly critical role in maintaining public safety, optimizing urban planning and improving management efficiency. Traditional crowd counting techniques, especially those that use a single processing stream to process static RGB images, often show significant limitations when faced with environments with poor lighting conditions.
[0003] To overcome these challenges, researchers have begun to explore cross-modal crowd counting methods that enhance the system's scene adaptability and counting accuracy by integrating data from different sensors or modalities. RGB-T crowd counting combines RGB images with thermal imaging data. RGB images provide color visual information of the scene, while thermal imaging cameras capture the thermal radiation distribution of the scene to generate thermal images. Thermal images reflect the heat distribution of objects in the scene, so they are very effective in distinguishing between human bodies and other objects, especially in low-light environments. RGB-T crowd counting can improve the performance of crowd counting in complex lighting conditions and environments by fusing image data from these two different modalities, especially in dark, foggy or smoky environments. Although RGB-T crowd counting methods have shown great potential in improving the accuracy and robustness of crowd counting. However, there are several problems in the field of RGB-T crowd counting that need to be solved: First, existing methods ignore the suppression and removal of noise information in the process of cross-modal feature fusion. Second, the spatial distribution information of the crowd is not fully utilized. The above problems significantly affect the counting accuracy of RGB-T crowd counting methods. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide an RGB-T crowd counting method based on cross-modal and multi-scale perception, which can effectively share features between different modalities and remove noise data, while capturing the spatial distribution information of the crowd to cope with the challenge of uneven crowd distribution in RGB-T crowd counting.
[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0006] An RGB-T crowd counting method based on cross-modal and multi-scale perception includes the following steps:
[0007] Step 1: Input the paired RGB and thermal images into a two-stream network consisting of Swin Transformer to extract the specific feature maps of each modality;
[0008] Step 2: Input the specific feature maps of RGB mode and heat mode extracted by the last three stages of the two-stream network composed of Swin Transformer into the cross-modal feature fusion module respectively, so as to achieve effective denoising while maintaining the shared features between each modality, and obtain the cross-modal fusion feature map output by the cross-modal feature fusion module;
[0009] Step 3: The multi-scale feature aggregation module receives the cross-modal fusion feature map from the cross-modal feature fusion module, models the crowd spatial structure of each scale through the spatial context-aware graph convolution unit, and then integrates the feature maps of different scales;
[0010] Step 4: Send the multi-scale fusion feature map output by the multi-scale feature aggregation module to the regression head. After a series of processing, the predicted density map is obtained by regression.
[0011] Step 5: Add the obtained predicted density map pixel by pixel to get the predicted total number of people.
[0012] A further improvement of the technical solution of the present invention is that in step 1, the paired RGB images and thermal images are respectively input into each branch of the two-stream network composed of Swin Transformer, and specific feature maps of the RGB mode and thermal mode of each stage are extracted. The two-stream network composed of Swin Transformer is divided into 4 stages.
[0013] A further improvement of the technical solution of the present invention is that in step 2, the specific process is as follows:
[0014] S2.1, the specific feature maps of RGB modality and heat modality extracted by the last three stages of the two-stream network composed of Swin Transformer are processed by adaptive maximum pooling layer and fully connected layer respectively, and the most critical and noise-suppressed information is extracted from the feature map of each modality to obtain the denoised feature maps of RGB modality and heat modality;
[0015] S2.2, the denoised feature maps of the RGB modality and the heat modality are added together and then subjected to Dropout processing, followed by a full connection operation to obtain a cross-modal shared feature map;
[0016] S2.3, the cross-modal shared feature map is processed by the Sigmoid activation function and converted into a cross-modal shared weight map;
[0017] S2.4, multiplying the cross-modal shared weight map by the original input feature maps of the RGB modality and the heat modality element by element to obtain a cross-modal RGB feature map and a cross-modal heat feature map;
[0018] S2.5, further filtering the cross-modal RGB feature map and the cross-modal heat feature map through the residual structure to remove any residual noise, thereby obtaining a denoised cross-modal RGB feature map and a denoised cross-modal heat feature map;
[0019] S2.6, adding the denoised cross-modal RGB feature map and the denoised cross-modal heat feature map to obtain a cross-modal fusion feature map.
[0020] A further improvement of the technical solution of the present invention is that in step 2, the fusion process of the cross-modal feature fusion module on the specific feature graphs of the RGB modality and the heat modality is expressed as:
[0021]
[0022] F F,i =E F,i ·F RGB,i +F RGB,i +E F,i ·F T,i +F T,i (4)
[0023] Where i=2, 3, and 4 are cross-modal feature fusion modules connected by the last three stages of the two-stream network composed of Swin Transformer; AMP() represents adaptive maximum pooling; and
[0024] They respectively represent the full connection of single modality features and the full connection of cross-modal fusion features; ReLU() represents the rectified linear unit; Drop() represents dropout; Sig() represents the sigmoid activation function.
[0025] A further improvement of the technical solution of the present invention is that in step 3, the specific process is as follows:
[0026] S3.1, the cross-modal fusion feature maps of each scale are upsampled to the same size, and then the number of channels of the feature maps of each scale are aligned using 3×3 convolution to obtain the aligned feature maps of each scale;
[0027] S3.2, the aligned feature maps of each scale are subjected to batch normalization respectively, and then the ReLU activation function is applied to obtain the feature maps to be modeled at each scale;
[0028] S3.3, inputting the feature graphs to be modeled at each scale into the spatial context-aware graph convolution unit, modeling the crowd spatial structure at different scales, and obtaining the modeling feature graphs at each scale;
[0029] S3.4, add the modeling feature maps of each scale to obtain a multi-scale fusion feature map.
[0030] A further improvement of the technical solution of the present invention is that in step 3.3, the specific process is as follows:
[0031] S3.3.1, subject the feature map to be modeled to 1×1 convolution processing to compress the number of channels, and then take the average in the height dimension to obtain a reduced-dimensional feature map;
[0032] S3.3.2, perform dimension expansion processing on the reduced-dimensional feature map, respectively, perform dimension expansion after the last dimension and before the last dimension, and obtain two increased-dimensional feature maps;
[0033] S3.3.3, performing a bilateral difference operation on the two dimension-enhanced feature maps to obtain a differential feature map;
[0034] S3.3.4, the differential feature map is processed by the Tanh activation function, and then the number of channels is increased by 1×1 convolution to obtain the modeling weight map;
[0035] S3.3.5, the modeling weight map and the original input feature map to be modeled are processed by Einstein sum operation to obtain a modeling feature map containing rich population structure information.
[0036] A further improvement of the technical solution of the present invention is that in step 3.3, the modeling process of the spatial context-aware graph convolution unit is expressed as:
[0037]
[0038] In the formula, Represents a convolution with a kernel size of 1×1; Avg H (·) means taking the average in the height dimension; ε1(·) and ε2(·) mean dimensional expansion after the last dimension and before the last dimension, respectively; Represents the Tanh activation function; represents the Einstein summation operation; || represents the concatenation operation; represents the feature map to be modeled for the jth channel; x i:,j Represents the modeling weight map of the j-th channel.
[0039] A further improvement of the technical solution of the present invention is that in step 3.4, the fusion process of each scale feature map by the multi-scale feature aggregation module is expressed as:
[0040]
[0041] Y i =SCGC(X i ) (9)
[0042]
[0043] In the formula, UP j (·) indicates upsampling by j times, where j = 2, 4, 8; represents a convolution with a kernel size of 3×3; SCGC(·) represents a spatial context-aware graph convolution unit.
[0044] A further improvement of the technical solution of the present invention is that in step 4, the specific process is as follows:
[0045] S4.1, downsample the multi-scale fusion feature map through 3×3 convolution to obtain the feature map to be regressed;
[0046] S4.2, the feature map to be regressed is processed by three parallel-connected dilated convolutions, with dilation rates of 1, 2, and 3 respectively. After each dilated convolution, it is processed by batch normalization and ReLU activation function to obtain a dilated feature map. Subsequently, the three dilated feature maps are connected to obtain a multi-receptive field feature map;
[0047] S4.3, the feature map to be regressed is processed by 1×1 convolution, and then processed by batch normalization and ReLU activation function. According to the idea of residual network, residual connection is performed with the dilated feature map to obtain the residual feature map;
[0048] S4.4, the residual feature map is processed by 3×3 convolution, batch normalization, ReLU activation function, 1×1 convolution, and ReLU activation function to obtain the predicted density map.
[0049] A further improvement of the technical solution of the present invention is that in step 4, the regression process of the regression head is expressed as:
[0050]
[0051] Z i =ReLU(BN(DConv i (Z))) (12)
[0052]
[0053] Where i = 1, 2, 3; DConv i (·) denotes a dilated convolution with a dilation rate of i; Cat(·) denotes a concatenation operation.
[0054] Due to the adoption of the above technical solution, the technical progress achieved by the present invention is:
[0055] 1. The present invention proposes a novel RGB-T crowd counting method based on cross-modal and multi-scale perception, which can effectively solve the counting problems caused by the introduction of noise information in the cross-modal feature fusion process and the uneven distribution of the crowd.
[0056] 2. The present invention proposes a novel cross-modal feature fusion module, which achieves the effect of eliminating redundant and noisy data while sharing features among various modalities, thereby effectively solving the problem of introducing noise information in the process of cross-modal feature fusion.
[0057] 3. The present invention proposes a novel multi-scale feature aggregation module, which uses a novel spatial context-aware graph convolution unit to model crowd structures at various scales. The spatial context-aware graph convolution unit captures the contextual information of a local group in the entire crowd, including the spatial position of the local group in the entire crowd and the correlation between the density of the local group and other local groups. On the basis of the spatial context-aware graph convolution unit modeling crowd structures at various scales, feature graphs of different scales are integrated, thereby effectively alleviating the counting problem caused by uneven crowd distribution. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.
[0059] Figure 1 is a flow chart of an RGB-T crowd counting method based on cross-modal and multi-scale perception provided in an embodiment of the present invention;
[0060] Figure 2 : is a schematic diagram of the overall structure of an RGB-T crowd counting model based on cross-modal and multi-scale perception provided in an embodiment of the present invention;
[0061] Figure 3 is a detailed schematic diagram of a cross-modal feature fusion module in an embodiment of the present invention;
[0062] Figure 4 Detailed schematic diagram of a spatial context-aware graph convolution unit in an embodiment of the present invention;
[0063] Figure 5 is a schematic diagram of a regression head in an embodiment of the present invention;
[0064] Figure 6 It is a schematic diagram of crowd counting based on the RGB-T crowd counting method based on cross-modal and multi-scale perception provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0065] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatus.
[0066] The present invention is further described in detail below with reference to the accompanying drawings and embodiments:
[0067] like Figure 1 , 2 As shown, a RGB-T crowd counting method based on cross-modal and multi-scale perception is provided. The RGB-T crowd counting model based on cross-modal and multi-scale perception includes two parallel Swin Transformer backbones, three cross-modal feature fusion modules, a multi-scale feature aggregation module, and a regression head. The multi-scale feature aggregation module includes three spatial context-aware graph convolution modules. Specifically, the following steps are included:
[0068] Step 1: Input the paired RGB images and thermal images into the branches of the two-stream network composed of Swin Transformer to extract the specific feature maps of RGB mode and thermal mode at each stage. The two-stream network composed of Swin Transformer is designed to conduct in-depth analysis of RGB visual information and thermal information through two independent branches.
[0069] The two-stream network composed of Swin Transformer is divided into four stages. Since the underlying features contain too much noise information and consume more computing resources, only high-level features are used for fusion in subsequent steps.
[0070] Step 2: Input the specific feature maps of RGB mode and heat mode extracted by the last three stages (stages 2, 3, and 4) of the two-stream network composed of Swin Transformer into the cross-modal feature fusion module respectively, so as to achieve effective denoising while maintaining the shared features between each modality, and obtain the cross-modal fusion feature map output by the cross-modal feature fusion module.
[0071] Combine the following Figure 3 The cross-modal feature fusion module used in the present invention is described as follows:
[0072] like Figure 3 As shown, in order to obtain the cross-modal fusion feature map extracted by the cross-modal feature fusion module, the following contents are specifically included:
[0073] S2.1, the specific feature map F of the RGB modality extracted by the last three stages of the two-stream network composed of Swin Transformer RGB,i and the specific feature map F of the heat mode T,i After being processed by the adaptive maximum pooling layer and the fully connected layer, the most critical information is extracted from the feature map of each modality, the noise information is suppressed, and the denoised feature map E of the RGB modality is obtained. RGB,i And the denoised feature map of the heat mode E T,i ;
[0074] S2.2, based on the extracted significant features, the denoised feature maps of the RGB modality and the heat modality are fused to explore and utilize the complementary information between different modalities. The cross-modal shared feature map is obtained by adding the denoised feature maps of the RGB modality and the heat modality and then performing the Dropout process, followed by a full connection operation;
[0075] S2.3, the cross-modal shared feature map is processed by the Sigmoid activation function and converted into a cross-modal shared weight map E F,i ;
[0076] S2.4, multiplying the cross-modal shared weight map by the original input feature maps of the RGB modality and the heat modality element by element to obtain a cross-modal RGB feature map and a cross-modal heat feature map;
[0077] S2.5, further filtering the cross-modal RGB feature map and the cross-modal heat feature map through the residual structure to remove any residual noise, thereby obtaining a denoised cross-modal RGB feature map and a denoised cross-modal heat feature map;
[0078] S2.6, add the denoised cross-modal RGB feature map and the denoised cross-modal heat feature map to obtain the cross-modal fusion feature map F F,i .
[0079] The fusion process of the cross-modal feature fusion module for the specific feature maps of the RGB modality and the heat modality can be expressed as:
[0080]
[0081] F F,i =E F,i ·F RGB,i +F RGB,i +E F,i ·F T,i +F T,i (4)
[0082] Where i=2, 3, and 4 are the cross-modal feature fusion modules connected by the last three stages of the two-stream network composed of Swin Transformer; AMP(·) represents adaptive maximum pooling; and They represent the full connection of single modality features and the full connection of cross-modality fusion features respectively; ReLU(·) represents the rectified linear unit; Drop(·) represents dropout; Sig(·) represents the sigmoid activation function.
[0083] Step 3: Input the cross-modal fusion feature map output by the cross-modal feature fusion module into the multi-scale feature aggregation module to obtain a multi-scale fusion feature map. The multi-scale feature aggregation module models the spatial structure of crowds at various scales through a spatial context-aware graph convolution unit.
[0084] Combine the following Figure 2 The multi-scale feature aggregation module used in the present invention is described as follows:
[0085] like Figure 2 As shown in the figure, the multi-scale fusion feature map extracted by the multi-scale feature aggregation module is obtained. Specifically, it includes the following contents:
[0086] S3.1, the cross-modal fusion feature map F of each scale F,i After upsampling, they are unified to the same size for subsequent fusion processing; then 3×3 convolution is used to align the number of channels of feature maps of each scale to obtain aligned feature maps of each scale;
[0087] S3.2, the aligned feature maps of each scale are batch normalized, and then the ReLU activation function is applied to obtain the feature maps to be modeled X of each scale i ;
[0088] S3.3, input the feature graphs to be modeled at each scale into the spatial context-aware graph convolution unit, model the spatial structure of the crowd at different scales, and deeply mine the information therein to obtain the modeling feature graphs Y at each scale i ;
[0089] Combine the following Figure 4 The spatial context-aware graph convolution unit used in the present invention is described as follows:
[0090] like Figure 4 As shown in the figure, the modeling feature map extracted by the spatial context-aware graph convolution unit is obtained. Specifically, it includes the following contents:
[0091] S3.3.1, in order to reduce the amount of calculation, linear transformation is used to reduce the feature dimension. After 1×1 convolution, the number of channels is compressed and then averaged in the height dimension to obtain the reduced dimension feature map.
[0092] S3.3.2, perform dimension expansion on the reduced-dimensional feature map. Here, staggered expansion is used to expand the dimensions after the last dimension and before the last dimension, respectively, to obtain two increased-dimensional feature maps;
[0093] S3.3.3, perform a bilateral difference operation on the two dimension-enhanced feature maps to obtain a differential feature map, where the difference reflects the similarity of the population density between regions. A large difference reflects a large difference in the population density between regions, and a small difference reflects a similar population density between regions.
[0094] S3.3.4, the differential feature map is processed by the Tanh activation function, and then the number of channels is increased by 1×1 convolution to obtain the modeling weight map Finally, the modeling weight map and the original input feature map to be modeled are subjected to the Einstein sum operation to obtain a modeling feature map containing rich crowd structure information.
[0095] The modeling process of the spatial context-aware graph convolutional unit can be expressed as:
[0096]
[0097] In the formula, Represents a convolution with a kernel size of 1×1; Avg H (·) means taking the average in the height dimension; ε1(·) and ε2(·) mean dimensional expansion after the last dimension and before the last dimension, respectively; Represents the Tanh activation function; represents the Einstein summation operation; || represents the concatenation operation; represents the feature map to be modeled for the jth channel; x i:,j Represents the modeling weight map of the j-th channel.
[0098] S3.4, add the modeling feature maps of each scale to obtain the multi-scale fusion feature map Z, which integrates the semantic information of high-level features and the detailed information of low-level features, providing an accurate and comprehensive feature representation for the crowd density estimation task.
[0099] The fusion process of each scale feature map by the multi-scale feature aggregation module can be expressed as:
[0100]
[0101] Y i =SCGC(X i ) (9)
[0102]
[0103] In the formula, UP j (·) indicates upsampling by j times, where j = 2, 4, 8; represents a convolution with a kernel size of 3×3; SCGC(·) represents a spatial context-aware graph convolution unit.
[0104] Step 4: Send the multi-scale fusion feature map output by the multi-scale feature aggregation module to the regression head. After a series of processing, the predicted density map is regressed.
[0105] Combine the following Figure 5 The regression head in the present invention is described as follows:
[0106] like Figure 5 As shown, the predicted density map obtained after the regression head processing is obtained.
[0107] S4.1, downsample the multi-scale fusion feature map Z through 3×3 convolution to obtain the feature map Z to be regressed;
[0108] S4.2, the feature map to be regressed is processed by three parallel-connected dilated convolutions, with dilation rates of 1, 2, and 3 respectively. After each dilated convolution, it is processed by batch normalization and ReLU activation function to obtain the dilated feature map Z i ;
[0109] S4.3, connect the three dilated feature maps to obtain a multi-receptive field feature map Again, the feature map to be regressed is processed by 1×1 convolution, followed by batch normalization and ReLU activation function. According to the idea of residual network, a residual connection is made with the dilated feature map to obtain the residual feature map M.
[0110] S4.4, the residual feature map is processed by 3×3 convolution, batch normalization, ReLU activation function, 1×1 convolution, and ReLU activation function to obtain the predicted density map N. The regression process of the regression head can be expressed as:
[0111] Z=C 3×3 (Z) (11)
[0112] Z i =ReLU(BN(DConv i (Z))) (12)
[0113]
[0114]
[0115] Where i = 1, 2, 3; DConv i (·) denotes a dilated convolution with a dilation rate of i; Cat(·) denotes a concatenation operation.
[0116] Step 5: Add the obtained predicted density map pixel by pixel to get the predicted total number of people.
[0117] Figure 6 Schematic diagram of crowd counting using the crowd counting method of the present invention. Figure 6 When the crowd counting method of the present invention is used for crowd counting, the RGB image and the thermal image of the target scene are input into the RGB-T crowd counting model based on cross-modal and multi-scale perception, and the trained and optimized RGB-T crowd counting model is used for crowd counting. The obtained predicted density map is added pixel by pixel to obtain the predicted total number of people.
[0118] In summary, the present invention aims at the counting problem caused by the introduction of noise information and uneven crowd distribution in the existing cross-modal feature fusion process, and proposes an RGB-T crowd counting method based on cross-modal and multi-scale perception to achieve denoising and counting of scenes with uneven crowd distribution. By inputting paired RGB and thermal images into a two-stream network composed of Swin Transformer, specific feature maps of each modality are extracted; the cross-modal feature fusion module receives the output of the 2nd, 3rd, and 4th stages of the two-stream network to achieve the effect of effective denoising while sharing features between various modalities; the multi-scale feature aggregation module receives the cross-modal fusion feature map from the cross-modal feature fusion module, which models the spatial structure of the crowd at each scale through the spatial context-aware graph convolution unit, and then integrates the feature maps of different scales; the multi-scale feature map integrated by the multi-scale feature aggregation module is sent to the regression head to obtain a density map; the density map is added pixel by pixel to obtain the crowd counting result. The RGB-T crowd counting method based on cross-modal and multi-scale perception proposed in this paper can effectively realize denoising while sharing features among various modalities, model the spatial structure of the crowd and fuse multi-scale information to complete the counting task in scenes with uneven crowd distribution.
[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. An RGB-T crowd counting method based on cross-modal and multi-scale perception, characterized in that: The following steps are involved: Step 1: Input the paired RGB and thermal images into a two-stream network consisting of Swin Transformer to extract the specific feature maps of each modality; Step 2: Input the specific feature maps of RGB mode and heat mode extracted by the last three stages of the two-stream network composed of Swin Transformer into the cross-modal feature fusion module respectively, so as to achieve effective denoising while maintaining the shared features between each modality, and obtain the cross-modal fusion feature map output by the cross-modal feature fusion module; Step 3: The multi-scale feature aggregation module receives the cross-modal fusion feature map from the cross-modal feature fusion module, models the crowd spatial structure of each scale through the spatial context-aware graph convolution unit, and then integrates the feature maps of different scales; Step 4: Send the multi-scale fusion feature map output by the multi-scale feature aggregation module to the regression head. After a series of processing, the predicted density map is obtained by regression. Step 5: Add the obtained predicted density map pixel by pixel to get the predicted total number of people.
2. According to claim 1, the RGB-T crowd counting method based on cross-modal and multi-scale perception is characterized in that: In step 1, the paired RGB images and thermal images are respectively input into each branch of the two-stream network composed of Swin Transformer, and the specific feature maps of the RGB mode and the thermal mode of each stage are extracted. The two-stream network composed of Swin Transformer is divided into 4 stages.
3. According to claim 1, the RGB-T crowd counting method based on cross-modal and multi-scale perception is characterized in that: In step 2, the specific process is as follows: S2.1, the specific feature maps of RGB modality and heat modality extracted by the last three stages of the two-stream network composed of Swin Transformer are processed by adaptive maximum pooling layer and fully connected layer respectively, and the most critical and noise-suppressed information is extracted from the feature map of each modality to obtain the denoised feature maps of RGB modality and heat modality; S2.2, the denoised feature maps of the RGB modality and the heat modality are added together and then subjected to Dropout processing, followed by a full connection operation to obtain a cross-modal shared feature map; S2.3, the cross-modal shared feature map is processed by the Sigmoid activation function and converted into a cross-modal shared weight map; S2.4, multiplying the cross-modal shared weight map by the original input feature maps of the RGB modality and the heat modality element by element to obtain a cross-modal RGB feature map and a cross-modal heat feature map; S2.5, further filtering the cross-modal RGB feature map and the cross-modal heat feature map through the residual structure to remove any residual noise, thereby obtaining a denoised cross-modal RGB feature map and a denoised cross-modal heat feature map; S2.6, adding the denoised cross-modal RGB feature map and the denoised cross-modal heat feature map to obtain a cross-modal fusion feature map.
4. According to claim 1, the RGB-T crowd counting method based on cross-modal and multi-scale perception is characterized in that: In step 2, the fusion process of the cross-modal feature fusion module on the specific feature maps of the RGB modality and the thermal modality is expressed as: F F,i =E F,i ·F RGB,i +F RGB,i +E F,i ·F T,i +F T,i (4) Where i=2, 3, and 4 are the cross-modal feature fusion modules connected by the last three stages of the two-stream network composed of Swin Transformer; AMP(·) represents adaptive maximum pooling; and They represent the full connection of single modality features and the full connection of cross-modality fusion features respectively; ReLU(·) represents the rectified linear unit; Drop() represents dropout; Sig() represents the sigmoid activation function.
5. According to claim 1, the RGB-T crowd counting method based on cross-modal and multi-scale perception is characterized in that: In step 3, the specific process is as follows: S3.1, the cross-modal fusion feature maps of each scale are upsampled to the same size, and then the number of channels of the feature maps of each scale are aligned using 3×3 convolution to obtain the aligned feature maps of each scale; S3.2, the aligned feature maps of each scale are subjected to batch normalization respectively, and then the ReLU activation function is applied to obtain the feature maps to be modeled at each scale; S3.3, inputting the feature graphs to be modeled at each scale into the spatial context-aware graph convolution unit, modeling the crowd spatial structure at different scales, and obtaining the modeling feature graphs at each scale; S3.4, add the modeling feature maps of each scale to obtain a multi-scale fusion feature map.
6. The RGB-T crowd counting method based on cross-modal and multi-scale perception according to claim 5 is characterized in that: In step 3.3, the specific process is as follows: S3.3.1, subject the feature map to be modeled to 1×1 convolution processing to compress the number of channels, and then take the average in the height dimension to obtain a reduced-dimensional feature map; S3.3.2, perform dimension expansion processing on the reduced-dimensional feature map, respectively, perform dimension expansion after the last dimension and before the last dimension, and obtain two increased-dimensional feature maps; S3.3.3, performing a bilateral difference operation on the two dimension-enhanced feature maps to obtain a differential feature map; S3.3.4, the differential feature map is processed by the Tanh activation function, and then the number of channels is increased by 1×1 convolution to obtain the modeling weight map; S3.3.5, the modeling weight map and the original input feature map to be modeled are processed by Einstein sum operation to obtain a modeling feature map containing rich population structure information.
7. According to claim 5, the RGB-T crowd counting method based on cross-modal and multi-scale perception is characterized in that: In step 3.3, the modeling process of the spatial context-aware graph convolution unit is expressed as: In the formula, Represents a convolution with a kernel size of 1×1; Avg H (·) means taking the average in the height dimension; ε1(·) and ε2(·) mean dimensional expansion after the last dimension and before the last dimension, respectively; Represents the Tanh activation function; represents the Einstein summation operation; || represents the concatenation operation; represents the feature map to be modeled for the jth channel; x i:,j Represents the modeling weight map of the j-th channel.
8. The RGB-T crowd counting method based on cross-modal and multi-scale perception according to claim 5 is characterized in that: In step 3, the fusion process of each scale feature map by the multi-scale feature aggregation module is expressed as: Y i =SCGC(X i ) (9) In the formula, UP j (·) indicates upsampling by j times, where j = 2, 4, 8; represents a convolution with a kernel size of 3×3; SCGC(·) represents a spatial context-aware graph convolution unit.
9. The RGB-T crowd counting method based on cross-modal and multi-scale perception according to claim 1 is characterized in that: In step 4, the specific process is as follows: S4.1, downsample the multi-scale fusion feature map through 3×3 convolution to obtain the feature map to be regressed; S4.2, the feature map to be regressed is processed by three parallel-connected dilated convolutions, with dilation rates of 1, 2, and 3 respectively. After each dilated convolution, it is processed by batch normalization and ReLU activation function to obtain a dilated feature map. Subsequently, the three dilated feature maps are connected to obtain a multi-receptive field feature map; S4.3, the feature map to be regressed is processed by 1×1 convolution, and then processed by batch normalization and ReLU activation function. According to the idea of residual network, residual connection is performed with the dilated feature map to obtain the residual feature map; S4.4, the residual feature map is processed by 3×3 convolution, batch normalization, ReLU activation function, 1×1 convolution, and ReLU activation function to obtain the predicted density map.
10. The RGB-T crowd counting method based on cross-modal and multi-scale perception according to claim 1 is characterized in that: In step 4, the regression process of the regression head is expressed as: WITH i =ReLU(BN(DConv i (Z)))(12) Where i = 1, 2, 3; DConv i () represents a dilated convolution with a dilation rate of i; Cat() represents a concatenation operation.
