A multi-scale alignment fusion-based multi-modal crowd counting method

Abstract

This invention provides a multimodal crowd counting method based on multi-scale alignment and fusion. First, a multimodal crowd scene dataset is acquired and divided into training, validation, and test sets. Preprocessed RGB images and multimodal auxiliary images are input into a VGG16 backbone network to extract multi-stage high-level feature maps. These are then sequentially fed into a density-sharing local contrastive learning module, a local feature fusion module, and an adaptive Mamba context-aware fusion module to achieve cross-modal fine-grained alignment, local feature fusion, and global context modeling. Subsequently, the global fused features are input into a dynamically upsampled multi-scale feature decoder to generate a high-resolution crowd density map. Supervised training is performed using a composite loss function, and the optimal model is saved for testing. This invention adopts an "align-then-fuse" architecture, effectively mitigating cross-modal heterogeneity and density fluctuation problems through multi-module collaborative design, significantly improving the accuracy and robustness of crowd counting in complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent monitoring technology, and in particular relates to a multimodal crowd counting method based on multi-scale alignment fusion, which is applicable to crowd density statistics and number estimation tasks in all-weather and complex scenarios such as public safety monitoring, urban traffic management, and security for large-scale events. Background Technology

[0002] Crowd counting, as a core technology in intelligent monitoring and public safety management, aims to accurately count the number of people in specific scenarios or generate crowd density distribution maps, providing data support for resource scheduling and risk warning. With the development of deep learning technology, crowd counting methods based on convolutional neural networks have made significant progress, but existing technologies still have many bottlenecks that limit their application in complex real-world scenarios. Traditional crowd counting methods primarily rely on unimodal RGB data, extracting image features through manual feature extraction or deep learning models to complete the counting. However, RGB sensors are susceptible to lighting conditions, exhibiting texture degradation and information loss in low-light, backlight, and nighttime scenarios, making it difficult to distinguish foreground objects from the background. Furthermore, in complex scenarios with drastic fluctuations in crowd density (such as the coexistence of sparse and ultra-high-density areas) and severe occlusion, convolutional operations with a fixed receptive field struggle to capture both fine-grained local details and global contextual information, resulting in a significant decrease in counting accuracy.

[0003] To address the limitations of single-modal RGB data, the fusion of multimodal data (such as depth and thermal imaging) has become a research hotspot. Among these, multimodal (visible light-thermal imaging / depth) fusion, with its advantages of thermal imaging / depth data being unaffected by lighting changes and capable of capturing pedestrian depth / thermal features, has become an ideal choice for all-weather crowd counting. However, multimodal crowd counting still faces two major challenges: First, cross-modal heterogeneity: RGB images have rich textures but are sensitive to lighting, while thermal imaging / depth data have clear outlines but low resolution and lack detail. Furthermore, sensor displacement can easily lead to spatial misalignment, resulting in artifacts and counting biases when directly fused. Second, poor adaptability to density fluctuations: large differences in crowd scale within a single frame make it difficult for existing methods to effectively capture multi-scale features, and they are insufficient for long-distance dependency modeling of ultra-high-density crowds. Existing multimodal crowd counting methods mostly employ implicit feature fusion strategies, lacking explicit cross-modal alignment mechanisms, leading to spatial misalignment and semantic inconsistencies. While some methods introduce attention mechanisms or multi-branch structures, they suffer from high computational complexity, insufficient global context capture, and inefficient multi-scale feature fusion, failing to simultaneously meet the requirements of counting accuracy and real-time performance. Therefore, designing a multimodal crowd counting method that achieves accurate cross-modal alignment, efficient multi-scale feature fusion, and considers both global and local information has become a pressing technical problem in this field. Summary of the Invention

[0004] To achieve the above-mentioned objectives, the technical solution adopted by this invention is: a multimodal crowd counting method based on multi-scale alignment fusion, comprising the following steps:

[0005] Step 1) Dataset Acquisition and Preprocessing: Acquire a multimodal crowd scene dataset, dividing it proportionally into training, validation, and test sets. Perform augmented preprocessing on the data to obtain standardized input data. Backbone Network Feature Extraction: Input the preprocessed RGB images and multimodal auxiliary images (thermal imaging / depth maps) into a VGG backbone network with shared weights, and extract high-level feature maps at different stages through convolution and pooling operations.

[0006] Step 2) Cross-modal fine-grained alignment: Input the feature map output by the backbone network into the density-sharing local contrastive learning module. Through block contrastive learning and density-guided mask reconstruction, achieve spatial alignment and semantic complementarity of cross-modal features and suppress inter-modal noise.

[0007] Step 3) Multi-scale local feature fusion: The aligned feature map is input into the local feature fusion module. Through cross-modal feature injection and multi-branch dilated convolution, multi-scale local texture features are extracted to adapt to population density fluctuations.

[0008] Step 4) Global Context Modeling: Input the local fusion features into the Adaptive Mamba Context-Aware Fusion Module, and take advantage of the linear complexity of the Mamba architecture to capture long-distance contextual dependencies and achieve global-local feature collaboration.

[0009] Step 5) Dynamic multi-scale decoding: Input the global fused features into the dynamic upsampling multi-scale feature decoder, and generate a high-resolution population density map through neighboring layer modulation and cross-scale aggregation strategies.

[0010] Step 5) The attention calculation result is input into the upsampling module to restore the spatial resolution. During the restoration process, long connections are maintained with the shallow feature map to enhance the feature expression ability. Finally, the resolution is restored to the initial image to obtain the segmentation result.

[0011] Model training and testing: Supervised training of the model is performed using a composite loss function. The optimal model is selected through the validation set, and the model performance is evaluated using the test set. The final population counting results and density map are then output.

[0012] Furthermore, the multimodal crowd scene dataset mentioned in step 1 includes the open-source datasets RGBT-CC, DroneRGBT, and ShanghaiTechRGBD, with the input image normalized to a resolution of 256×256 (multimodal thermal imaging dataset) or 1024×1024 (multimodal depth dataset).

[0013] Furthermore, the specific operations of the density-sharing local contrastive learning module described in step 3 include:

[0014] Feature segmentation: Perform non-overlapping segmentation on the RGB branch feature map and the multimodal auxiliary branch feature map to obtain a set of local feature blocks, mathematically represented as:

[0015]

[0016] Density-guided contrastive alignment: A shared density template is constructed, and through contrastive learning, corresponding feature blocks (positive sample pairs) in the distance space are brought closer together, while non-corresponding feature blocks (negative sample pairs) are pushed apart, achieving fine-grained spatial alignment. Mathematically, this is expressed as:

[0017] Furthermore, the depth spatial module specifically involves: first, performing a 1×1 convolution to expand the channels; then, summing the results pixel-by-pixel through a spatially separable extraction block and a 3×3 convolution block; finally, passing the results through a depth-separable extraction block and performing a residual connection with the initial input. The specific mathematical representation is as follows:

[0018]

[0019] Mask Reconstruction and Density Modulation: A complementary binary mask (M_V・M_A = 0) is used to simulate single-modal information loss. Missing information is reconstructed through cross-modal channel attention. The residual is modulated using a shared density prior (with the weights of multimodal auxiliary data adjusted to 0.6 in low-light scenes). The mathematical representations are as follows:

[0020]

[0021]

[0022]

[0023] Where σ is the Sigmoid activation function and A is the cross-modal channel attention operation.

[0024] Feature fusion output: The reconstructed RGB and multimodal auxiliary feature blocks are fused after adjusting the number of channels by 1×1 convolution to obtain the aligned feature map.

[0025] Cross-modal feature injection: This involves fusing backbone network features with aligned features from density-shared local contrastive learning modules to enrich the representation of single-modal features. The mathematical representation is as follows:

[0026]

[0027]

[0028] in, For multimodal auxiliary backbone features, It is an RGB backbone feature.

[0029] Multiply-add fusion operation: The injected features are fused by combining element-wise multiplication and addition to obtain the module input, mathematically represented as:

[0030]

[0031] Where ・ represents element-wise multiplication and + represents element-wise addition.

[0032] Multi-scale feature extraction: A structure combining multi-branch depthwise separable convolution and dilated convolution is adopted. The kernel sizes are 1, 3, 5, and 7, and the dilation rates are 1, 3, 5, and 7. Multi-scale local features are extracted and concatenated, mathematically represented as follows:

[0033]

[0034]

[0035] Channel-spatial attention filtering: Feature importance is calibrated by channel attention and background noise is suppressed by spatial attention to obtain the final local fusion features.

[0036] Furthermore, the specific operations of the adaptive Mamba context-aware fusion module described in step 5 include: Feature enhancement: The backbone network features are fused with the alignment features from the density-sharing local contrastive learning module and the output features from the local fusion module to enhance cross-modal alignment characteristics. The mathematical representation is as follows:

[0037]

[0038]

[0039] Multi-scale context injection: Context information is aggregated and upsampled using dual-branch pooling (steps 2 and 4) to restore resolution, avoiding the loss of local information caused by Mamba serialization. Mathematically, this is expressed as:

[0040]

[0041] Dynamic masking and Mamba modeling: Normalized RGB data is fused with multimodal auxiliary features to generate a dynamic mask that suppresses the background. Long-range dependencies are captured using the ES2D (Efficient Spatial 2D Scanning) module. The mathematical representations are as follows:

[0042]

[0043]

[0044] Residual fusion output: The input features, mask features, and channel attention features are fused residually to obtain the global fused features, mathematically represented as:

[0045]

[0046] Furthermore, the specific operations of the dynamic upsampling multi-scale feature decoder described in step 6 include:

[0047] Neighbor layer modulation: Starting from the deepest layer features, the deep semantic features are upsampled as a mask to modulate the shallow features, suppressing background noise. Mathematically, this can be expressed as:

[0048]

[0049] in, This represents an upsampling operation with a scaling factor of s.

[0050] Cross-scale aggregation: fusing modulation features from adjacent layers with deep skip connection features to preserve multi-scale information, mathematically represented as:

[0051]

[0052] Furthermore, the composite loss function described in step 7 includes Bayesian density regression loss, cross-modal local contrast loss, and mask reconstruction loss. The total loss is a weighted sum of the three, mathematically expressed as:

[0053]

[0054] Where λ1+λ2=1 is the balancing weight; To address the labeling ambiguity problem, a density probability distribution is modeled for the Bayesian density regression loss. For cross-modal local contrastive loss, maximize the mutual information of corresponding feature blocks; To compensate for mask reconstruction loss, ensure cross-modal complementary learning.

[0055] Where λ1+λ2=1 is the balancing weight; To address the labeling ambiguity problem, a density probability distribution is modeled for the Bayesian density regression loss. For cross-modal local contrastive loss, maximize the mutual information of corresponding feature blocks; To compensate for mask reconstruction loss, ensure cross-modal complementary learning.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] This invention proposes a core architecture of "alignment before fusion". Through a density-sharing local contrastive learning module, it achieves cross-modal fine-grained spatial alignment and semantic complementarity, effectively solving the problems of spatial misalignment and modal heterogeneity of multimodal data, suppressing fusion artifacts and counting bias, and improving robustness in complex lighting scenarios. The local feature fusion module designed in this invention adopts cross-modal injection and multi-branch dilated convolution structure, which can accurately capture multi-scale local texture features and adapt to scenes with drastic fluctuations in crowd density. At the same time, it filters background noise through channel-spatial attention and enhances the feature expression of foreground targets.

[0058] This invention introduces an adaptive Mamba context-aware fusion module, which leverages the linear complexity advantage of the Mamba architecture to efficiently capture long-distance contextual dependencies of ultra-high-density crowds while avoiding the high computational cost of Transformer, achieving global-local feature collaboration and solving the problem of insufficient global modeling in traditional CNNs.

[0059] The dynamic upsampling multi-scale feature decoder of this invention adopts a neighboring layer modulation and cross-scale aggregation strategy. It guides the recovery of shallow details through deep semantic features, effectively bridging the semantic gap between deep and shallow features, and generating a high-resolution density map that combines accuracy and detail consistency.

[0060] This invention employs a composite loss function consisting of Bayesian density regression loss, cross-modal local contrast loss, and mask reconstruction loss to supervise model training from three dimensions: density modeling, cross-modal alignment, and semantic complementarity, thereby improving the model's generalization ability and counting accuracy.

[0061] The model of this invention achieves the best performance on three multimodal benchmark datasets: RGBT-CC, DroneRGBT, and ShanghaiTechRGBD. It is stable in complex scenarios such as low light, ultra-high density, and UAV overhead view, and has a moderate number of parameters and computational complexity, making it valuable for practical engineering applications. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of a multimodal crowd counting method based on multi-scale alignment fusion according to the present invention;

[0064] Figure 2 This is a schematic diagram of the feature alignment module structure of the present invention;

[0065] Figure 3 This is a schematic diagram of the context modeling module structure of the present invention; Detailed Implementation

[0067] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The embodiments are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.

[0068] This embodiment uses the multimodal dataset RGBT-CC as the experimental object. The dataset contains 2030 pairs of RGB-thermal imaging images (1013 bright scenes and 1017 dark scenes), with a total of 138389 labeled pedestrians. The multimodal crowd counting method based on multi-scale alignment fusion of the present invention is described in detail, and the specific steps are as follows:

[0069] Step 1) Dataset Acquisition and Preprocessing: Download the RGBT-CC multimodal dataset from the open-source dataset website and divide it into training set (1421 pairs), validation set (203 pairs), and test set (406 pairs) in a 7:1:2 ratio. Perform augmentation preprocessing on all data, performing random horizontal flipping, mean normalization, and random scaling (scale factor 0.75, 1.0, 1.25, 1.5) in sequence. The standardized input image resolution is 256×256, and the number of channels is 3 (RGB) + 1 (thermal imaging).

[0070] Step 2) Backbone network feature extraction: The preprocessed RGB image and multimodal thermal imaging image are input into the VGG16 backbone network, which initializes the ImageNet pre-trained weights. Through 5 convolutional layers and 3 pooling layers, high-level feature maps of 4 stages are extracted, where the resolution of the k-th stage feature map is 1 / 2^k of the input image, and the number of channels is 64, 128, 256, and 512, respectively.

[0071] Step 3) Cross-modal fine-grained alignment: Input the RGB feature map (512 channels, 8×8 resolution) and multimodal thermal imaging feature map (512 channels, 8×8 resolution) output from the fourth stage of VGG16 into the density-sharing local contrast learning module.

[0072] Feature segmentation: Perform 8×8 non-overlapping segmentation to obtain a set of 64 1×1 feature blocks, mathematically represented as:

[0073]

[0074] Density-guided contrast alignment: Calculating shared density templates By contrastive learning, corresponding feature blocks in the embedding space are brought closer together, while non-corresponding blocks are pushed apart, thus achieving coarse-to-fine spatial alignment. Mathematically, this can be represented as:

[0075]

[0076] Mask Reconstruction: Generate a complementary binary mask with an occlusion rate set to 0.3. Reconstruct the RGB mask region information using multimodal thermal imaging features through cross-modal channel attention, and vice versa. The mathematical representations are as follows:

[0077]

[0078]

[0079] Density modulation: The reconstructed residuals are modulated using a shared density template, ensuring that detail enhancement is concentrated in high-density population areas. Mathematically, this is expressed as:

[0080]

[0081] Feature fusion: The reconstructed feature blocks are adjusted to 256 channels by 1×1 convolution, and then element-wise summed to obtain an aligned feature map (256 channels, 8×8 resolution).

[0082] Step 4: Multi-scale local feature fusion

[0083] Input the aligned feature map into the local feature fusion module:

[0084] Cross-modal feature injection: The aligned feature map is fused with the VGG16 stage 3 feature map to obtain a rich feature representation, mathematically represented as follows:

[0085]

[0086]

[0087] Multiplication-addition fusion: By combining element-wise multiplication and addition, the module input (512 channels, 8×8 resolution) is obtained, mathematically represented as:

[0088]

[0089] Multi-scale extraction: A 4-branch depthwise separable convolution is used with kernel sizes of 1, 3, 5, and 7, and dilatations of 1, 3, 5, and 7, respectively. Multi-scale features are extracted and concatenated into a feature map with 1024 channels. The mathematical representation is as follows:

[0090]

[0091]

[0092]

[0093] Attention filtering: Feature weights are calculated through channel attention, and background noise is suppressed by combining 7×7 spatial attention, outputting local fusion features (512 channels, 8×8 resolution).

[0094] Step 5: Global Context Modeling

[0095] The local fusion features are input into the adaptive Mamba context-aware fusion module:

[0096] Feature Enhancement: Enhanced features are obtained by fusing VGG16 Stage 4 backbone features, density-sharing local contrastive learning module alignment features, and local fusion features. The mathematical representations are as follows:

[0097]

[0098]

[0099] Multi-scale context injection: Context is aggregated using max pooling with strides of 2 and 4, and upsampled to recover to 8×8 resolution to obtain multi-scale context features, mathematically represented as:

[0100]

[0101] Dynamic masking: By fusing normalized features, a dynamic mask is generated to highlight foreground crowd regions. Mathematically, this is represented as:

[0102]

[0103] Mamba modeling: A dynamic mask is input into the ES2D module to capture long-range contextual dependencies. After channel attention calibration, global features are obtained, mathematically represented as:

[0104]

[0105] Residual fusion: The input features, mask features, and global feature residuals are added together to output a global fused feature (512 channels, 8×8 resolution), mathematically represented as:

[0106]

[0107] Step 6: Dynamic Multi-Scale Decoding

[0108] The globally fused features are input into a dynamically upsampled multi-scale feature decoder and decoded in order from deepest to shallowest: Modulation from Stage 4 to Stage 3: The global fusion feature is upsampled by a factor of 2 (resolution 16×16) and multiplied by the elements of the Stage 3 feature map (256 channels) to obtain the modulation feature (256 channels, 16×16 resolution), mathematically represented as:

[0109]

[0110] Phase 3 to Phase 2 aggregation: The modulation feature is upsampled by 2 times (resolution 32×32), multiplied by the elements of the Phase 2 feature map (128 channels), and simultaneously incorporated by the Phase 4 upsampled feature. The resulting aggregated feature (384 channels, resolution 32×32) is then concatenated, mathematically represented as:

[0111]

[0112] Phase 2 to Phase 1 aggregation: The aggregated feature is upsampled by 2 times (resolution 64×64), multiplied by the elements of the Phase 1 feature map (64 channels), and then combined with the features upsampled by 4 times in Phase 3 and 8 times in Phase 4 to obtain the high-dimensional aggregated feature (512 channels, resolution 64×64), mathematically represented as:

[0113]

[0114] Density map generation: The number of high-dimensional aggregated feature channels is adjusted to 1 through 1×1 convolution, and upsampled by 4 times to a resolution of 256×256 to obtain the final crowd density map, mathematically represented as:

[0115]

[0116] Step 7: Model Training and Testing

[0117] 1. Model Training: Supervised training using a composite loss function, with Adam as the optimizer, a learning rate of 1×10^-5, a batch size of 1, and 600 training epochs. The mathematical representation of the loss function is as follows:

[0118]

[0119] After each round of training, the model performance is evaluated using a validation set, and the model with the lowest MAE and RMSE is saved.

[0120] 2. Model testing: Input the test set into the optimal model, output the crowd density map for each image, and integrate the density map to obtain the final population estimation result;

[0121] 3. Performance Evaluation: On the RGBT-CC multimodal test set, the results show MAE=11.02 and RMSE=19.04, and RMSE=37.27 in ultra-high density scenarios, all of which are better than existing mainstream methods.

[0122] The technical content and features of this invention have been disclosed above. Those skilled in the art can make partial modifications or substitutions to the technical solution of this invention without departing from the spirit of this invention. All content that does not depart from the technical solution of this invention falls within the protection scope of this invention.

Claims

1. A multimodal population counting method based on multi-scale alignment fusion, characterized in that: The method includes the following steps: Step 1) Obtain a multimodal crowd scene dataset, divide the dataset into training set, validation set and test set according to the proportion, perform augmentation preprocessing on the data to obtain standardized input data; Step 2) Input the preprocessed RGB image and multimodal auxiliary image into the VGG16 backbone network with shared weights, and extract multi-stage high-level feature maps through convolution and pooling operations; Step 3) Input the high-level feature map into the density-sharing local contrastive learning module. Through feature segmentation, density-guided contrastive alignment, mask reconstruction, density modulation and feature fusion, fine-grained alignment of cross-modal features is achieved to obtain the aligned feature map. Step 4) Input the aligned feature map into the local feature fusion module. After cross-modal feature injection, multiply-accumulate fusion, multi-scale extraction and channel-space attention filtering, the local fused features are obtained. Step 5) Input the local fusion features into the adaptive Mamba context-aware fusion module. Through feature enhancement, multi-scale context injection, dynamic masking, Mamba modeling and residual fusion, global context modeling is completed to obtain global fusion features. Step 6) Input the global fusion features into the dynamic upsampling multi-scale feature decoder, and generate a high-resolution population density map by performing neighboring layer modulation, cross-scale aggregation and density map generation in order from deep to shallow. Step 7) Use a composite loss function to supervise the training and validation of the model, save the model with the best validation performance, use the model to process the test set, and obtain the final population counting result.

2. The multimodal crowd counting method based on multi-scale alignment fusion according to claim 1, characterized in that: Step 1) further includes: Step 1.1 Downloading RGBT-CC, DroneRGBT, and ShanghaiTechRGBD multimodal crowd scene datasets from open-source dataset websites; Step 1.2 Allocating each dataset to a training set, validation set, and test set in a 7:1:2 ratio; The input image is normalized to a resolution of 256×256 (multimodal thermal imaging dataset) or 1024×1024 (multimodal depth dataset).

3. The multimodal crowd counting method based on multi-scale alignment fusion according to claim 1, characterized in that: The specific operations of the density-shared local contrastive learning module in step 3) include: Feature segmentation: performing non-overlapping segmentation on the RGB branch feature map and the multimodal auxiliary branch feature map to obtain a set of local feature blocks; Density-guided contrastive alignment: constructing a shared density template, using contrastive learning to bring corresponding feature blocks closer together and push away non-corresponding feature blocks to achieve fine-grained spatial alignment; Mask reconstruction and density modulation: using complementary binary masks to simulate single-modal information loss, reconstructing missing information through cross-modal channel attention, and using shared density prior modulation to reconstruct residuals, where the weight of multimodal auxiliary data is adjusted to 0.6 in low-light scenes; Feature fusion output: fusing the reconstructed RGB and multimodal auxiliary feature blocks after adjusting the number of channels through 1×1 convolution to obtain an aligned feature map.

4. The multimodal crowd counting method based on multi-scale alignment fusion according to claim 3, characterized in that: In the mask reconstruction and density modulation, the complementary binary mask satisfies M_V・M_A=0, the shared density prior is 0.5D_V+0.5D_A, and the reconstruction residual modulation achieves detail enhancement in high-density regions through the Sigmoid activation function.

5. The multimodal crowd counting method based on multi-scale alignment fusion according to claim 1, characterized in that: The specific operations of the local feature fusion module in step 4) include: cross-modal feature injection: fusing the backbone network features with the alignment features of the density-shared local contrastive learning module to enrich the single-modal feature expression; multiply-add fusion: fusing the injected features through a combination of element-wise multiplication and addition to obtain the module input; multi-scale extraction: using a structure combining multi-branch depthwise separable convolution and dilated convolution, with the convolution kernel size set to {1,3,5,7} and the dilation rate set to {1,3,5,7}, extracting and concatenating multi-scale local features; attention filtering: calibrating feature importance through channel attention, combining 7×7 spatial attention to suppress background noise, and outputting local fused features.

6. The multimodal crowd counting method based on multi-scale alignment fusion according to claim 1, characterized in that: The specific operation of the adaptive Mamba context-aware fusion module in step 5) Includes: Feature enhancement: fusing backbone network features with density-shared local contrastive learning module alignment features and local fusion module output features to enhance cross-modal alignment characteristics; Multi-scale context injection: aggregating context information through dual-branch pooling with strides of 2 and 4, upsampling to restore resolution, and avoiding local information loss caused by Mamba serialization; Dynamic masking: Normalized RGB and multimodal auxiliary features are fused to generate a dynamic mask to highlight the foreground crowd region; Mamba modeling: The dynamic mask is input into the ES2D module to capture long-range contextual dependencies, and global features are obtained after channel attention calibration; Residual fusion: The input features, mask features and global features are fused together to obtain global fused features.

7. A multimodal crowd counting method based on multi-scale alignment fusion according to claim 1, characterized in that: The specific operations of the dynamic upsampling multi-scale feature decoder in step 6) include: Neighbor layer modulation: Starting from the deepest layer features, the deep semantic features are used as masks to modulate the shallow features through upsampling to suppress background noise. The upsampling scale factor is set to 2. Cross-scale aggregation: The adjacent layer modulation features are fused with the deep skip-connected features. The upsampling scale factors of the deep skip-connected features are set to 4 and 8 respectively. Density map generation: All scale features are aggregated, the number of channels is adjusted through 1×1 convolution, and then upsampled with a scale factor of 4 to obtain a high-resolution crowd density map with the same resolution as the input image.

8. The multimodal crowd counting method based on multi-scale alignment fusion according to claim 1, characterized in that: In step 7), the composite loss function is a weighted sum of Bayesian density regression loss, cross-modal local contrast loss, and mask reconstruction loss, with all weighting coefficients set to 1. The Bayesian density regression loss is used to solve the labeling ambiguity problem, the cross-modal local contrast loss is used to enhance cross-modal alignment, and the mask reconstruction loss is used to ensure cross-modal complementary learning.

Images