Intelligent processing and crowd counting method based on photo-thermal image enhancement

Abstract

The invention discloses an intelligent processing and crowd counting method based on photo-thermal image enhancement, and relates to the technical field of computer vision, and the method comprises the steps: 1, collecting a multi-modal image of a target scene, and constructing a training data set; 2, constructing a bimodal calculation architecture based on a deep learning network, and carrying out training optimization by adopting the training data set to obtain a crowd counting model; the crowd counting model comprises an all-weather multi-modal fusion crowd counting network and a weak supervision multi-modal graph convolution counting network; and 3, collecting a to-be-counted multi-modal image, and inputting the to-be-counted multi-modal image into the crowd counting model to obtain a counting result. Visible light image and thermal infrared image information captured by acquisition equipment is deeply fused through a bimodal calculation architecture, and the environment illumination condition is adaptively sensed, so that the crowd counting accuracy and generalization ability in various dense public places are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to an intelligent processing and crowd counting method based on photothermal image enhancement. Background Technology

[0002] In the field of crowd counting, existing products and methods still face multiple challenges. First, most traditional methods cannot effectively cope with the fluctuations in imaging quality under complex all-weather environments (such as noise at night, overexposure in strong light, and blurring due to fog). Their performance often drops significantly due to the limitations of a single visible light (RGB) modality, making it difficult to achieve stable all-weather monitoring. Second, in aerial photography and densely populated scenes, traditional methods struggle to accurately capture the multi-scale features of large targets at close range and small targets at long distances, and their ability to identify severely occluded individuals is insufficient, resulting in limited counting accuracy. Furthermore, even when some multimodal fusion schemes are proposed, they generally adopt fixed or simple fusion strategies, failing to fully exploit the complementary advantages of different modalities such as visible light and thermal infrared under different signal-to-noise ratio environments, and thus failing to achieve intelligent adaptive fusion. In addition, traditional high-precision crowd counting methods heavily rely on time-consuming and labor-intensive pixel-level densely labeled data, greatly increasing training costs and application barriers. While existing weakly supervised methods strive to reduce labeling costs, they often struggle to maintain high accuracy while ensuring the integrity and robustness of spatial information, limiting their generalization ability in complex environments.

[0003] Therefore, how to improve the accuracy and generalization ability of crowd counting in various dense public places is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide an intelligent processing and crowd counting method based on photothermal image enhancement to overcome or at least partially solve the above problems. By deeply fusing visible light (RGB) images and thermal infrared (TIR) ​​image information captured by the acquisition device, and adaptively sensing the ambient lighting conditions, the method effectively improves the accuracy and generalization ability of crowd counting in various dense public places.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, embodiments of the present invention provide an intelligent processing and crowd counting method based on photothermal image enhancement, comprising the following steps: Step 1: Collect multimodal images of the target scene and annotate the multimodal images to construct a training dataset; Step 2: Construct a dual-modal computing architecture based on deep learning networks. This architecture includes two parallel processing branches: fully supervised and weakly supervised. The appropriate processing branch is selected for training and optimization according to the labeling type of the training dataset to obtain the crowd counting model. The crowd counting model includes an all-weather multimodal fusion crowd counting network and a weakly supervised multimodal graph convolutional counting network. The all-weather multimodal fusion crowd counting network includes an encoder, an ambient light discriminator, a multi-scale multimodal fusion module, and a decoder. The encoder extracts multi-scale multimodal features from the multimodal image, the ambient light discriminator extracts light intensity information from the multimodal features, the multi-scale multimodal fusion module performs deep fusion of the multi-scale multimodal features and light intensity information to generate multimodal fusion features, and the decoder decodes the multimodal fusion features to generate a crowd density map. The weakly supervised multimodal graph convolutional counting network includes an encoder, an illumination guidance cue generator, an adaptive graph context modeler, a multimodal feature optimizer, and a decoder. The encoder extracts multi-scale multimodal features from the multimodal image. The illumination guidance cue generator generates cue information based on the visible light features in the multimodal features and injects it into the visible light features. The adaptive graph context modeler performs context modeling and fusion on the visible light features of the injected cue information to generate updated multimodal features and context fusion features. The multimodal feature optimizer enhances the updated multimodal features and context fusion features to obtain enhanced features. The decoder decodes and predicts the counting results based on the enhanced features. Step 3: Collect the multimodal images to be counted and the counting requirements, input them into the crowd counting model, and select either an all-weather multimodal fusion crowd counting network or a weakly supervised multimodal graph convolutional counting network to recognize the multimodal images to be counted and obtain the counting results.

[0007] Preferably, the multimodal images include acquired visible light images and thermal infrared images.

[0008] Preferably, a training dataset containing pixel-level density icon annotations is used to train the all-weather multimodal fusion crowd counting network for the fully supervised processing branch; and a training dataset containing only image-level total number of people annotations is used to train the weakly supervised multimodal graph convolutional counting network for the weakly supervised processing branch.

[0009] Preferably, the all-weather multimodal fusion crowd counting network includes two encoders, an ambient light discriminator, several multi-scale multimodal fusion modules, and a decoder. Visible light images and thermal infrared images are respectively input to one encoder for multi-scale feature extraction, obtaining multi-scale visible light features and thermal infrared features respectively. At the same time, the visible light image is input to the ambient light discriminator to extract light intensity weights. The visible light features, thermal infrared features, and light intensity weights at each scale are respectively input to one multi-scale multimodal fusion module for deep fusion, obtaining multimodal fusion features at the corresponding scale. The multimodal fusion features at all scales are input to the decoder for decoding, generating a crowd density map and counting results.

[0010] Preferably, the encoder uses a Sven transformer as the backbone network, including a block layer and four sets of feature extraction units. The first set of feature extraction units includes a linear projection layer and a main coding layer, while the remaining feature extraction units include an image patch merging layer and a main encoder. The block layer divides the input image into fixed-size patches and transmits them to the first set of feature extraction units. The linear projection layer of the first set of feature extraction units performs linear embedding on the patches to obtain patch embeddings, and the main encoder uses window multi-head self-attention and shift window multi-head self-attention mechanisms to capture local and global features in the patch embeddings, obtaining features F1 at the first scale, which is then transmitted to the second set of feature extraction units. The second set of feature extraction units reduces the resolution of features F1 at the first scale through the image patch merging layer, while increasing the number of channels, and extracts features F2 at the second scale through the main encoder, which is then transmitted to the third set of feature extraction units. The third set of feature extraction units extracts features F3 at the third scale based on features F2 at the second scale, and transmits them to the fourth set of feature extraction units. The fourth set of feature extraction units extracts features F4 at the fourth scale based on features F3 at the third scale. The number of channels for features F1, F2, F3, and F4 at different scales are 128, 256, 512, and 1024, respectively. These features reflect image information from local to global perspectives. Multi-scale visible light features are extracted from the visible light image using an encoder. The thermal infrared image is processed by an encoder to extract multi-scale thermal infrared features. .

[0011] Preferably, the ambient lighting discriminator includes a multi-layer convolutional neural network, a global average pooling layer, a fully connected layer, and a processing unit; the multi-layer convolutional neural network extracts deep semantic features of the image; the global average pooling layer compresses the extracted two-dimensional features into a one-dimensional global ambient fingerprint vector, which highly condenses the overall attributes of the current scene; the fully connected layer performs dimensionality reduction mapping on the global ambient fingerprint vector to obtain a probability distribution; the processing unit calculates the illumination intensity weights of the visible light image based on the probability distribution and transmits them to the multi-scale multimodal fusion module.

[0012] Preferably, the fully connected layer (FC) includes two sets of cascaded fully connected operations, so that the output of the module is no longer interpreted as a simple day / night classification, but is regarded as the dominant probability P of the RGB mode in the current scene. d With the dominant probability P of the thermal infrared mode n The soft prediction is expressed as: ; ; Where P represents the activation function after a nonlinear activation function. Feature response intensity after nonlinear activation; F represents the global environment fingerprint vector; P d and P n These represent the probability distributions that are more suitable for using high-confidence RGB features and the probability distributions that are more suitable for using low-confidence thermal infrared features, respectively. This indicates the first group of fully connected operations; This indicates the second group of fully connected operations.

[0013] Preferably, the processing unit includes a normalization layer and a nonlinear activation function layer. The normalization layer normalizes the probability distribution, and the nonlinear activation function Sigmoid is used to calculate the light intensity weights based on the normalized probability distribution; the expression is: ; ; in, Indicates the weight of light intensity; These are the parameters of the nonlinear activation function; This indicates the probability that a thermal infrared image contains valid information under the current ambient lighting conditions. This indicates the probability that a visible light image contains valid information under the current ambient lighting conditions.

[0014] Preferably, the multi-scale multimodal fusion module includes a dimensional cross-modal perception module, a pixel and channel optimizer, and a fusion feature module; The dimensional cross-modal perception module includes a cross-modal attention computation unit and a single-modal self-attention computation unit. The cross-modal attention computation unit processes visible light features at multiple scales. and thermal infrared characteristics Perform cross-modal interactive attention calculations to obtain adjusted visible light features. and adjusting thermal infrared characteristics Single-modal self-attention computational units are used to adjust visible light characteristics. and adjusting thermal infrared characteristics Visible light attention weights are obtained by sequentially performing splicing, multilayer perceptron, and sigmoid activation. and thermal infrared attention weights Then, based on the weight of light intensity Visible light characteristics and visible light attention weight Dynamic weighted modulation is performed to obtain calibrated visible light enhancement features. According to light intensity weight Thermal infrared features and visible light attention weight Dynamic weighted modulation is performed to obtain calibrated thermal infrared enhancement features. For calibrating visible light enhancement features and calibration of thermal infrared enhancement features Residual joins are performed to achieve element-wise addition and obtain preliminary fusion and shared features. ; The pixel and channel optimizer includes depthwise separable convolutional units and optimization fusion units. The depthwise separable convolutional units consist of sequentially stacked depthwise convolutional layers, pointwise convolutional layers, GELU activation layers, and residual connections. The optimization fusion unit includes a stitching layer, a softmax activation function, a product layer, and a residual connection layer. Visible light enhancement features are calibrated. and calibration of thermal infrared enhancement features Depthwise convolution, pointwise convolution, and GELU activation are performed separately, and residual connections are made with the corresponding original features to obtain the final visible light enhancement features. Or thermal infrared enhancement features Final visible light enhancement features Or thermal infrared enhancement features After concatenation, softmax activation, and element-wise multiplication with the corresponding original features, element-wise addition is performed to obtain the fused enhanced features. ; The feature fusion module combines features through channel stitching. and enhanced fusion features Feature aggregation is performed, and the aggregated features are projected onto a unified high-dimensional manifold space using 1×1 convolutions. This step eliminates dimensional redundancy caused by multi-branch processing, compresses and filters out the most significant crowd density features, and then performs a nonlinear transformation on the projected features through a GELU activation layer to obtain multimodal fusion features. .

[0015] Preferably, the decoder of the all-weather multimodal fusion crowd counting network receives multimodal fusion features. And utilize the multi-scale visible light features extracted by the encoder. and thermal infrared characteristics Skip connections are made to preserve more spatial details. Then, features are projected to the same dimension using a multilayer perceptron (MLP) and progressively upsampled using bilinear interpolation to align low-resolution semantic information with high-resolution spatial information. Finally, a 1x1 convolutional layer is used to generate the final crowd density map, where each pixel value represents the population density of the region.

[0016] Preferably, the weakly supervised multimodal graph convolutional counting network includes two encoders, one illumination guidance cue generator, one adaptive graph context modeler, one multimodal feature optimizer, and one decoder; Visible light images and thermal infrared / depth images are respectively input into a set of encoders for multi-scale feature extraction, thereby obtaining multi-scale visible light features. and thermal infrared characteristics Visible light characteristics The input is fed into the lighting guidance cue generator, which generates cue information and injects it into the visible light feature. The adaptive graph context modeler uses visible light features based on injected cue information. and thermal infrared characteristics Contextual modeling and fusion are performed to obtain updated visible light features. Update thermal infrared features Context fusion features The multimodal feature optimizer updates the visible light features respectively. Update thermal infrared features Context fusion features Enhancement is performed to obtain optimized and enhanced visible light features. Optimize and enhance thermal infrared characteristics and enhanced context fusion features The decoder optimizes and enhances visible light characteristics. Optimize and enhance thermal infrared characteristics and enhanced context fusion features Decode the data to predict the total number of people and obtain the count result.

[0017] Preferably, the encoder structure and processing procedure in the weakly supervised multimodal graph convolutional counting network are the same as those in the all-weather multimodal fusion crowd counting network.

[0018] Preferably, the illumination-guided cue generator includes a channel-adaptive convolutional layer, a cue generator, a block processing unit, and a multiplication layer; the channel-adaptive convolutional layer includes a two-dimensional convolutional layer Conv2d and a ReLU activation layer, which are adapted to visible light features. By sequentially performing two-dimensional convolution and ReLU activation, visible light features at different scales are processed. Projecting onto a unified prior dimension yields aligned visible light features, achieving feature alignment. The cue generator comprises two sets of 2D convolutional layers (Conv2d), ReLU activation layers, and Sigmoid activation layers. These layers sequentially perform 2D convolution, ReLU activation, and another 2D convolution followed by Sigmoid activation on the aligned visible light features to extract global contextual information of the scene. A block processing unit sequentially performs flat pooling and bilinear interpolation on the global contextual information to obtain a prior map. Finally, a multiplication layer performs element-wise multiplication of the prior map and learnable cue parameters to generate cue information. .

[0019] Preferably, the adaptive graph context modeler includes topological manifold building units, dynamic graph convolutional layers, and adaptive topological aggregation units; the topological manifold building units are based on visible light features at different scales. and thermal infrared characteristics The k-nearest neighbor algorithm is used to construct visible light and thermal infrared topological maps. Dynamic graph convolutional layers perform barrier convolution, residual connections, global average pooling, softmax activation, and weighted summation on the visible light and invisible light topological maps, respectively, to obtain visible light single-modality multi-scale aggregated features and thermal infrared single-modality multi-scale aggregated features. Adaptive topological aggregation units reshape the visible light and thermal infrared single-modality multi-scale aggregated features respectively before concatenation. The concatenated features are then used to calculate the visible light weights using a gating mechanism. and thermal infrared weight The context fusion features are obtained by weighting and summing the two sets of weights and the corresponding features of the original input, and then summing them element by element. The two sets of features after weighted summation are then fused with the context features. Perform residual connections to obtain updated visible light features and updated thermal infrared features The data is then transmitted to the multimodal feature optimizer.

[0020] Preferably, the multimodal feature optimizer includes a deep convolutional unit, a splitting unit, a pointwise convolutional unit, a residual connection layer, and a product layer; the deep convolutional unit includes a deep convolutional layer and a GELU activation layer; the splitting unit includes a channel splitting unit (Split), a pointwise convolutional layer (PW-Conv), and a GELU activation layer; the pointwise convolutional unit includes a pointwise convolutional layer (PW-Conv) and a GELU activation layer; the deep convolutional unit performs deep convolution operations and GELU activation functions on the input features sequentially to extract local context information; the local context information and the original input features are added element-wise in the residual connection layer, and then sequentially passed through the channel difference, pointwise convolution, and GELU activation function set of the splitting unit to obtain interactive features; the interactive features are sequentially passed through the pointwise convolutional layer (PW-Conv) of the pointwise convolutional unit and activated by the GELU activation function, and then residually connected with the original input features to obtain optimized and enhanced visible light features. Optimize and enhance thermal infrared characteristics Or enhance context fusion features .

[0021] Preferably, the decoder structure and processing of the weakly supervised multimodal graph convolutional counting network are the same as those of the decoder of the all-weather multimodal fusion crowd counting network. It receives the enhanced features output by the multimodal feature optimizer, generates a latent crowd density map through operations such as MLP and bilinear interpolation, and then directly outputs the predicted total number of people as the counting result by summing the crowd density map and taking its absolute value.

[0022] Preferably, a composite loss function is used to train the model, which includes Bayesian loss, illumination loss, and cue alignment loss. The composite loss function designed for multimodal crowd counting tasks effectively optimizes the model under both fully supervised and weakly supervised settings, improves prediction accuracy, and ensures the synergistic effect of different modalities under different illumination conditions.

[0023] Preferably, the decoder output of the all-weather multimodal fusion crowd counting network is a density map, which is used to calculate the illumination loss with the real labels. The density map is used to calculate the Bayesian loss, and the total loss is calculated by weighted sum of the Bayesian loss and the illumination loss. The model parameters are optimized based on the total loss. The decoder output of the weakly supervised multimodal graph convolutional counting network is the predicted total number of people, which is used to calculate the cue alignment loss with the real total number of people labels, thereby optimizing the model parameters.

[0024] As can be seen from the above technical solution, compared with the prior art, the present invention discloses an intelligent processing and crowd counting method based on photothermal image enhancement, and constructs a dual-modal computing architecture that adapts to different data annotation granularities, including an all-weather multimodal fusion crowd counting network (AllWeatherNet) and a weakly supervised multimodal graph convolutional counting network (WSGCN), which are two targeted data processing branches. The two branches can run independently or switch and cooperate according to the annotation type of the training data (fully supervised pixel-level annotation or weakly supervised image-level annotation). This invention starts with multimodal feature fusion and weakly supervised learning. First, AllWeatherNet focuses on modal reliability perception and orthogonal geometry decoupling under full supervision, introducing an ambient lighting discriminator that can intelligently perceive the lighting conditions of the shooting environment and adaptively adjust the fusion weights and feature contributions of visible light and thermal infrared modalities to ensure high accuracy and robustness in all-weather scenarios. By designing a multi-scale multimodal fusion module and an adaptive graph context modeler, it can capture multi-scale target information from the device's high-altitude perspective more comprehensively and effectively, and enhance the understanding of crowd distribution, significantly improving counting accuracy in dense and occluded scenarios. Second, WSGCN, under weak supervision, effectively solves the problem of dense labeling dependency through environmental prior injection and topological manifold modeling. This weakly supervised learning framework only requires image-level total number of people labels for training, greatly reducing the dependence on expensive densely labeled data. At the same time, while maintaining competitive counting accuracy, it effectively takes into account the integrity of spatial information, greatly improving the practicality and deployment efficiency of the model. In summary, this invention not only effectively improves the overall accuracy and environmental adaptability of crowd counting, but also significantly reduces application costs through an efficient learning strategy. It achieves high-precision, robust, and low-annotation-cost crowd counting in complex scenarios, providing strong technical support for equipment deployment in fields such as smart cities and security monitoring. It has profound value potential and broad application prospects.

[0025] The beneficial effects of this invention include: (1) Dynamic modal reliability weighting: It can automatically sense the global signal-to-noise ratio of the image acquired by the device, and intelligently generate modal weighting coefficients by quantifying the effective information content of the RGB image. When RGB information is missing (such as dark light, blur) or redundant, its weight is automatically suppressed to ensure that the counting network is always locked on the highest quality feature stream.

[0026] (2) Decoupling of orthogonal geometric features: To address the perspective distortion problem of high-point top view, the horizontal distribution features and vertical scale change features of the crowd are handled by orthogonal interaction strategy in the horizontal and vertical directions respectively, so as to accurately preserve the spatial geometric structure of multi-scale targets during the fusion process.

[0027] (3) Sparse supervised topological reasoning: Under the condition of only the total number of people (sparse labels), by constructing a feature topological manifold, density propagation and reasoning are carried out by utilizing the semantic similarity between pixels, which greatly reduces the dependence on expensive manual dense point annotation, while maintaining the ability to perceive the distribution of the population.

[0028] (4) High robustness in complex scenarios: The model's performance is improved under complex challenges such as occlusion, background interference and perspective effect from the high-altitude view of the device, making the counting results more stable and reliable.

[0029] (5) Spatial-semantic hierarchical refinement: By utilizing a depthwise separable convolutional structure, feature processing is decoupled into two steps: spatial filtering and semantic reconstruction. This utilizes the rich texture semantics of visible light images and combines the thermal radiation positioning information of thermal infrared images, achieving complementary advantages between modalities. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0031] Figure 1 This is a diagram of the overall framework of AllWeatherNet provided in this embodiment of the invention; Figure 2 This is a schematic diagram of the ambient light discrimination device structure provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the multi-scale multimodal fusion module structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the encoder structure provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the cross-dimensional sensing fusion module structure provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the pixel and channel optimizer structure provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the fusion feature module structure provided in an embodiment of the present invention; Figure 8 This is a diagram of the overall framework of WSGCN provided in this embodiment of the invention; Figure 9 This is a schematic diagram of the structure of the illumination guidance prompt generator provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the adaptive graph context modeler structure provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of the multimodal feature optimizer structure provided in an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] This invention discloses an intelligent processing and crowd counting method based on photothermal image enhancement, comprising the following steps: S1: Collect multimodal images of the target scene and annotate the multimodal images to construct a training dataset; S2: Construct a dual-modal computing architecture based on deep learning networks. This architecture includes two parallel processing branches: fully supervised and weakly supervised, for different data annotation granularities. The two branches independently implement crowd counting. The corresponding processing branches are trained and optimized based on images with different annotation types in the training dataset to obtain a crowd counting model. The crowd counting model includes an all-weather multimodal fusion crowd counting network and a weakly supervised multimodal graph convolutional counting network. An all-weather multimodal fusion crowd counting network is trained using images with pixel-level density annotations. The all-weather multimodal fusion crowd counting network with fully supervised processing branches includes an encoder, an ambient light discriminator, a multi-scale multimodal fusion module, and a decoder. The encoder extracts multi-scale multimodal features from the multimodal image, the ambient light discriminator extracts light intensity weights from the multimodal features, the multi-scale multimodal fusion module performs deep fusion of the multi-scale multimodal features and the light intensity weights to generate multimodal fusion features, and the decoder decodes the multimodal fusion features to generate a crowd density map. A weakly supervised multimodal graph convolutional counting network is trained using images containing only image-level total number of people annotations. The weakly supervised processing branch of the weakly supervised multimodal graph convolutional counting network includes an encoder, a lighting guidance cue generator, an adaptive graph context modeler, a multimodal feature optimizer, and a decoder. The encoder extracts multi-scale multimodal features from the multimodal image. The lighting guidance cue generator generates cue information based on the visible light features in the multimodal features and injects it into the visible light features. The adaptive graph context modeler performs context modeling and fusion on the multimodal features of the injected cue information to generate context fusion features. The multimodal feature optimizer enhances the multimodal features of the injected cue information and the context fusion features to obtain enhanced features. The decoder decodes and predicts the output counting result based on the enhanced features. S3: Collect the multimodal images to be counted and the counting requirements, input them into the crowd counting model, and select either an all-weather multimodal fusion crowd counting network or a weakly supervised multimodal graph convolutional counting network to recognize the multimodal images to be counted and obtain the counting results.

[0034] The technical effects of the above solutions are as follows: AllWeatherNet, an all-weather multimodal fusion crowd counting network, is used when the input data has high-quality pixel-level density annotations. It focuses on modal reliability perception and orthogonal geometric decoupling under full supervision, performing pixel-level feature decoupling and reconstruction to achieve high-precision crowd density estimation. Meanwhile, the weakly supervised multimodal graph convolutional counting network (WSGCN) is used when the input data only has image-level total number of people annotations. Through environment prior injection and topological manifold modeling, it effectively solves the problem of dense annotation dependency. The process of inputting multimodal images, using an encoder for feature extraction, using a core fusion / learning module for feature processing and fusion, and using a decoder to generate the final counting result enables crowd counting from fused visible light and thermal infrared images, achieving a complete data closed loop from environment perception and feature enhancement to cross-modal fusion.

[0035] Furthermore, when the counting requirement is high-precision population statistics, an all-weather multimodal fusion population counting network is used for identification and counting; when the counting requirement is low-precision population statistics, a weakly supervised multimodal graph convolutional counting network is used for identification and counting.

[0036] Furthermore, the multimodal images include acquired visible light images and thermal infrared images; the multi-scale multimodal features extracted by the encoder include visible light features and thermal infrared features.

[0037] In one specific embodiment, an all-weather multimodal fusion crowd counting network, such as... Figure 1 As shown, the system includes two encoders, an ambient light discriminator, several multi-scale multimodal fusion modules, and a decoder. Visible light images and thermal infrared images are respectively input to one encoder for multi-scale feature extraction, obtaining multi-scale visible light features and thermal infrared features to form multimodal features. Simultaneously, the visible light image is input to the ambient light discriminator to extract light intensity weights. The visible light features, thermal infrared features, and light intensity weights at each scale are respectively input to one multi-scale multimodal fusion module for deep fusion, obtaining multimodal fusion features at the corresponding scale. The multimodal fusion features at all scales are input to the decoder for decoding to generate a crowd density map.

[0038] The technical effects of the above solution are as follows: AllWeatherNet solves the problem of generating high-precision density maps. The ambient light discriminator (LE) processes the input raw image, analyzes the global signal-to-noise ratio based on the visible light image, and generates illumination confidence weight data as illumination intensity weights, which are responsible for global explicit discrimination. The data flows to the multi-scale multimodal fusion module to macroscopically adjust the overall contribution ratio of RGB and thermal infrared modes. The encoder extracts RGB feature data at multiple scales from the visible light image and thermal infrared feature data at multiple scales from the thermal infrared image. The multi-scale multimodal fusion module uses an orthogonal geometric decoupling mechanism to perform weighted calibration and deep fusion on the RGB feature data, thermal infrared feature data, and illumination intensity weights in the feature dimension, and outputs high-fidelity multimodal fusion features. The decoder uses the multimodal fusion features to decode and generate pixel-level crowd density maps.

[0039] Furthermore, the encoder uses Sven transformers as the backbone network, such as... Figure 4 As shown, it includes a patch partitioning layer and four stages for feature extraction. The first stage for feature extraction includes a linear embedding layer and a main coding layer, the Swing Transformer Block. The remaining feature extraction units include a patch merging layer. Merging and the main encoder: The block layer divides the input image into fixed-size patches and transmits them to the first set of feature extraction units. The linear projection layer of the first set of feature extraction units performs linear embedding on the patches to obtain patch embeddings. The main encoder uses windowed multi-head self-attention (W-MSA) and shifted windowed multi-head self-attention (SW-MSA) mechanisms to capture local and global features in the patch embeddings, obtaining the first-scale feature F1, which is then transmitted to the second set of feature extraction units. The second set of feature extraction units reduces the resolution of the first-scale feature F1 through the image patch merging layer while increasing the number of channels. The main encoder then extracts the second-scale feature F2, which is transmitted to the third set of feature extraction units. The third set of feature extraction units extracts the third-scale feature F3 based on the second-scale feature F2 and transmits it to the fourth set of feature extraction units. The fourth set of feature extraction units extracts the fourth-scale feature F4 based on the third-scale feature F3. The number of channels for features F1, F2, F3, and F4 at different scales are 128, 256, 512, and 1024, respectively. These features reflect image information from local to global perspectives. Visible light images are processed by an encoder to extract multi-scale visible light features. The thermal infrared image is processed by an encoder to extract multi-scale thermal infrared features. The Encoder extracts deep semantic features from images through a multi-layer convolutional neural network (SwinTransformer backbone network), obtaining two-dimensional features at four scales.

[0040] Furthermore, ambient light discriminators such as Figure 2 As shown, it includes a global average pooling layer, a fully connected layer (FC), and processing units; the global average pooling layer extracts two-dimensional features at four scales (visible light features). or thermal infrared characteristics The image is compressed into a one-dimensional global environment fingerprint vector, which highly condenses the overall attributes of the current scene (such as sharpness and contrast distribution). The fully connected layer performs dimensionality reduction mapping on the global environment fingerprint vector to obtain the probability distribution. The processing unit calculates the illumination intensity weight of the visible light image based on the probability distribution and transmits it to the multi-scale multimodal fusion module.

[0041] Furthermore, the fully connected layer (FC) includes two sets of cascaded fully connected operations, so that the module's output is no longer interpreted as a simple day / night classification, but rather as a probability P of the dominant RGB mode in the current scene. d With the dominant probability P of the thermal infrared mode n The soft prediction is expressed as: ; ; Where P represents the activation function after a nonlinear activation function. Feature response intensity after nonlinear activation; F represents the global environment fingerprint vector; P d and P n These represent the probability distributions that are more suitable for using RGB features (high confidence) and the probability distributions that are more suitable for using thermal infrared features (low confidence), respectively. This indicates the first group of fully connected operations; This represents the second set of fully connected operations. The first set of fully connected operations is used for feature dimensionality reduction and extraction, while the second set is used to map to the final class probabilities. This two-layer design can introduce non-linear activation, improving the accuracy of illumination perception.

[0042] Furthermore, the processing unit includes a normalization layer and a non-linear activation function layer. The probability distribution is normalized through the normalization layer, and the light intensity weights are calculated based on the normalized probability distribution using the non-linear activation function Sigmoid; the expression is: ; + =1 ; in, Indicates the weight of light intensity; These are the parameters of the nonlinear activation function; This indicates the probability that a thermal infrared image contains valid information under current ambient lighting conditions, such as in complete darkness. It will approach 1, indicating that the reliability of the thermal infrared mode is much higher than that of the visible light mode at this time; This indicates the probability that a visible light image contains valid information under the current ambient lighting conditions.

[0043] Furthermore, multi-scale multi-modal fusion modules such as Figure 3 As shown, it includes a dimensional cross-modal perception module, a pixel and channel optimizer, and a fusion feature module.

[0044] Dimensional cross-modal sensing module, such as Figure 5 As shown in (a), it includes a cross-modal attention computation unit and a single-modal self-attention computation unit. It utilizes orthogonal decomposition to dynamically weight and modulate features, obtaining preliminary fused shared features, calibrated visible light enhancement features, and calibrated thermal infrared enhancement features. Specifically, the cross-modal attention computation unit processes multi-scale visible light features... and thermal infrared characteristics Lateral cross-modal interactive attention computation is performed along the horizontal distribution domain (width W), utilizing information from high-confidence modes to correct information from low-confidence modes, thereby obtaining adjusted visible light features with consistency across the horizontal distribution. and adjusting thermal infrared characteristics The single-modal self-attention computational unit includes a splicing layer, a multilayer perceptron (MLP), a sigmoid activation function, a product layer, and a residual connection layer, capable of handling multi-scale visible light features. and multi-scale non-visible light features Vertical single-modal self-attention calculations are performed separately in the vertical perspective domain direction (height direction H), preserving the near-size and far-size perspective information of each modality from the vertical viewpoint, and adjusting visible light characteristics. and adjusting thermal infrared characteristics After being stitched together by the stitching layer, the data is input into the MLP for processing to obtain visible light enhancement features and thermal infrared enhancement features. These features are then activated by the Sigmoid activation function to obtain visible light attention weights. and thermal infrared attention weights Light intensity weights are used in the product layer. Visible light characteristics and visible light attention weight Dynamic weighted modulation is performed to obtain calibrated visible light enhancement features. Using light intensity weights Thermal infrared features and visible light attention weight Dynamic weighted modulation is performed to obtain calibrated thermal infrared enhancement features. The calibrated visible light enhancement feature is achieved in the residual connection layer. and calibration of thermal infrared enhancement features Residual joins are performed to achieve element-wise addition and obtain preliminary fusion and shared features. .

[0045] Figure 5 (b) shows the process of lateral cross-modal interactive attention computation, using visible light features. Taking the calculation of cross-modal interaction attention as an example, the expression is: ; ; Where Q, K, and V represent the query, key, and value matrices generated from the input feature mapping, respectively; the subscripts rgb and T represent the visible light and thermal infrared modes, respectively; H and W represent independent operations performed in the height or width directions, respectively. This is a scaling factor used to adjust numerical stability.

[0046] Pixel and channel optimizers such as Figure 6 As shown in (a), a depthwise separable convolutional structure is adopted, including depthwise separable convolutional units and optimized fusion units. The depthwise separable convolutional units are as follows: Figure 6 As shown in (b), the system includes a series of stacked deep convolutional layers (Depthwise Conv), pointwise convolutional layers (Pointwise Conv), GELU activation layers, and residual connections. The optimization fusion unit includes a stitching layer, a softmax activation function, a product layer, and a residual connection layer. The calibrated visible light enhancement features output by the dimensionality cross-modal perception module are also shown. and calibration of thermal infrared enhancement features The inputs are fed into a depthwise convolutional layer (DCV) containing depthwise separable convolutional units. Spatial convolution operations are performed independently on each feature channel to filter out local background noise while preserving human edge information, yielding filtered features. In a pointwise convolutional layer (Pointwise Conv), 1×1 convolutions are used to linearly combine and upscale the filtered features between channels, transforming spatial information into a high-dimensional semantic representation, yielding high-dimensional features. A GELU activation layer performs a non-linear transformation on these high-dimensional features, obtaining transformed features. Finally, a residual connection layer combines these transformed features with the corresponding calibration visible light enhancement features, which are the original input features. Or calibrate thermal infrared enhancement features Element-by-element addition is performed to obtain the final visible light enhancement feature. Or thermal infrared enhancement features This hierarchical collaboration enhances feature representation while ensuring effective gradient propagation; the optimized fusion unit enhances the final visible light features. Or thermal infrared enhancement features The data is stitched together in the stitching layer, then activated by the softmax activation function to obtain fusion weights. In the product layer, these fusion weights are then combined with the calibrated visible light enhancement features. Calibration of thermal infrared enhancement features Element-wise multiplication is performed to obtain visible light intermediate fusion features and thermal infrared intermediate fusion features respectively. Residual connections are then performed on the visible light intermediate fusion features and thermal infrared intermediate fusion features in the residual connection layer to achieve element-wise addition, thus obtaining the fusion enhancement feature. .

[0047] Fusion feature modules such as Figure 7 As shown, channel splicing will initially integrate shared features. and fusion enhancement features Feature aggregation is performed, and the aggregated features are projected onto a unified high-dimensional manifold space using 1×1 convolutions. This step eliminates dimensional redundancy caused by multi-branch processing, compresses and filters out the most significant crowd density features, and then performs a nonlinear transformation on the projected features through a GELU activation layer to obtain multimodal fusion features. .

[0048] The technical effect of the above solution is that the multi-scale multimodal fusion module (MSMFM) is a hierarchical feature interaction structure that efficiently integrates complementary information from visible light and thermal infrared modes. It is particularly good at processing multi-scale features. It adopts a geometric dimension orthogonal decoupling mechanism, deeply fuses the multimodal features extracted by the encoder, and uses the illumination intensity weight of the LE to fuse them through the dimension-aware cross-modal fusion module (DAF) to achieve adaptive adjustment to generate high-quality fused features. After being refined by the pixel and channel optimizer (PCO) and the fusion feature module (FFB), the final output multimodal fused features are obtained. Specifically, DAF leverages the geometric characteristics of overhead scenes to decompose the feature space into an orthogonal horizontal distribution domain (W-axis) and a vertical perspective domain (H-axis). In the horizontal distribution domain, it utilizes the continuity of crowd distribution to enhance the consistency between modalities, while in the vertical perspective domain, it uses perspective projection rules to preserve the scale specificity of each modality. This effectively addresses multi-scale targets and occlusion issues in overhead shots, efficiently captures global contextual semantics, avoids the mixing of irrelevant information, significantly improves fusion accuracy, and directly supports the accurate capture of multi-scale targets. Combined with PCO, it achieves deep fusion from geometric space calibration to semantic feature refinement, effectively solving the feature aliasing problem under dense occlusion. PCO and FFB further optimize the fused feature representation to ensure the expressive power and robustness of the final fused features, and enhance the model's ability to perceive complex details in images through deep network structures.

[0049] Furthermore, the decoder of the all-weather multimodal fusion crowd counting network receives multimodal fusion features. And utilize the multi-scale visible light features extracted by the encoder. Non-visible light characteristics Skip connections are made to preserve more spatial details. Then, features are projected to the same dimension using a multilayer perceptron (MLP) and progressively upsampled using bilinear interpolation to align low-resolution semantic information with high-resolution spatial information. Finally, a 1x1 convolutional layer is used to generate the final crowd density map, where each pixel value represents the population density of the region.

[0050] In one specific embodiment, a weakly supervised multimodal graph convolutional counting network (WSGCN) is as follows: Figure 8 As shown, it includes two encoders, one lighting guidance cue generator, one adaptive graph context modeler, one multimodal feature optimizer, and one decoder; Visible light images and thermal infrared images are respectively input into a set of encoders for multi-scale feature extraction, thereby obtaining multi-scale visible light features. and thermal infrared characteristics Visible light characteristics The input is fed into the lighting guidance cue generator, which generates cue information and injects it into the visible light feature. The adaptive graph context modeler uses visible light features based on injected cue information. and thermal infrared characteristics Perform context modeling and fusion to obtain context fusion features. The multimodal feature optimizer optimizes the visible light features for injecting cue information. Thermal infrared characteristics Context fusion features Enhancement is performed to further improve feature representation and obtain optimized and enhanced visible light features. Optimize and enhance thermal infrared characteristics and enhanced context fusion features The decoder optimizes and enhances visible light characteristics. Enhanced thermal infrared characteristics and enhanced context fusion features Decode the data to predict the total number of people and obtain the count result.

[0051] The technical effects of the above solution are as follows: the weakly supervised multimodal graph convolutional counting network is specifically designed for sparsely labeled scenes. By introducing prior knowledge and topology modeling, it solves the problem of missing location information and addresses the counting problem with only the total number of people labeled. Specifically, the illumination guidance cue generator (PG) extracts prior scene information from RGB features, generates embedded cue information, and injects it into the RGB features of the backbone network. Next, the adaptive graph context modeler (GC) receives the RGB features and thermal infrared features after the cue, transforms the regular image grid data into graph topology data in non-Euclidean space, and infers the distribution relationship of the crowd through multi-scale graph convolution. Then, the multimodal feature optimizer (MO) aggregates the topology features (visible light features injected with cue information) and optimizes them. and thermal infrared characteristics The system performs channel interaction and purification, outputting a global feature vector for regressing the total number of people (including optimized and enhanced visible light features). Optimize and enhance thermal infrared characteristics and enhanced context fusion features Finally, the decoder predicts the total number of people. PG extracts the implicit scene distribution patterns from RGB features, generates environmental prior feature maps, and injects them into the backbone network. This injects prior knowledge based on scene context into the RGB features, explicitly guiding the model to focus on high-frequency areas where crowds may exist. This is equivalent to providing the model with a "pre-defined cognitive context," accelerating feature convergence under weak supervision. GC projects image features onto a non-Euclidean graph topology space. By constructing a correlation graph between feature points, it captures the manifold distribution of crowds in the feature space, thereby inferring local density using topological correlations in the absence of location supervision, compensating for the lack of spatial location supervision. MO performs feature denoising and purification after topological aggregation, further enhancing the complementarity between modalities and improving the robustness of the model in complex backgrounds.

[0052] Furthermore, the encoder structure and processing procedure in the weakly supervised multimodal graph convolutional counting network are the same as those in the airborne all-weather multimodal fusion crowd counting network.

[0053] Furthermore, lighting guidance prompt generators such as Figure 9As shown, it includes a channel-adaptive convolutional layer, a cue generator, a block processing unit, and a multiplication layer; the channel-adaptive convolutional layer includes a two-dimensional convolutional layer Conv2d and an activation function ReLU, which are applied to visible light features. By sequentially performing two-dimensional convolution and ReLU activation, visible light features at different scales are processed. Projecting onto a unified prior dimension yields aligned visible light features, achieving feature alignment. The cue generator comprises two sets of 2D convolutional layers (Conv2d), ReLU activation, and Sigmoid activation. It sequentially performs 2D convolution, ReLU activation, and another 2D convolution followed by Sigmoid activation on the aligned visible light features to extract global contextual information of the scene. The block processing unit includes an average pooling layer and a bilinear interpolation layer. It sequentially performs average pooling and bilinear interpolation on the global context information to obtain the prior graph. ; Product layer on prior graph The system generates prompts by multiplying the learnable prompt parameters from the network element-wise. The lighting-guided prompt generator generates prompts based on visible light characteristics and implicit scene priors.

[0054] Furthermore, the expressions for the cue generator and the product layer in the lighting guidance cue generator are as follows: ; ; in, Represents a priori graph; This indicates a prompt message; Indicates the activation function; This represents a nonlinear mapping function, consisting of two two-dimensional convolutional layers and the ReLU activation function. Indicates learnable prompt parameters; Represents the aligned visible light feature at the i-th scale; In this context, H and W represent height and width, respectively, with a channel count of 1; i represents the hierarchical index of the multi-scale feature, with a value set of {1, 2, 3}; the lighting guidance cue generator automatically captures the common distribution patterns of different scenes (such as sparse / dense, day / night) to generate a prior map. Pointwise modulation is performed with learnable cue parameters to generate cue information. This step is equivalent to "weighting" the original features, suppressing background noise, and highlighting potential crowd areas.

[0055] The technical effect of the above solution is as follows: the PG (Programming Pixel) is responsible for local implicit guidance. It processes deep feature maps and generates lighting cues containing spatial location information, embedding them as cues to correct the feature responses of local shadow or highlight areas in the image. The global illumination probability distribution is used as a learnable cue parameter, injected into the PG as prior environmental data, and used as an auxiliary parameter to guide the PG in generating more accurate local cues. The interaction between the two enables the PG to actively sense and respond to changes in the illumination of the device's shooting environment. Through the learned illumination features, weights or cues are dynamically generated, intelligently guiding the fusion of visible light and thermal infrared modes. This achieves a complementary closed loop from "coarse adjustment of global weights" to "fine refinement of local features," jointly solving the problem of feature failure under complex dynamic illumination. This is more flexible and adaptable than traditional fixed fusion strategies, ensuring the best modal contribution under different illumination conditions, thereby realizing the function of intelligent illumination adaptation.

[0056] Furthermore, adaptive graph context modelers such as Figure 10 As shown, it includes topological manifold building units, dynamic graph convolutional layers, and adaptive topological aggregation units; the topological manifold building units are based on visible light features at different scales. and thermal infrared characteristics The k-nearest neighbor algorithm is used to construct visible light and thermal infrared topological maps at different scales. The dynamic graph convolutional layer includes three sets of dilated convolutional layers and an adaptive feature fusion unit. The adaptive feature fusion unit includes a residual connection layer, an average pooling layer, a softmax activation function, and an aggregation layer. The visible light and thermal infrared topological maps are processed by the dynamic graph convolutional layer to obtain visible light single-modality multi-scale aggregated features and thermal infrared single-modality multi-scale aggregated features, respectively. Three dilated convolutional layers with different dilation rates are used for convolution operations. After the three sets of convolution results are element-wise summed by the residual connection layer, they are then sequentially processed by the global average pooling layer for average pooling, activated by the softmax activation function, and weighted summed by the aggregation layer to obtain the single-modality multi-scale aggregated features. The adaptive topology aggregation unit includes a reshape layer, a stitching layer, a gate layer, a product layer, and a residual connection layer. The reshape layer reshapes the visible light single-mode multi-scale aggregation features and the thermal infrared single-mode multi-scale aggregation features respectively. After stitching by the stitching layer, a gating mechanism is used in the gate layer to calculate the visible light weights separately. and thermal infrared weight In the product layer, the visible light features are respectively... and visible light weight Thermal infrared characteristics and thermal infrared weight Weighted summation is performed to obtain the initial fusion features of visible light and thermal infrared, respectively. Then, the initial fusion features of visible light and thermal infrared are added element-wise through a residual connection layer to obtain the context fusion features. Context fusion features The initial visible light fusion features are then element-wise added together with the residual connection layer to obtain updated visible light features. Context fusion features The initial thermal infrared fusion features are then element-wise added together with the residual linker layer to obtain updated thermal infrared features. The data is then transmitted to a multimodal feature optimizer. Weights generated by global average pooling are used to dynamically fuse the outputs of graph convolutions at different scales. A gate mechanism is introduced to adjust the complementary relationship between visible and non-visible light features in the topological space, thus obtaining contextual fusion features. .

[0057] Furthermore, the expression for constructing a topology graph using topological manifold building blocks is: ; in, This indicates taking the 2-norm; Represents a node With nodes Euclidean distance in feature space (i.e., feature difference); , These represent the i-th and j-th nodes in the constructed topology graph, respectively, corresponding to different spatial positions in the feature graph; , Representing nodes respectively and nodes The corresponding high-dimensional feature vector; during the construction process, the features are regarded as a set of points, and the nodes are connected based on the similarity of the feature content (rather than the physical distance). A feature manifold is constructed to generate a topological graph. This construction method enables topological connections to be established even for population regions that are not physically adjacent but have similar semantics (such as populations that are divided into two parts by occlusion).

[0058] Furthermore, the expression for the dynamic graph convolutional layer is: ; in, Indicates the activation function; This represents the output feature vector of node v in the dilated convolutional layer at the (l+1)th layer; Let v represent the set of neighboring nodes of node v; u represents the set of nodes of node v. Any neighboring node in the list; , represents the degree of node v and its neighbor node u, respectively, used for normalization of feature aggregation; W represents the learnable weight matrix in the dynamic graph convolutional layer. This represents the input feature vector of the neighbor node u at layer l. The dynamic graph convolutional layer implements multi-scale density propagation, performing multi-hop information transfer across the topology to simulate the flow and diffusion of crowd density, thereby capturing contextual information from local individuals to the global group.

[0059] Furthermore, the expression for the dynamic graph convolutional layer is: ; ; ; ; in, Indicates the activation function; This represents the output feature vector of node v in the dilated convolutional layer at the (l+1)th layer; Let v represent the set of neighboring nodes of node v; u represents the set of nodes of node v. Any neighboring node in the list; , represents the degree of node v and its neighbor node u, respectively, used for normalization of feature aggregation; W represents the learnable weight matrix in the dynamic graph convolutional layer. This represents the input feature vector of neighbor node u at layer l; This indicates a global average pooling operation; Indicates the activation function; This represents the global scale context vector obtained after global average pooling (GAP). represents the element-wise sum of the output features from three branches with different dilation rates (Dilation Rate = 1, 2, 3) after passing through the residual connection layer; w represents the multi-scale attention weight vector generated after Softmax normalization, which contains three components, namely... , , ; , , These represent the features output by the branches of three dilated convolutional layers with dilation rates of 1, 2, and 3, respectively; "shortcut" represents the features of the residual connection path. This represents the single-modal multi-scale aggregated features after adaptive fusion. The dynamic graph convolutional layer realizes multi-scale density propagation, performs multi-hop information transmission on the topology, simulates the flow and diffusion of crowd density, and thus captures contextual information from local individuals to the global group.

[0060] The technical effects of the above-mentioned solution are as follows: GC enables the model to learn efficiently even under weak supervision with only image-level labels. Its structural advantages lie in graph structure modeling and multi-scale dilated graph convolution. Graph structure modeling transforms image features into graph structures, which can explicitly capture local and global relationships between features. This is crucial for understanding complex crowd distributions in images, compensating for the lack of spatial information in weakly supervised learning and achieving high efficiency in weakly supervised learning. The multi-scale dilated graph convolution internally contains three branches (dilation rates 1, 2, and 3) graph convolution, which can simultaneously capture contextual information from different receptive fields. Combined with an adaptive feature fusion mechanism, it dynamically adjusts the feature weights at each scale to ensure optimal fusion results, thereby improving the accuracy of multi-scale target capture and the efficiency of weakly supervised learning.

[0061] Furthermore, multimodal feature optimizers such as Figure 11 As shown, it includes depthwise convolutional units, splitting units, pointwise convolutional units, residual connection layers, and multiplication layers, which are used to update visible light features. Update thermal infrared features Context fusion features Processing is performed to obtain optimized and enhanced visible light characteristics. Optimize and enhance thermal infrared characteristics and enhanced context fusion features The deep convolutional unit includes a deep convolutional layer (DWConv) and a GELU activation layer. The splitting unit includes a channel splitting unit (Split), a pointwise convolutional layer (PW-Conv), and a GELU activation layer. The pointwise convolutional unit also includes a pointwise convolutional layer (PW-Conv) and a GELU activation layer. The DWConv layer performs spatial filtering on the features, and the GELU activation layer extracts local context information. The local context information and the original input features are added element-wise in the residual connection layer. Then, the features are sequentially split by the channel splitting unit (Split) in the splitting unit into two paths for feature fusion and further processed. The pointwise convolution of the PW-Conv pointwise convolutional layer and the activation function of GELU are used to enhance the semantic features. The features from one path of addition and fusion are multiplied element-wise with the features from another path that have passed through the pointwise convolutional layer PW-Conv and the GELU activation layer in a product layer. Feature interaction and noise suppression are applied to the two high-dimensional fused features to obtain interactive features. These interactive features are then activated sequentially through the pointwise convolutional layer PW-Conv and the GELU activation function of the pointwise convolutional unit, and then residually connected with the original input features in a residual connection layer for final feature integration and nonlinear transformation, resulting in optimized and enhanced visible light features. Optimize and enhance thermal infrared characteristics Or enhance context fusion features And transmit it to the decoder.

[0062] The technical effect of the above solution is that MO further enhances the fusion features after GC. Its advantage is that it combines depthwise separable convolution and pointwise convolution, and introduces a channel interaction mechanism. Through the synergistic effect of the main and auxiliary branches, it further mines deep semantic information, significantly improves the feature expression ability and generalization ability of the model under weak supervision in complex scenarios, and directly contributes to the efficiency of weak supervision learning and the high robustness of complex scenarios.

[0063] Furthermore, the decoder structure and processing of the weakly supervised multimodal graph convolutional counting network are the same as those of the airborne all-weather multimodal fusion crowd counting network. It receives the enhanced features from the MO output and generates a potential crowd density map through operations such as MLP and bilinear interpolation (although the density map is not directly supervised under weak supervision). Then, by summing the crowd density maps and taking the absolute value, it directly outputs the predicted total number of people as the counting result.

[0064] In one specific embodiment, a composite loss function is used to train the model, which includes Bayesian loss, illumination loss, and cue alignment loss. Designing a composite loss function for multimodal crowd counting tasks effectively optimizes the model under both fully supervised and weakly supervised settings, improves prediction accuracy, and ensures the synergistic effect of different modalities under different illumination conditions.

[0065] Furthermore, the decoder output of the all-weather multimodal fusion crowd counting network is a density map, which is used to calculate the illumination loss (Lbayes, Lill) with the real labels. The density map is used to calculate the Bayesian loss, and the total loss is calculated by weighting the Bayesian loss and the illumination loss. The model parameters are then optimized based on the total loss. The decoder output of the weakly supervised multimodal graph convolutional counting network is the predicted total number of people, which is used to calculate the cue alignment loss (Lcount, LpromptAlign, Lthermal) with the real total number of people labels. The model parameters are then optimized based on the cue alignment loss.

[0066] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0067] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for intelligent processing and crowd counting based on photothermal image enhancement, characterized in that, Includes the following steps: Step 1: Collect multimodal images of the target scene and annotate them to build a training dataset; Step 2: Construct a bimodal computing architecture based on deep learning networks, and train and optimize it using a training dataset to obtain a crowd counting model; the crowd counting model includes an all-weather multimodal fusion crowd counting network and a weakly supervised multimodal graph convolutional counting network; The all-weather multimodal fusion crowd counting network includes an encoder, an ambient light discriminator, a multi-scale multimodal fusion module, and a decoder. The encoder extracts multi-scale multimodal features from the multimodal image, the ambient light discriminator extracts light intensity information from the multimodal features, the multi-scale multimodal fusion module performs deep fusion of the multi-scale multimodal features and light intensity information to generate multimodal fusion features, and the decoder decodes the multimodal fusion features to generate crowd density maps and counting results. The weakly supervised multimodal graph convolutional counting network includes an encoder, an illumination-guided cue generator, an adaptive graph context modeler, a multimodal feature optimizer, and a decoder. The encoder extracts multi-scale multimodal features from the multimodal image. The illumination-guided cue generator generates cue information based on the multimodal features and injects it into the multimodal features. The adaptive graph context modeler performs context modeling and fusion on the multimodal features of the injected cue information to generate updated multimodal features and context fusion features. The multimodal feature optimizer enhances the updated multimodal features and context fusion features to obtain enhanced features. The decoder decodes and predicts the counting results based on the enhanced features. Step 3: Collect the multimodal images to be counted and the counting requirements, input them into the crowd counting model, and select either an all-weather multimodal fusion crowd counting network or a weakly supervised multimodal graph convolutional counting network to recognize the multimodal images to be counted and obtain the counting results.

2. The intelligent processing and crowd counting method based on photothermal image enhancement as described in claim 1, characterized in that, The multimodal images include visible light images and thermal infrared images acquired by the device; the all-weather multimodal fusion crowd counting network includes two sets of encoders, one set of ambient light discriminators, several sets of multi-scale multimodal fusion modules, and one set of decoders; the visible light images and thermal infrared images are respectively input to one set of encoders for multi-scale feature extraction, obtaining multi-scale visible light features and thermal infrared features respectively, while the visible light images are input to the ambient light discriminator to extract light intensity weights; Visible light features, thermal infrared features, and illumination intensity weights at each scale are input into a set of multi-scale multimodal fusion modules for deep fusion to obtain multimodal fusion features at the corresponding scale. Multimodal fusion features at all scales are input into the decoder for decoding, generating crowd density maps and counting results.

3. The intelligent processing and crowd counting method based on photothermal image enhancement as described in claim 2, characterized in that, The encoder uses a Swing transformer as its backbone network, which includes a block layer and four sets of feature extraction units. The first set of feature extraction units includes a linear projection layer and a main coding layer. The remaining feature extraction units include an image patch merging layer and a main encoder. The block layer divides the input image into fixed-size patches and transmits them to the first set of feature extraction units. The linear projection layer of the first set of feature extraction units performs linear embedding on the patch to obtain the patch embedding. The main encoder uses window multi-head self-attention and shift window multi-head self-attention mechanisms to capture local and global features in the patch embedding, obtains the first scale feature F1, and transmits it to the second set of feature extraction units. The second set of feature extraction units reduces the resolution of feature F1 at the first scale through an image patch merging layer, while increasing the number of channels. It then extracts feature F2 at the second scale using the main encoder and transmits it to the third set of feature extraction units. The third set of feature extraction units extracts feature F3 at the third scale based on feature F2 at the second scale and transmits it to the fourth set of feature extraction units. The fourth set of feature extraction units extracts feature F4 at the fourth scale based on feature F3 at the third scale. The visible light image is then processed by the encoder to extract multi-scale visible light features. The thermal infrared image is processed by an encoder to extract multi-scale thermal infrared features. .

4. The intelligent processing and crowd counting method based on photothermal image enhancement as described in claim 3, characterized in that, The ambient illumination discriminator includes a global average pooling layer, a fully connected layer, and a processing unit. The global average pooling layer compresses the extracted features into a global ambient fingerprint vector. The fully connected layer performs dimensionality reduction mapping on the global ambient fingerprint vector to obtain a probability distribution. The processing unit calculates the illumination intensity weights of the visible light image based on the probability distribution and transmits them to the multi-scale multimodal fusion module.

5. The intelligent processing and crowd counting method based on photothermal image enhancement as described in claim 4, characterized in that, The multi-scale multimodal fusion module includes a dimensional cross-modal perception module, a pixel and channel optimizer, and a fusion feature module; The dimensional cross-modal perception module includes a cross-modal attention computation unit and a single-modal self-attention computation unit. The cross-modal attention computation unit processes visible light features at multiple scales. and thermal infrared characteristics Perform cross-modal interactive attention calculations to obtain adjusted visible light features. and adjusting thermal infrared characteristics Single-modal self-attention computational units are used to adjust visible light characteristics. and adjusting thermal infrared characteristics Visible light attention weights are obtained by sequentially performing splicing, multilayer perceptron, and sigmoid activation. and thermal infrared attention weights Then, based on the weight of light intensity Visible light characteristics and visible light attention weight Dynamic weighted modulation is performed to obtain calibrated visible light enhancement features. According to light intensity weight Thermal infrared features and visible light attention weight Dynamic weighted modulation is performed to obtain calibrated thermal infrared enhancement features. For calibrating visible light enhancement features and calibration of thermal infrared enhancement features Residual joins are performed to achieve element-wise addition and obtain preliminary fusion and shared features. ; The pixel and channel optimizer includes a depthwise separable convolutional unit and an optimization fusion unit. The depthwise separable convolutional unit includes a depthwise convolutional layer, a pointwise convolutional layer, a GELU activation layer, and a residual connection layer stacked in sequence. The optimization fusion unit includes a stitching layer, a softmax activation function, a product layer, and a residual connection layer. Calibrate visible light enhancement features and calibration of thermal infrared enhancement features Depthwise convolution, pointwise convolution, and GELU activation are performed separately, and residual connections are made with the corresponding original features to obtain the final visible light enhancement features. Or thermal infrared enhancement features Final visible light enhancement features Or thermal infrared enhancement features After concatenation, softmax activation, and element-wise multiplication with the corresponding original features, element-wise addition is performed to obtain the fused enhanced features. ; The feature fusion module combines features through channel stitching. and enhanced fusion features Feature aggregation is performed, and the aggregated features are projected using a 1×1 convolution. Then, the projected features are nonlinearly transformed using a GELU activation layer to obtain multimodal fusion features. .

6. The intelligent processing and crowd counting method based on photothermal image enhancement as described in claim 3, characterized in that, The decoder of the all-weather multimodal fusion crowd counting network receives multimodal fusion features and utilizes multi-scale visible light features extracted by the encoder. Non-thermal infrared features Skip connections are made, then feature projection is performed through a multilayer perceptron, and bilinear interpolation is used for progressive upsampling. Finally, a 1x1 convolutional layer is used to generate the final crowd density map.

7. The intelligent processing and crowd counting method based on photothermal image enhancement as described in claim 4, characterized in that, The weakly supervised multimodal graph convolutional counting network consists of two encoders, an illumination guidance cue generator, an adaptive graph context modeler, a multimodal feature optimizer, and a decoder; Visible light images and thermal infrared images are respectively input into a set of encoders for multi-scale feature extraction, thereby obtaining multi-scale visible light features. and thermal infrared characteristics ; Visible light characteristics The input is fed into the lighting guidance cue generator, which generates cue information and injects it into the visible light feature. The adaptive graph context modeler uses visible light features based on injected cue information. and thermal infrared characteristics Contextual modeling and fusion are performed to obtain updated visible light features. Update thermal infrared features Context fusion features ; The multimodal feature optimizer updates the visible light features separately. Update thermal infrared features Context fusion features Enhancement is performed to obtain optimized and enhanced visible light features. Optimize and enhance thermal infrared characteristics and enhanced context fusion features ; The decoder optimizes and enhances visible light features. Optimize and enhance thermal infrared characteristics and enhanced context fusion features Decode the data to predict the total number of people and obtain the count result.

8. The intelligent processing and crowd counting method based on photothermal image enhancement as described in claim 7, characterized in that, The illumination-guided cue generator consists of a channel-adaptive convolutional layer, a cue generator, a block processing unit, and a multiplication layer; the channel-adaptive convolutional layer adapts to visible light features. By sequentially performing two-dimensional convolution and ReLU activation, aligned visible light features are obtained. The cue generator sequentially performs 2D convolution, ReLU activation, and 2D convolution followed by Sigmoid activation on aligned visible light features to extract global context information of the scene. The block processing unit sequentially performs flat pooling and bilinear interpolation on the global context information to obtain a prior map. The multiplication layer performs element-wise multiplication of the prior map and the learnable cue parameters to generate cue information. .

9. The intelligent processing and crowd counting method based on photothermal image enhancement as described in claim 7, characterized in that, The adaptive graph context modeler includes topological manifold building units, dynamic graph convolutional layers, and adaptive topological aggregation units; the topological manifold building units are based on visible light features at different scales. and thermal infrared characteristics The k-nearest neighbor algorithm is used to construct visible light topology graphs and thermal infrared topology graphs. Dynamic graph convolutional layers perform barrier convolution, residual connection, global average pooling, softmax activation and weighted summation on the visible light topology graphs and the invisible light topology graphs respectively to obtain visible light single-modality multi-scale aggregated features and thermal infrared single-modality multi-scale aggregated features. The adaptive topology aggregation unit reshapes the visible light single-mode multi-scale aggregated features and the thermal infrared single-mode multi-scale aggregated features separately before stitching them together. The visible light weights are then calculated using a gating mechanism on the stitched features. and thermal infrared weight The context fusion features are obtained by weighting and summing the two sets of weights and the corresponding features of the original input, and then summing them element by element. The two sets of features after weighted summation are then fused with the context features. Perform residual connections to obtain updated visible light features and updated thermal infrared features The data is then transmitted to the multimodal feature optimizer.

10. The intelligent processing and crowd counting method based on photothermal image enhancement as described in claim 7, characterized in that, The multimodal feature optimizer includes deep convolutional units, split units, pointwise convolutional units, residual connection layers, and product layers. The deep convolutional unit includes deep convolutional layers and GELU activation layers. The split unit includes channel splitting units (Split), pointwise convolutional layers (PW-Conv), and GELU activation layers. The pointwise convolutional unit also includes pointwise convolutional layers (PW-Conv) and GELU activation layers. The deep convolutional unit sequentially performs deep convolution operations and GELU activation on the input features to extract local context information. The local context information is then element-wise added to the original input features in the residual connection layer, and then sequentially passed through the channel difference, pointwise convolution, and GELU activation set of the split unit to obtain interactive features. These interactive features are then sequentially passed through the pointwise convolutional layers (PW-Conv) and activated by the GELU activation function of the pointwise convolutional unit, and finally residually connected to the original input features to obtain optimized and enhanced visible light features. Optimize and enhance thermal infrared characteristics Or enhance context fusion features .

Images