Crowd counting model and method based on cross-modal density and brightness adaptation

By introducing cross-modal feature extraction and brightness adaptive adjustment in the population counting model, the problems of insufficient population counting performance and uneven density maps under low light conditions are solved, and more accurate and stable population counting is achieved.

CN118823684BActive Publication Date: 2024-11-15XIAN UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411296948.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-18
Publication Date
2024-11-15
Estimated Expiration
2044-09-18

AI Technical Summary

Technical Problem

The existing cross-modal population counting technology has limited performance under low light conditions and the density map estimates are uneven, resulting in inaccurate counting results.

Method used

Using a population counting model based on cross-modal density and brightness adaptability, the backbone network, FRGV module and prediction regression network are extracted through feature, combined with the CSCA module and BAF module, the feature weight and density prediction weight are adaptively adjusted to generate fine feature maps and density map, adapt to different light and density conditions.

Benefits of technology

It improves the accuracy and robustness of population counting, reduces the impact of density map estimation imbalance, and can obtain accurate population number analysis results under various lighting conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118823684B_ABST
    Figure CN118823684B_ABST
Patent Text Reader

Abstract

The present invention discloses a crowd counting model and method based on cross-modal density and brightness adaptation, which adaptively adjusts the prediction weight through the regional density distribution state to reduce the influence of the imbalance of the prediction density map estimation, ensures that relatively accurate counting results can be obtained in different density areas, and improves the accuracy of crowd counting; dynamically adjusts the fusion ratio of RGB image and infrared image features through brightness, and can adaptively adjust the weights of RGB and infrared image features according to the brightness of the current image, thereby suppressing the influence of RGB features when the brightness is low, so as to improve the crowd counting performance under various lighting conditions, so as to meet the counting requirements under different lighting conditions. The present invention greatly improves the accuracy and robustness of crowd counting, so that it can adapt to various complex environments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image vision technology and relates to real-time counting of high-density crowds in a scene, and specifically to a crowd counting model and method based on cross-modal density and brightness adaptation. Background Art

[0002] In order to prevent potential dangers caused by high crowd density in public places, such as trampling and other malicious incidents, computer vision technology is needed to analyze the number of people in the scene in real time. Existing crowd counting technologies can be divided into two categories: traditional methods and deep learning-based methods. Traditional methods mainly count pedestrians by detecting the whole body or partial features (such as wavelets, HOG, edges, etc.). However, in high-density crowd scenes, this method is difficult to successfully detect all pedestrians due to low resolution and many occlusions. Regression-based methods use foreground features, edge features, texture and gradient features, and learn the mapping relationship from low-level features to the number of people through models such as linear regression and ridge regression.

[0003] In recent years, crowd counting technology based on deep learning has significantly improved performance. The article "Reducing the Dimensionality of Data with Neural Networks" published by Hinton et al. in Science started a boom in this research field; in 2012, Alex Krizhevsky proposed AlexNet in "ImageNet Classification with Deep Convolutional Neural Networks", which attracted widespread attention from the academic community. Deep learning-based methods can automatically extract higher-level and more expressive features through technologies such as convolutional neural networks (CNNs), and show stronger robustness and accuracy when dealing with high-density crowd scenes. Since most traditional methods use features obtained by CNN, different studies have explored how to extract features at different scales.

[0004] In 2016, Zhang proposed adaptive filters and geometric adaptive kernels in "Single-Image Crowd Counting via Multi-Column Convolutional Neural Network" to extract features at different scales, achieving accurate estimation of different crowd densities and perspective effects. In 2018, Li found that the multi-column structure of MCNN did not significantly improve the counting performance, but brought a large computational burden. In "CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes", he proposed the CSRNet model, using a single-column VGG to replace the multi-column front-end network. Without increasing parameters, it effectively captured a wide range of contextual information and significantly improved the counting performance in high-density crowd scenes.

[0005] However, the performance of existing technologies in low-light conditions is still limited, and information that relies solely on RGB images is also affected by changes in lighting. To this end, cross-modal methods utilize information from other data sources such as infrared images to improve counting accuracy under various lighting conditions. In 2021, Liu proposed the IADM method in "Cross-modal Collaborative Representation Learning and a Large-scale RGBT Benchmark for Crowd Counting", attempting to coordinate the representation of RGB and thermal images. In 2022, Zhang designed the CSCA module (channel and space compound attention module, channel and space compound) in "Spatio-channel Attention Blocksfor Cross-modal Crowd Counting", which adaptively aggregates cross-modal features based on spatial and channel attention to enhance fusion efficiency and performance.

[0006] The existing cross-modal feature crowd counting network has the same problem of uneven density map estimation as the traditional network, that is, the predicted density map results are often higher in areas with higher actual crowd density, while the predicted density map results are often lower in areas with lower actual density. This problem will affect the accuracy of the prediction and ultimately produce crowd count analysis results that do not conform to the actual situation. Summary of the invention

[0007] The technical problem solved by the present invention is to provide a crowd counting model and method based on cross-modal density and brightness adaptation, which can be adaptively adjusted based on density level and brightness conditions to improve the accuracy of crowd counting in various scenes.

[0008] The present invention is achieved through the following technical solutions:

[0009] A crowd counting model based on cross-modal density and brightness adaptation, including a feature extraction backbone network, a FRGV module (Feature Rescale on Global Variance) and a prediction regression network;

[0010] The feature extraction backbone network includes a plurality of convolution blocks, and the convolution blocks extract RGB features and infrared features from the input RGB image and infrared image respectively through convolution layers and pooling layers;

[0011] The convolution block is integrated with the CSCA module for fusion processing. It generates the context information of the extracted features through the attention mechanism, and uses the fully connected network and softmax function to splice the RGB features and infrared features, and obtains the fusion features by learning the weight vector;

[0012] A BAF module (Brightness-Aware Fusion) is provided at the end of the CSCA module. The BAF module obtains the brightness weight according to the brightness level of each area of ​​the input image. When the CSCA module obtains the fusion feature, it performs brightness adaptive adjustment on the RGB features and infrared features in the convolution layer based on the brightness weight and passes it to the next convolution block.

[0013] The final CSCA module combines the final RGB features with the infrared features to generate a fused feature map.

[0014] The FRGV module receives a data set and a fused feature map, and obtains the global density variance and the variance of the fused feature map respectively, identifies and divides different density areas in the fused feature map according to the position of the variance of the fused feature map in the global density variance, sets density prediction weights for each area, and generates and outputs a fine feature map;

[0015] The prediction regression network receives the refined feature map passed by the FRGV module and generates a density map of the mapping counts.

[0016] Furthermore, when the brightness of the RGB feature is adaptively adjusted, if the input image is of low brightness, the weight of the RGB feature is reduced and the weight of the infrared feature is increased;

[0017] If the input image is high brightness, the weight of the RGB feature is increased and the weight of the infrared feature is reduced.

[0018] Furthermore, the scaling factor corresponding to the cluster Set to:

[0019] If the density distribution of the region is uniform, the variance is small and the scaling factor has little effect on the weakening of the feature;

[0020] If the density distribution of the region is scattered, the variance is large and the scaling factor weakens the features significantly.

[0021] The present invention also proposes a crowd counting method based on a cross-modal density and brightness adaptive crowd counting model, comprising the following operations:

[0022] S1: Obtain the data set for model training and testing, which includes training set, validation set and test set; each RGB image in the data set has a corresponding infrared image;

[0023] S2: Construct a crowd counting model based on cross-modal density and brightness adaptation;

[0024] S3: Training a crowd counting model based on cross-modal density and brightness adaptation:

[0025] S3.1: Take the images in the training set as input;

[0026] S3.2: The BAF module calculates the grayscale according to the brightness level of the current input image, and obtains the corresponding brightness weight R according to the grayscale value in different areas. a :

[0027] ;

[0028] in, M and N is the height and width of the current input image, i is the image pixel height position, , j is the image pixel width position; I gray (i,j) is the pixel value of the pixel at position (i, j) in the grayscale image of the current input image, λ Adjust the scale for brightness;

[0029] S3.3: The convolution block extracts features from the input RGB image and infrared image through the convolution layer and the pooling layer; and calculates the query value, key value, and value value of the RGB image and the infrared image;

[0030] S3.4: The CSCA module uses the multi-head attention mechanism to exchange the query values ​​in the RGB image and the infrared image, calculates the similarity between the query value and the key value, and uses the Softmax function to obtain the attention weights of the two. , extract the context information C of the feature;

[0031] The CSCA module channel feature aggregation module uses the MLP layer and softmax function to calculate the result based on the multi-head attention mechanism. Get the learning weight vector and , and concatenate the RGB features and infrared features of each layer to obtain the fusion features ;

[0032] ;

[0033] in, is the RGB feature of the Lth layer, is the infrared feature of layer L;

[0034] Each time you fuse features, use the brightness weight R a For each layer of RGB features Perform adaptive weighting:

[0035] ;

[0036] in, is the RGB feature, To fusion features, For infrared characteristics;

[0037] S3.5: The FRGV module uses the K-Means clustering algorithm to divide the entire data set into multiple sub-regions with different variance characteristics. Determine the location of the input image in its cluster and obtain the scaling factor based on the cluster ;

[0038] The FRGV module uses the Sigmoid function to perform nonlinear transformation on the obtained variance value and combines it with the scaling factor Set density prediction weights for each region:

[0039] ;

[0040] in, is the Sigmoid function, is a fine feature map, is the fusion feature map, i is the cluster category corresponding to the processed data, is the kth cluster; is the maximum variance in the data set, is the minimum variance in the data set, is the variance of the feature map;

[0041] S4: In the process of training the cross-modal density and brightness adaptive crowd counting model, the brightness weight and density prediction weight of the model are continuously updated, and the weight file is saved;

[0042] S5: Input the test set image or the real-time image from the camera device into the trained cross-modal density and brightness adaptive crowd counting model for testing, load the saved weight file, and obtain the predicted density map and predicted crowd results.

[0043] Furthermore, the attention weight The acquisition is:

[0044] ;

[0045] in, is the RGB image attention weight, is the attention weight of infrared image; and are the query values ​​of RGB image and infrared image respectively, and They are the transposition of the Key values ​​of the RGB image and the infrared image respectively. is the dimension of the Key vector;

[0046] The extraction of the context information C of the feature is:

[0047] ;

[0048] is the context information of RGB features, is the context information of infrared features; and They are the Value values ​​of RGB image and infrared image respectively.

[0049] Compared with the prior art, the present invention has the following beneficial technical effects:

[0050] The crowd counting model and method based on cross-modal density and brightness adaptation provided by the present invention can perform density and brightness adaptive adjustment; first, the prediction weight is adaptively adjusted through the regional density distribution state to reduce the influence of the imbalance of the predicted density map estimation, to ensure that relatively accurate counting results can be obtained in different density areas, and to improve the accuracy of crowd counting; second, the fusion ratio of RGB image and infrared image features is dynamically adjusted based on brightness, and the weights of RGB and infrared image features can be adaptively adjusted according to the brightness of the current image, thereby suppressing the influence of RGB features when the brightness is low, so as to improve the crowd counting performance under various lighting conditions, to meet the counting needs under different lighting conditions.

[0051] The solution to the problem of density map imbalance in traditional crowd counting algorithms is often to intervene after obtaining the density map. The present invention directly intervenes in the fusion feature map before the regression network, eliminating the influence of manually set hyperparameters and being more adaptable to different scenarios. On the other hand, compared with the density map stage adjustment, it can better avoid the influence of background information on the density map prediction process and obtain more accurate crowd prediction results.

[0052] The crowd counting model and method based on cross-modal density and brightness adaptation provided by the present invention can better integrate the information of RGB and infrared modes through brightness adaptive adjustment, giving full play to the advantages of different modes in crowd counting; in low-brightness environments, infrared images can make up for the shortcomings of RGB images and provide richer information; while in high-brightness environments, RGB images can provide more accurate color information. This cross-modal fusion method enables the present invention to obtain good counting effects under various lighting conditions.

[0053] The present invention adopts an adaptive adjustment mechanism based on density level and brightness conditions, so that the model can dynamically adjust the prediction weight and RGB / infrared feature fusion ratio according to changes in actual scenes; this mechanism greatly improves the accuracy and robustness of crowd counting, enabling it to adapt to various complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of adding a BAF module and a FRGV module based on a cross-modal framework;

[0055] Figure 2 It is a schematic diagram of a high-density crowd counting model based on cross-modality of the present invention;

[0056] Figure 3 It is an RGB image at night;

[0057] Figure 4 For nighttime infrared images;

[0058] Figure 5 for Figure 3 and Figure 4 The fusion feature map of

[0059] Figure 6 is the generated density map;

[0060] Figure 7 Visualization results of four selected lenses with different brightness under different models;

[0061] Figure 8 RGBT-CC counting performance evaluation results. DETAILED DESCRIPTION

[0062] The present invention is further described in detail below in conjunction with the embodiments, which are intended to explain the present invention rather than to limit it.

[0063] For cross-modal crowd counting, the front-end network includes feature extraction and feature fusion modules. The feature extraction backbone extracts modality-specific features from RGB and infrared images through two independent branches. These features are further embedded into the feature fusion module after different convolutional blocks in the modality-specific branches to obtain complementary fusion information. The obtained features will be passed to the back-end regression network to generate an estimated density map; finally, the density map can be mapped to the predicted counts.

[0064] However, in the field of high-density crowd counting, two core challenges need to be addressed: one is the uneven accuracy of density map estimation, and the other is the impact of low-brightness changes on counting results; these two problems often occur at the same time, bringing great difficulties to traditional CNN networks and cross-modal crowd counting.

[0065] Traditional CNN networks and cross-modal counting often have the problem of unbalanced density map estimation when dealing with crowd areas of different density levels. Specifically, for high-density areas, the predicted counts are often overestimated, while for low-density areas they may be underestimated.

[0066] In images with lower brightness, such as night scenes, RGB images cannot provide enough feature information for convolutional neural networks to extract features. Although the cross-modal method uses infrared images to make up for this shortcoming, the brightness changes of RGB images are not fully considered during the feature fusion process, resulting in an excessively large fusion weight of RGB images, which affects the accuracy of crowd density prediction.

[0067] 1. Density adaptive adjustment:

[0068] In order to overcome the problem of uneven density map estimation, the present invention introduces a density prediction weight adjustment mechanism based on density level. The mechanism first identifies and divides different density areas in the input image, and then sets the corresponding density prediction weight according to the distribution state of each area feature; after determining the density prediction weight, the feature map before prediction is adjusted to finally obtain a more accurate estimated density map; in this way, the model of the present invention can more accurately capture the characteristics of people in different density areas and improve the accuracy of overall counting.

[0069] 2. Adaptive brightness adjustment:

[0070] In order to cope with the impact of brightness changes on counting results, the present invention designs a brightness-adaptive cross-modal feature fusion mechanism. This mechanism can adaptively adjust the weights of RGB and infrared image features in the fusion process according to the brightness conditions of the image; when the brightness is low, the weight of infrared features is increased to take advantage of its advantages in dark environments; when the brightness is high, the weight of RGB features is appropriately increased to fully utilize the color information of RGB images.

[0071] Based on density adaptive adjustment and brightness adaptive adjustment mechanism, such as Figure 1 As shown, in the cross-modal framework, the present invention sets a BAF module (based on brightness perception fusion module) in the front-end network to perform brightness adaptive adjustment; and sets a FRGV module (feature redistribution module of global variance) in front of the back-end network to perform density adaptation;

[0072] like Figure 2 As shown, a crowd counting model based on cross-modal density and brightness adaptation is characterized by including a feature extraction backbone network, an FRGV module and a prediction regression network;

[0073] The feature extraction backbone network includes a plurality of convolution blocks, and the convolution blocks extract RGB features and infrared features from the input RGB image and infrared image respectively through convolution layers and pooling layers;

[0074] The convolution block is integrated with the CSCA module for fusion processing. It generates the context information of the extracted features through the attention mechanism, and uses the fully connected network and softmax function to splice the RGB features and infrared features, and obtains the fusion features by learning the weight vector;

[0075] A BAF module is provided at the end of the CSCA module. The BAF module obtains the brightness weight according to the brightness level of each area of ​​the input image. When the CSCA module obtains the fusion feature, it performs brightness adaptive adjustment on the RGB features and infrared features in the convolution layer based on the brightness weight and passes it to the next convolution block.

[0076] The final CSCA module combines the final RGB features with the infrared features to generate a fused feature map.

[0077] The FRGV module receives a data set and a fused feature map, and obtains the global density variance and the variance of the fused feature map respectively, identifies and divides different density areas in the fused feature map according to the position of the variance of the fused feature map in the global density variance, sets density prediction weights for each area, and generates and outputs a fine feature map;

[0078] The prediction regression network receives the refined feature map passed by the FRGV module and generates a density map of the mapping counts.

[0079] Specifically, the present invention integrates the BAF module and the FRGV module with the BL-based prediction network and the CSCA attention module; for the front-end feature extraction network, the present invention uses VGG to extract modal features from RGB images and infrared images, and integrates them with CSCA in all modules.

[0080] CSCA is a multi-head attention module used to improve the process of cross-modal feature fusion. Since the CFA module (channel feature aggregation module) in CSCA may be affected by low-brightness images during the feature fusion process, the BAF module is attached to the end of CSCA for brightness adaptive adjustment to address this situation.

[0081] When performing adaptive brightness adjustment, if the input image is of low brightness, the weight of the RGB feature is reduced and the weight of the infrared feature is increased;

[0082] If the input image is high brightness, the weight of the RGB feature is increased and the weight of the infrared feature is reduced.

[0083] A feature fusion module is set after the final CSCA module. After the fused feature map is generated, it will be sent to the back-end regression network to obtain the estimated density map.

[0084] In order to handle the unbalanced estimation of density maps, the FRGV module identifies and divides different density regions in the fused feature map, sets density prediction weights for each region, and converts the fused feature map into a refined feature map with density prediction weights. If the density distribution in a certain area of ​​the image is uniform, the variance is small, and the scaling factor weakens the feature slightly; if the density distribution in a certain area of ​​the image is scattered, the variance is large, and the scaling factor weakens the feature strongly; by weakening the features to different degrees through density prediction weights, FRGV is expected to optimize the quality of the estimated density map and improve counting performance.

[0085] The crowd counting method based on cross-modal density and brightness adaptation proposed in the present invention includes the following operations:

[0086] Step 1: Obtain a data set for model training and testing, which includes a training set, a validation set, and a test set; each RGB image in the data set has a corresponding infrared image;

[0087] Specifically, the RGBT-CC dataset is used for model training and testing. The RGBT-CC dataset contains 2030 pairs of RGB-T images with a resolution of 640×480, divided into a training set (1030 pairs), a validation set (200 pairs), and a test set (800 pairs). For each pair of RGB images, there is a corresponding infrared image. The scale and dimensionality of the RGBT-CC dataset make it an ideal benchmark for evaluating the performance of high-density crowd counting methods with cross-modal features.

[0088] Step 2: Build a high-density crowd counting model with cross-modal features

[0089] Specifically, the feature extraction backbone network consists of five main modules (Blocks), which effectively extract multi-level features from the input RGB images and heat map images by combining convolutional layers and maximum pooling layers.

[0090] The feature fusion module uses a multi-scale convolution module (MSC module) to process RGB and heat map features respectively, and generates their contextual information through the attention mechanism.

[0091] Specifically, the model calculates Query (Q), Key (K), and Value (V) for each feature type (RGB and infrared). In the multi-head attention mechanism, RGB-Query and infrared-Query are exchanged, the similarity between Q and K is calculated, and the attention weight is obtained using the Softmax function to extract the context information of each feature. With the assistance of the BAF module, the RGB and heat map features are adaptively fused.

[0092] Step 3: Train a high-density crowd counting model with cross-modal features:

[0093] 1) Data input, take the pictures in the training set as input, such as Figure 3 The RGB image shown, and Figure 4 Infrared image shown;

[0094] Specifically, 1030 pairs of RGB-T images in the RGBT_CC training set are used as input.

[0095] 2) BAF (Brightness-Aware Fusion) module brightness weight calculation

[0096] The grayscale is calculated according to the brightness level of the current input image, and the adaptive weight matrix R is obtained by obtaining the grayscale values ​​in different areas. a :

[0097] ;

[0098] Where M and N represent the height and width of the image, i is the pixel height position of the image, , j is the image pixel width position; I gray (i,j) is the pixel value of the pixel at position (i, j) in the grayscale image of the current input image, λ Adjust the scale for brightness; under low brightness conditions, enhance the weight of infrared image features while appropriately reducing the contribution of RGB features; under high brightness conditions, the opposite is true.

[0099] 3) Feature extraction

[0100] The feature extraction network consists of five main modules (Blocks), in which the attention mechanism is added. Each module uses multiple convolutional layers and pooling layers to extract features from the input RGB and heat map. In the feature extraction stage, the Query (Q), Key (K), and Value (V) values ​​of the RGB features and infrared features are calculated:

[0101] ;

[0102] Among them, Q1, K1, V1 correspond to the Query (Q), Key (K) and Value (V) values ​​in the attention mechanism respectively, and W1 is the different weight matrices corresponding to Q1, K1, V1 during the model training process.

[0103] The Q, K and V values ​​in the attention mechanism are obtained by transforming the input data. For example, the input data is a sentence containing multiple words, and each word is represented by a vector. By learning three different transformation matrices to process them, each word can be converted into a corresponding query, key and value, and multiple words are combined into a matrix.

[0104] 4) Feature Fusion

[0105] In the feature fusion module, the multi-head attention mechanism is used to exchange the feature Q in the RGB and heat map, calculate the similarity between Q and K, and use the Softmax function to obtain the attention weights of the two. , thereby extracting the contextual information C of each feature.

[0106] Specifically, the attention weight The acquisition is:

[0107] ;

[0108] in, is the RGB image attention weight, is the attention weight of infrared image; and are the query values ​​of RGB image and infrared image respectively, and They are the transposition of the Key values ​​of the RGB image and the infrared image respectively. is the dimension of the Key vector;

[0109] The extraction of the context information C of the feature is:

[0110] ;

[0111] is the context information of RGB features, is the context information of infrared features; and They are the Value values ​​of RGB image and infrared image respectively.

[0112] The CSCA module channel feature aggregation module uses the MLP layer () and the softmax function to calculate the result based on the multi-head attention mechanism. Get the learning weight vector and , concatenate the RGB and infrared features of each layer and learn the weight vector and Calculate the fusion features of each layer :

[0113] ;

[0114] in, is the RGB feature of the Lth layer, is the infrared feature of layer L;

[0115] When fusing features for the second time, use the weights obtained by BAF For each layer of RGB features Perform adaptive weighting:

[0116] ;

[0117] The obtained fusion feature map is as follows Figure 5 As shown;

[0118] Here is a specific R a and Calculation example:

[0119] R a Calculation: Assuming that the grayscale value obtained in the height and width directions is 77, the numerator is 50, and the setting λ is 1 / 2550; then R a is 0.0196;

[0120] After normalization for:

[0121] ;

[0122] 5) Calculation of FRGV module density prediction weight

[0123] In the FRGV module, the global variance of the entire data set is calculated, and the maximum variance and minimum variance values ​​within the data set and the variance of the feature map are further obtained.

[0124] The K-Means clustering algorithm is used to effectively divide the entire data set into three sub-areas with different variance characteristics; based on the variance of the feature map Determine the location of the input image in its cluster and obtain the scaling factor based on the cluster ;

[0125] Scaling factors corresponding to clusters It is set as follows: if the density distribution of the region is uniform, the variance is small and the scaling factor weakens the feature slightly; if the density distribution of the region is scattered, the variance is large and the scaling factor weakens the feature strongly.

[0126] Calculate the variance of the feature map , and the maximum variance of the global variance and minimum variance Normalize the variance values ​​using the Sigmoid function and combine it with the scaling factor. Set density prediction weights for each region of the fused feature map:

[0127] ;

[0128] Assume that the data to be processed belongs to the first cluster, and the corresponding scaling factor is -0.24, the maximum value is 0.4, the minimum is 0.1, the variance of the feature map that needs to be processed is 0.3, and after normalization, we can get .

[0129] Step 4: During the training of the cross-modal density and brightness adaptive crowd counting model, the weights of the network model are continuously updated and the weight files are saved.

[0130] Step 5: Test the crowd counting model based on cross-modal density and brightness adaptation.

[0131] Specifically, the test set image or the real-time image from the camera device is input into the trained cross-modal high-density crowd counting model for testing, and the saved weight file is loaded to obtain Figure 6 Density plots are shown, along with the mapped predicted population results.

[0132] Compared with the traditional crowd density counting method based on deep learning, the high-density crowd counting model based on cross-modal features of BAF and FRGV modules proposed in the present invention effectively solves the problem of imbalanced density maps and the weight problem of low-light scenes in the process of cross-modal feature fusion, and has significant advantages in improving prediction accuracy.

[0133] The validation effect data support is given below.

[0134] Experimental data: The test results on the RGBT-CC dataset show that the method of the present invention is significantly better than the existing multimodal crowd counting method in terms of mean square error (MSE). Specifically, the performance of the model of the present invention in MSE is improved by 2.52, which fully verifies the superior performance of the present invention under different lighting conditions and complex scenes.

[0135] Figure 7 The visualization results of four selected lenses with different brightness are shown, where (a) and (b) are RGB and infrared inputs, (c) is the ground-true density map, (d) to (f) are the results of implementing the corresponding modules respectively, and (g) is the prediction result of the present invention; the areas with large differences are marked with red frames.

[0136] The results show that for darker shots, the infrared image contributes more to the density map, indicating the advantage of the cross-modal strategy. In (d), the estimated density map with the attention module successfully focuses on the area where the pedestrian is located. Through the designed FRGV and BAF modules, the weights of high / low density areas and feature fusion weights are adjusted from (e) to (g), making the estimated density map more accurately reflect the ground truth. In addition, the estimated counts of all four sample shots are most matched with the estimated density maps of the ground truth labels.

[0137] The performance is evaluated using the Grid Average Mean Absolute Error (GAME) and the Root Mean Squared Error (RMSE). To evaluate the performance at different grid levels, the value of L is set to Specifically, GAME(0) is equal to the normal Mean Absolute Error (MAE), and GAME(1) to GAME(3) are the sum of the MAEs of the image grid.

[0138] pass Figure 5 The RGBT-CC counting performance evaluation results shown in the figure show the following observations:

[0139] (1) The proposed method achieves the best GAME(0) and RMSE among all reference methods. This observation indicates that the proposed FRGV module successfully solves the problem of unbalanced density map estimation before the regression process. In addition, BAF can potentially enhance the quality of feature fusion.

[0140] (2) For any network, the cross-modal version performs better than the unimodal version, which clearly demonstrates that the cross-modal strategy can improve the accuracy of crowd counting.

[0141] (3) The performance of all networks is improved by integrating the multi-head attention mechanism. For some old-fashioned methods, such as MCNN and CSRNet, adapting the attention module can significantly improve the corresponding RMSE. The excellent effect of the attention mechanism in crowd counting makes its optimization a direction worthy of further exploration.

[0142] In summary, the existing crowd density counting models are often less robust to changes in lighting conditions and are difficult to work stably under different brightness conditions. The present invention introduces a brightness perception module that can adaptively adjust the feature extraction and fusion process according to the brightness changes of the input image, thereby effectively improving the adaptability of the model under different lighting conditions. Figure 7 , Figure 8 The experimental results shown indicate that the present invention can still maintain a high counting accuracy rate in scenes with poor lighting conditions.

[0143] When processing image features, existing methods often face the problem of unbalanced density estimation when estimating the accuracy of crowd areas at different density levels. The present invention uses the FRGV module to perform global feature redistribution on the feature map, dynamically adjusts the feature value according to the variance of the feature map, and adaptively adjusts the prediction weight to obtain relatively accurate counting results, thereby improving the stability and robustness of the model.

[0144] The above embodiments are preferred examples for implementing the present invention, and the present invention is not limited to the above embodiments. Any non-essential additions and substitutions made by those skilled in the art based on the technical features of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A crowd counting method based on a cross-modal density and brightness adaptive crowd counting model, characterized in that: The following operations are included: S1: Obtain the data set for model training and testing, which includes training set, validation set and test set; each RGB image in the data set has a corresponding infrared image; S2: Construct a crowd counting model based on cross-modal density and brightness adaptation; That Includes feature extraction backbone network, FRGV module and prediction regression network; The feature extraction backbone network includes a plurality of convolution blocks, and the convolution blocks extract RGB features and infrared features from the input RGB image and infrared image respectively through convolution layers and pooling layers; The convolution block is integrated with the CSCA module for fusion processing. It generates the context information of the extracted features through the attention mechanism, and uses the fully connected network and softmax function to splice the RGB features and infrared features, and obtains the fusion features by learning the weight vector; A BAF module is provided at the end of the CSCA module, which obtains the brightness weight according to the brightness level of each area of ​​the input image; When the CSCA module obtains the fusion features, it adaptively adjusts the brightness of the RGB features and infrared features in the convolution layer based on the brightness weight and passes them to the next convolution block; The final CSCA module combines the final RGB features with the infrared features to generate a fused feature map. The FRGV module receives a data set and a fused feature map, and obtains the global density variance and the variance of the fused feature map respectively, identifies and divides different density areas in the fused feature map according to the position of the variance of the fused feature map in the global density variance, sets density prediction weights for each area, and generates and outputs a fine feature map; The prediction regression network receives the fine feature map transmitted by the FRGV module and generates a density map of the mapping counts; S3: Training a crowd counting model based on cross-modal density and brightness adaptation: S3.1: Take the images in the training set as input; S3.2: The BAF module calculates the grayscale according to the brightness level of the current input image, and obtains the corresponding brightness weight according to the grayscale value in different areas. : in, M and N is the height and width of the current input image, is the image pixel height position, is the image pixel width position; I gray (i,j) is the position in the grayscale image of the current input image (i,j) The pixel value of the pixel, λ Adjust the scale for brightness; S3.3: The convolution block extracts features from the input RGB image and infrared image through the convolution layer and the pooling layer; and calculates the query value, key value, and value value of the RGB image and the infrared image; S3.4: The CSCA module uses the multi-head attention mechanism to exchange the query values ​​in the RGB image and the infrared image, calculates the similarity between the query value and the key value, and uses the Softmax function to obtain the attention weights of the two. , extract the context information C of the feature; The CSCA module channel feature aggregation module uses the MLP layer and softmax function to calculate the result based on the multi-head attention mechanism. Get the learning weight vector And the RGB features and infrared features of each layer are spliced ​​to obtain the fusion features of the layer in, For the The RGB features of the layer, For the Infrared signature of the layer; Each time you fuse features, use the brightness weight Adaptively weight each layer of RGB features: in, is the adaptively weighted RGB feature, is the fusion feature of this layer, is the infrared characteristic of the layer; S3.5: The FRGV module uses the K-Means clustering algorithm to divide the entire data set into multiple sub-regions with different variance characteristics. Determine the location of the input image in its cluster and obtain the scaling factor based on the cluster The FRGV module uses the Sigmoid function to perform nonlinear transformation on the obtained variance value and combines it with the scaling factor Set density prediction weights for each region: in, is the Sigmoid function, is a fine feature map, is the fusion feature map, i is the cluster category corresponding to the processed data, is the kth cluster; is the maximum variance in the data set, is the minimum variance in the data set, is the variance of the feature map; S4: in the process of training the crowd counting model based on cross-modal density and brightness adaptation, the brightness weight and density prediction weight of the model are continuously updated, and the weight file is saved; S5: Input the test set image or the real-time image from the camera device into the trained cross-modal density and brightness adaptive crowd counting model for testing, load the saved weight file, and obtain the predicted density map and predicted crowd results.

2. The crowd counting method based on the cross-modal density and brightness adaptive crowd counting model as claimed in claim 1, characterized in that: The attention weight The acquisition is: Among them, Attention RGB is the RGB image attention weight, Attention T is the infrared image attention weight; are the query values ​​of RGB image and infrared image respectively, They are the transposition of the Key values ​​of the RGB image and the infrared image respectively. is the dimension of the Key vector; The extraction of the context information C of the feature is: C RGB is the context information of RGB features, C T is the context information of infrared features; They are the Value values ​​of RGB image and infrared image respectively.

3. The crowd counting method based on the cross-modal density and brightness adaptive crowd counting model as claimed in claim 1, characterized in that: When performing adaptive brightness adjustment, if the input image is of low brightness, the weight of the RGB feature is reduced and the weight of the infrared feature is increased; If the input image is high brightness, the weight of the RGB feature is increased and the weight of the infrared feature is reduced.

4. The crowd counting method based on the cross-modal density and brightness adaptive crowd counting model as claimed in claim 1, characterized in that: The scaling factor corresponding to the cluster Set to: If the density distribution of the region is uniform, the variance is small and the scaling factor has little effect on the weakening of the feature; If the density distribution of the region is scattered, the variance is large and the scaling factor weakens the features significantly.

5. The crowd counting method based on the cross-modal density and brightness adaptive crowd counting model as claimed in claim 1, characterized in that: The convolutional block uses VGG to extract modality features from RGB images and infrared images.