Cross-modal crowd counting method based on semantic guidance multi-level fusion

Abstract

The invention discloses a cross-modal crowd counting method based on semantic guidance multi-level fusion, and aims to solve the problems of low accuracy and poor robustness of a multi-modal crowd counting method. The method is mainly composed of an information interaction module and a semantic guidance fusion module; wherein the information interaction module extracts a corresponding weight coefficient for each layer of extraction features to update so as to realize a feedback mechanism, and the semantic guidance fusion module is mainly used for fusing high-level features extracted by the semantic expansion module with modal features of corresponding levels. According to the method, timely feedback of extracted features of each layer is realized through an information interaction module and a semantic guidance fusion module, and meanwhile, the overall understanding ability of crowd scenes is improved through advanced semantic information, so that a high-quality crowd density map is generated, and high-precision crowd counting is realized.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and computer vision, and in particular relates to a cross-modal crowd counting method based on semantic-guided multi-level fusion. Background Art

[0002] Crowd counting is a method of calculating or estimating the number of individuals in an image using computer vision. The core purpose of this technology is to monitor and analyze the gathering trend of crowds and provide real-time crowd distribution information for city managers and planners of large-scale events. In recent years, crowd counting technology has made significant progress, and the introduction of deep learning models has greatly improved the accuracy of predictions. Traditional technical means are mainly divided into two categories: detection-based and regression-based methods. However, traditional methods are limited by high-density scenes, and the calculation process is relatively complicated for crowd density in complex scenes. Since density maps can not only reflect the spatial distribution information of crowds, but also improve counting accuracy, regression based on density maps has gradually become a popular classification, and most recent crowd counting models are based on density estimation methods.

[0003] From a macro perspective, most methods adopt a fully supervised approach, that is, all training images are marked with point-level annotations, and the number of people in the regression density map is compared with the true value. Some work reduces the annotation burden from a more economical perspective and proposes many weakly supervised methods that only rely on the total number of crowd annotations. Fully supervised models usually have better generalization capabilities due to the comprehensive and diverse training data, and can adapt to different test environments and conditions. However, weakly supervised models cannot achieve the same counting accuracy as fully supervised models due to the lack of accurate individual location information. On the one hand, the above-mentioned unimodal crowd counting may lead to insufficient counting accuracy in complex scenes (such as lighting changes, occlusions) due to information utilization limitations. On the other hand, unimodal models may have a strong dependence on specific conditions (such as crowd density, viewing angle, etc.), while multimodal models can reduce this dependence through cross-modal collaborative learning and improve the applicability of the model.

[0004] In scenes with occlusion and complex background, multimodal models can use complementary information between different modalities to improve the accuracy of counting. For example, RGB images may not be able to identify pedestrians in some areas due to occlusion, but thermal images can provide additional information to assist in recognition. For multimodal tasks, it is crucial to effectively fuse the information of RGB and another modality. There have been many works devoted to this work, such as progressive complementary fusion, dynamic filter fusion, coarse-to-fine fusion, etc. Recently, Liu et al. proposed a benchmark for the RGB-T crowd counting dataset and a cross-modal collaborative representation learning framework. The framework diagram of this method is shown in the figure below. Figure 1As shown. However, most of the current methods perform information fusion at a specific network level, which may fail to fully utilize the potential value of deep semantic information, while semantic-level features can achieve further understanding of crowd scenes. In addition, most fusion schemes improve performance in terms of modal fusion, while there is still room for improvement in information exchange and complementarity between modalities and the impact of modal image quality. The above two reasons lead to low accuracy of crowd counting. Multimodal crowd counting better estimates crowd density and distribution by fusing data from different modalities. On the one hand, existing cross-modal crowd counting methods ignore the value of high-level semantic information. The fusion of semantic information helps the model resist noise and interference, and further improves the consistency and complementarity of cross-modal features. Existing methods often have large errors when there are large changes in illumination, severe occlusion, or complex backgrounds. On the other hand, most methods have room for improvement in the information exchange and complementarity and feedback mechanisms between features. Due to the ignoring of the feasibility of the feedback mechanism, the fusion effect is always determined by the input of a single modality, resulting in a low-precision result. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a cross-modal crowd counting method based on semantic-guided multi-level fusion. The information interaction module enriches the shared representation between modalities and provides useful feedback to the corresponding feature extractor to promote modal feature learning. This method further guides with semantic information, further enriches the expressive power of features through feature decoding and aggregation strategies, and finally generates accurate crowd counts.

[0006] Technical solution: A cross-modal crowd counting method based on semantic-guided multi-level fusion of the present invention performs the following steps 1 to 7 for a crowd image to complete the crowd counting training process in the image:

[0007] Step 1: Select a color image and a thermal image from the preprocessed images, and use the ResNet backbone network to extract features. The first layer of features extracted is F t 1 , go to step 2;

[0008] Step 2: Extract the first layer of features F t 1 Enter the information interaction module and update the new features of the two modes as follows: F t 2 , use it as the second layer input of the ResNet backbone, repeat steps 1 to 2 until the first four layers of features are defined as and F t i(i∈{1,2,3,4}), go to step 3;

[0009] Step 3: Separately analyze the color image features Thermal imaging feature F t 4 Execute the semantic expansion module to generate corresponding more comprehensive semantic information RGB features T Features F t sem , go to step 4;

[0010] Step 4: The second, third, and fourth layer features and semantic features of each modality enter the semantic guided fusion module to obtain modality fusion features and F t h , go to step 5;

[0011] Step 5: Combine the modal features from step 4 and F t h Feed it to the regression module to generate the population distribution density map to produce the final population value, and then go to step 6;

[0012] Step 6: If the specified number of training rounds is reached, proceed to step 7, otherwise continue training and return to step 1;

[0013] Step 7: End, completing the crowd counting training in the image.

[0014] Furthermore, the images in step 1 are all from a large-scale benchmark crowd counting dataset, which contains 2,030 pairs of RGB thermal images and 138,389 annotated pedestrians. The dataset is a large number of color thermal images taken by photothermal cameras in various scenes (e.g., shopping malls, streets, playgrounds, train stations, subway stations, etc.). RGB images are composed of three primary colors: red, green, and blue, and contain three channels, each corresponding to a primary color, and thermal imaging has only a single channel. Before training, the dataset is preprocessed, the image size is adjusted to 640*480, and the image is randomly flipped horizontally with zero padding. Consistent with the above, the present invention takes the input of a color image and a thermal imaging sample as an example to introduce the working principle of the present invention during the training process.

[0015] Preferably, in step 1 of the present invention, a dual-stream extraction architecture is used, and a ResNet backbone is used to extract features. The gradient vanishing and gradient exploding problems in deep network training are solved by introducing residual learning. The backbone can effectively capture image features from different scales and enhance the representation ability of the network. First, the first layer of features of the modality are extracted as F t 1 .

[0016] Preferably, the extracted features of the modality in step 2 enter the information interaction module, which can improve the features extracted at each layer to establish a stronger correlation between the two modalities, so that the output of each layer incorporates the contextual information of the previous layer. Such a hierarchical structure helps the model capture more complex and hierarchical scene structures.

[0017] Furthermore, the specific steps of step 2 are as follows:

[0018] Step 2-1: For features and F t i Three independent 1×1 convolution operations are applied to reduce the dimensionality, producing two mapping features {Q r ,K r ,V r} and {Q t ,K t ,V t}, go to step 2-2;

[0019] Step 2-2: For the RGB modality, query Q from the thermal modality t With key K r Perform matrix multiplication to calculate relative attention. The same is true for the thermal mode, and proceed to step 2-3.

[0020] Step 2-3: For the two relative attentions calculated in the previous step, perform softmax layer normalization to generate relative attention weights W rt and W tr , proceed to step 2-4;

[0021] Step 2-4: Use two attention weights to calculate the value V of the modality r and V r Perform weighted summation to obtain the reweighted features and Go to step 2-5;

[0022] Step 2-5: Perform element-by-element multiplication of the weight and the value of the original modality to obtain the modality-enhanced features and Go to step 2-6;

[0023] Step 2-6: Introduce learning parameters alpha_rgb and alpha_t to perform element-by-element multiplication with the enhanced RGB and T features to filter the features, and at the same time multiply them with the original RGB and T features and F t i Perform residual connection to accelerate network convergence and obtain fused features and Go to step 2-6;

[0024] Step 2-7: Perform global average pooling on the obtained features, and concatenate them in the channel dimension to obtain the importance u of the feature response on the channel, and then go to step 2-7;

[0025] Step 2-8: Apply convolution operation to the importance and combine with and Perform element-wise multiplication to obtain the expression of RGB features and thermal features for strengthening bidirectional modality enhancement and Go to step 2-9;

[0026] Step 2-9: Re-weight the key channel information of feature expression and input it into the feature map to achieve more robust multimodal features and proceed to the subsequent steps.

[0027] Furthermore, the attention weight W rt and W tr Get the following formula:

[0028]

[0029] Where φ(*) represents the softmax operation, represents matrix multiplication; the characteristic formula after modal enhancement is as follows:

[0030]

[0031] Furthermore, the formula for introducing the learning parameters alpha_rgb and alpha_t and performing element-by-element multiplication with the enhanced RGB and T features is expressed as follows:

[0032]

[0033] Where ⊙ represents element-by-element multiplication;

[0034] The importance formula of the feature response on the channel dimension is as follows:

[0035]

[0036] Among them, Concat represents the cascade operation, and Avg represents the global average pooling operation.

[0037] Furthermore, a convolution operation is applied to the importance and combined with and Perform element-wise multiplication to obtain the expression of RGB features and thermal features for strengthening bidirectional modality enhancement and The formula is as follows:

[0038]

[0039] Among them, FC represents the fully connected layer operation, and the formula for re-weighting using residual connection and initial modality features is as follows:

[0040]

[0041] Furthermore, the semantic expansion module is further applied to the high-level features extracted from the modality in step 3. The module contains four parallel branches to expand the receptive field and an adaptive average pooling branch to retain the original information as much as possible. High-level features usually contain more abstract and global information. Further semantic extraction of these features can enhance the model's ability to express crowd characteristics under different scenes and conditions, so that the model can better understand and judge complex visual content. The module formula is expressed as follows:

[0042]

[0043] in is the kernel size k×k and the dilation rate d j =2×j dilated convolution, avg_pool(·) is adaptive average pooling, and j represents the branch layer.

[0044] Furthermore, in step 4, the modal features extracted in step 2 and the high-level semantic information in step 3 are fed into the semantic guidance fusion module. The semantic enhancement at a high level makes the model more accurate in estimating the number of people and reduces errors. At the same time, for multimodal learning, by performing semantic fusion at a high level, the information from different sources can be effectively integrated, improving the consistency and complementarity of modal features. The specific steps are as follows:

[0045] Step 4-1: Taking the second, third and fourth layer features extracted from the RGB image as an example, scale feature fusion is performed on each layer, and the number of feature channels is changed using the Relu activation function and 3×3 convolution, and then proceed to step 4-2;

[0046] Step 4-2: Use depth convolution with different expansion rates to capture the multi-scale information of the above features, thereby producing Go to step 4-3;

[0047] Step 4-3: Taking RGB features as an example, After upsampling and Perform element-by-element multiplication to obtain fused features Enter 4-4;

[0048] Step 4-4: Combine the semantic information of the two modalities With the corresponding layered fusion features Further aggregation is performed to obtain the final fusion result and proceed to the subsequent steps.

[0049] Furthermore, the Relu activation function and the depth convolution formula are as follows:

[0050]

[0051] Among them, CH j (·) represents a 3×3 depthwise separable convolution with a dilation rate of j;

[0052] The formula for gradually fusing the second, third and fourth layer features is as follows:

[0053]

[0054] Where Conv is a 3×3 convolutional layer with Relu activation function, Up represents upsampling, and C represents the concatenation operation.

[0055] Furthermore, the semantic information of the two modalities is With the corresponding layered fusion features The further aggregation formula is as follows:

[0056]

[0057] Among them, Conv is 1×1 convolution, DConv is transposed convolution, and Up is bilinear upsampling.

[0058] The present invention also discloses a testing method for verifying a crowd counting method of a multimodal hierarchical progressive interactive network, comprising the following steps:

[0059] Step 1: Input the image and true value in the test data and go to step 2;

[0060] Step 2: Use the model weights obtained through training to extract and fuse features of the dataset image input in step 1, and proceed to step 3;

[0061] Step 3: Output the density map in step 2, sum the pixels of the estimated crowd density map to get the estimated number of people and compare it with the actual value, and then go to step 4;

[0062] Step 4: Calculate the absolute error and square error between the estimated number of people and the true value annotation, and proceed to step 5;

[0063] Step 5: End.

[0064] Preferably, Bayesian loss is used as the loss function in step 4 of the test process, and the density map generated by the prediction is vectorized and converted into a one-dimensional vector. Subsequently, this vector is element-wise multiplied with the corresponding probability distribution to obtain the final predicted count. On this basis, the present invention quantifies the accuracy of the prediction by calculating the absolute error between the predicted result and the actual sample label value, thereby achieving an accurate evaluation of the model performance.

[0065] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0066] Most existing crowd counting methods fail to make full use of the semantic information of the modalities when performing multimodal learning, lack a panoramic understanding of the image as a whole, and the model has relatively large noise and interference, resulting in low counting efficiency. In response to the above problems, the present invention proposes a cross-modal crowd counting method based on semantic-guided multi-level fusion, which learns and distinguishes the feature representations of the two modalities. The high-level semantic fusion effectively reduces the amount of data that needs to be processed, increases the robustness to lighting, occlusion and background problems, and makes the model more accurate in estimating the number of people and reduces errors.

[0067] The evaluation indicators used in the present invention are root mean square error (RMSE) and grid mean absolute error (GAME). The smaller the values ​​of these two indicators, the higher the accuracy. Experiments show that the GAME[0] of the HDFNet method is 22.36, GAME[1] is 27.79, GAME[2] is 33.68, GAME[3] is 42.48, and the RMSE is 33.93; the GAME[0] of the BL+IADM method is 15.61, GAME[1] is 19.95, GAME[2] is 24.69, GAME[3] is 32.89, and the RMSE is 28.19;

[0068] The GAME[0] of the DHRNet method is 11.54, GAME[1] is 14.67, GAME[2] is 18.16, GAME[3] is 23.64, and the RMSE is 20.62. The GAME[0] of the present invention is 11.38, GAME[1] is 14.52, GAME[2] is 18.59, GAME[3] is 26.55, and the RMSE is 19.23. This proves that the present method can improve the accuracy compared with other multimodal counting methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a comparison diagram of the conventional method and the training phase flow chart of the present invention.

[0070] Figure 2 This is a network framework diagram of the present invention.

[0071] Figure 3 This is the framework diagram of the information interaction module, which mainly updates the modality-specific features under the interaction weights between modalities and avoids the loss of original information through residual connections.

[0072] Figure 4 This is the framework diagram of the semantic expansion module, which mainly obtains more comprehensive multi-scale feature information through the connection of five branches.

[0073] Figure 5 This is a diagram of the semantic-guided fusion framework, which mainly aggregates features at different levels through the guidance of semantic information. DETAILED DESCRIPTION

[0074] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0075] Embodiment 1:

[0076] In this experimental example, the present invention verifies the effectiveness and practicality of the present invention in real application scenarios through a series of experiments. The present invention selects a data set widely recognized by the industry as the experimental basis, which includes 4060 photos taken by photothermal cameras in various scenes (for example, shopping malls, streets, amusement parks, train stations, subway stations, etc.). In order to enhance the generalization ability of the model, all images have been zero-filled and randomly horizontally flipped, and random cropping has been implemented to prevent the model from overfitting. In the experimental setting, the present invention uses the Adam optimizer and adjusts the appropriate hyperparameters for it. In order to ensure the fairness of the comparison, the present invention applies a unified parameter configuration to all models and performs consistent processing operations on the image input.

[0077] The training process follows these steps:

[0078] 1. At the beginning of each training cycle, images equal to the batch size are randomly sampled from the training set, and the model is forward propagated to generate predictions and calculate the error with the actual ground truth through the loss function.

[0079] 2. The back-propagation algorithm is executed to calculate the error gradients, and based on these gradients, the optimizer adjusts the weights and bias parameters of the model to minimize the loss.

[0080] 3. After each training cycle, the model performance is evaluated. If the predetermined test conditions are met, the test is performed and the model parameters with the best performance are saved. If the conditions are not met, the training continues until the preset maximum number of training cycles is reached.

[0081] The results of the present invention and the comparative model are shown in Table 1 below:

[0082] Table 1: Experimental results of the technology proposed by the present invention and other technologies

[0083]

[0084] After verification by experimental indicators, the method proposed in this invention has achieved improvements in accuracy compared with other counting methods, highlighting its robustness and practical value in a variety of practical application scenarios.

[0085] Embodiment 2:

[0086] This embodiment will introduce an applicable scenario of the present invention:

[0087] With the acceleration of urbanization, urban public security management faces increasingly severe challenges. Especially during peak hours or large-scale events, security monitoring and traffic management in crowded areas are particularly important. In response to this demand, the present invention proposes a crowd counting method based on multimodal information fusion to improve the accuracy and robustness of counting. This embodiment focuses on a large city square, which is a place for citizens to relax and hold various public activities. Multiple high-definition optical cameras and infrared thermal imaging cameras are installed around the square to monitor crowd dynamics in real time.

[0088] System Configuration:

[0089] 1. Multimodal data acquisition unit: High-definition cameras and infrared thermal imaging cameras deployed around the square have the ability to see at night and penetrate smoke, ensuring stable operation under various lighting and weather conditions.

[0090] 2. Data transmission network: Use high-speed wireless communication network to ensure real-time data transmission to the central processing unit.

[0091] 3. Central processing unit: equipped with a high-performance computing server, installed with the multimodal crowd counting software of the present invention, responsible for processing the data from each camera.

[0092] Workflow:

[0093] 1. Real-time data collection: HD cameras and infrared thermal imaging cameras synchronously capture crowd images.

[0094] 2. Image preprocessing: perform preprocessing operations such as denoising, contrast adjustment, and size standardization on the image.

[0095] 3. Multimodal feature extraction: Using the algorithm of the present invention, color and texture features are extracted from color images, and temperature distribution features are extracted from thermal imaging images.

[0096] 4. Feature-level fusion: Through the unique feature fusion strategy of the present invention, the features of the two modalities are combined to enhance the expressiveness of crowd characteristics.

[0097] 5. Crowd counting and density estimation: Based on the fused features, crowd counting and density estimation are performed to generate a crowd distribution density map.

[0098] 6. Dynamic analysis and decision support: The system analyzes crowd flow trends, predicts possible crowded areas, and provides decision support for urban management.

[0099] 7. Real-time feedback and emergency response: The system will feed back the analysis results to the monitoring center in real time. Once it detects that the crowd density exceeds the safety threshold, the emergency plan will be immediately activated.

Claims

1. A cross-modal crowd counting method based on semantically guided multi-level fusion, characterized in that: For the crowd image, perform the following steps 1 to 7 to complete the crowd counting training process in the image: Step 1: Select a color image and a thermal image from the preprocessed images, and use the ResNet backbone network to extract features. The first layer of features extracted is F t 1 , go to step 2; Step 2: Extract the first layer of features F t 1 Enter the information interaction module and update the new features of the two modes as follows: F t 2 , use it as the second layer input of the ResNet backbone, repeat steps 1 to 2 until the first four layers of features are defined as and F t i (i∈{1,2,3,4}), go to step 3; Step 3: Separately analyze the color image features Thermal imaging feature F t 4 Execute the semantic expansion module to generate corresponding more comprehensive semantic information color features T Features F t sem , go to step 4; Step 4: The second, third, and fourth layer features and semantic features of each modality enter the semantic guided fusion module to obtain modality fusion features and F t h , go to step 5; Step 5: Combine the modal features from step 4 and F t h Feed it to the regression module to generate the population distribution density map to produce the final population value, and then go to step 6; Step 6: If the specified number of training rounds is reached, proceed to step 7, otherwise continue training and return to step 1; Step 7: End, completing the crowd counting training in the image.

2. According to claim 1, a cross-modal crowd counting method based on semantic guided multi-level fusion is characterized in that: The specific steps of step 2 are as follows: Step 2-1: For features and F t i Three independent 1×1 convolution operations are applied to reduce the dimensionality, producing two mapping features {Q r ,K r ,V r } and {Q t ,K t ,V t }, go to step 2-2; Step 2-2: For the color modality, query Q from the thermal modality t With key K r Perform matrix multiplication to calculate relative attention. The same is true for the thermal mode, and proceed to step 2-3. Step 2-3: For the two relative attentions calculated in the previous step, perform softmax layer normalization to generate relative attention weights W rt and W tr , proceed to step 2-4; Step 2-4: Use two attention weights to calculate the value V of the modality r and V r Perform weighted summation to obtain the reweighted features and Go to step 2-5; Step 2-5: Perform element-by-element multiplication of the weight and the value of the original modality to obtain the modality-enhanced features and Go to step 2-6; Step 2-6: Introduce learning parameters alpha_rgb and alpha_t to perform element-by-element multiplication with the enhanced RGB and T features to filter the features, and at the same time multiply them with the original RGB and T features and F t i Perform residual connection to accelerate network convergence and obtain fused features and Go to step 2-6; Step 2-7: Perform global average pooling on the obtained features, and concatenate them in the channel dimension to obtain the importance u of the feature response on the channel, and then go to step 2-7; Step 2-8: Apply convolution operation to the importance and combine with and Perform element-wise multiplication to obtain the expression of RGB features and thermal features for strengthening bidirectional modality enhancement and Go to step 2-9; Step 2-9: Re-weight the key channel information of feature expression and input it into the feature map to achieve more robust multimodal features and proceed to the subsequent steps.

3. According to claim 2, a cross-modal crowd counting method based on semantic guided multi-level fusion is characterized in that: The attention weight W rt and W tr Get the following formula: Where φ(*) represents the softmax operation, represents matrix multiplication; the characteristic formula after modal enhancement is as follows:

4. According to claim 2, a cross-modal crowd counting method based on semantic guided multi-level fusion is characterized in that: The formula for introducing learning parameters alpha_rgb and alpha_t and performing element-by-element multiplication with enhanced RGB and T features is expressed as follows: Where ⊙ represents element-by-element multiplication; The importance formula of the feature response on the channel dimension is as follows: Among them, Concat represents the cascade operation, and Avg represents the global average pooling operation.

5. According to claim 2, a cross-modal crowd counting method based on semantic guided multi-level fusion is characterized in that: Apply a convolution operation to the importance and and Perform element-wise multiplication to obtain the expression of RGB features and thermal features for strengthening bidirectional modality enhancement and The formula is as follows: Among them, FC represents the fully connected layer operation, and the formula for re-weighting using residual connection and initial modality features is as follows:

6. The method for cross-modal crowd counting based on semantically guided multi-level fusion according to claim 1, characterized in that: In step 3, the semantic expansion module is expressed as follows: The module formula is as follows: in is the kernel size k×k and the dilation rate d j =2×j dilated convolution, avg_pool(·) is adaptive average pooling, and j represents the branch layer.

7. The cross-modal crowd counting method based on semantic guided multi-level fusion according to claim 1 is characterized in that: The specific steps of step 4 are as follows: Step 4-1: Taking the second, third and fourth layer features extracted from the RGB image as an example, scale feature fusion is performed on each layer, and the number of feature channels is changed using the Relu activation function and 3×3 convolution, and then proceed to step 4-2; Step 4-2: Use depth convolution with different expansion rates to capture the multi-scale information of the above features, thereby producing Go to step 4-3; Step 4-3: Taking RGB features as an example, After upsampling and Perform element-by-element multiplication to obtain fused features Enter 4-4; Step 4-4: Combine the semantic information of the two modalities With the corresponding layered fusion features Further aggregation is performed to obtain the final fusion result and proceed to the subsequent steps.

8. The cross-modal crowd counting method based on semantic-guided multi-level fusion according to claim 7 is characterized in that: The Relu activation function and depth convolution formula are as follows: Among them, CH j (·) represents a 3×3 depthwise separable convolution with a dilation rate of j; The formula for gradually fusing the second, third and fourth layer features is as follows: Where Conv is a 3×3 convolutional layer with Relu activation function, Up represents upsampling, and C represents the concatenation operation.

9. The method for cross-modal crowd counting based on semantically guided multi-level fusion according to claim 7, characterized in that: The semantic information of the two modalities With the corresponding layered fusion features The further aggregation formula is as follows: Among them, Conv is 1×1 convolution, DConv is transposed convolution, and Up is bilinear upsampling.

10. A test method for verifying the crowd counting method of the multimodal hierarchical progressive interactive network according to any one of claims 1 to 9, characterized in that: The following steps are involved: Step 1: Input the image and true value in the test data and go to step 2; Step 2: Use the model weights obtained through training to extract and fuse features of the dataset image input in step 1, and proceed to step 3; Step 3: Output the density map in step 2, sum the pixels of the estimated crowd density map to get the estimated number of people and compare it with the actual value, and then go to step 4; Step 4: Calculate the absolute error and square error between the estimated number of people and the true value annotation, and proceed to step 5; Step 5: End.

Images