Crowd counting method of multi-modal hierarchical progressive interactive network

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to a crowd counting method of a multi-mode hierarchical progressive interactive network. According to the crowd counting method of the multi-modal hierarchical progressive interaction network, the information aggregation module is designed, data of different modals are allowed to be interacted and fused in each processing stage, it is ensured that key information can be effectively transmitted to a high level, meanwhile, low-level and high-level features are fused, and the accuracy of crowd counting is improved. Important local information can be reserved, so that the adaptability and flexibility of the model are enhanced, and the crowd counting precision is improved. According to the method, a root mean square error RMSE and a grid mean absolute error GMAE are adopted as measurement standards. Through verification of experimental indexes, compared with other unmanned aerial vehicle counting methods, the provided method makes a progress in precision, and the robustness and application value of the method in practical application are proved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a crowd counting method for a multimodal hierarchical progressive interactive network. Background Art

[0002] Crowd counting technology aims to count the number of people in a given scene. It has attracted widespread attention due to its important role in real-world applications such as crowd control, video surveillance, event monitoring, and traffic control. Common crowd counting methods can be divided into three types: detection-based, regression-based, and density estimation. Detection-based methods use bounding boxes to accurately locate each person in an image, while regression-based methods directly output the number of people in an image or a density map of the corresponding image. However, in complex environments or large crowded scenes, traditional methods often fail to achieve accurate estimates. Due to the strong learning ability of CNN, it can learn more feature information, which makes it perform well in the field of crowd counting, especially for densely populated areas, where better prediction results can be obtained.

[0003] Traditional single-modal crowd counting methods often perform poorly in the face of real-world scenarios such as occlusion, perspective changes, and lighting conditions. In order to address these challenges, multimodal crowd counting technology has emerged. Multimodal crowd counting combines multiple data sources, such as RGB images, thermal imaging (infrared) images, etc., and improves robustness and accuracy in complex scenes by fusing information from different modalities. This technology can more effectively cope with various complex situations. For example, in low-light or severely occluded environments, infrared images can provide useful information supplements.

[0004] Many methods have been proposed for modal fusion, which can be roughly divided into the following three categories based on the fusion strategy, namely early fusion, late fusion and mid-term fusion. Among them, the intermediate fusion method learns the intermediate features of the two modalities from the two-stream network, and then uses the subsequent network or decoder to generate the crowd density map. This fusion strategy can take into account the complex correlation between the two modalities and bring better performance for multimodal crowd counting. However, most fusion methods only extract image features from shallow to deep layers and generate the final fusion results in sequence, rather than providing new feature inputs to the corresponding feature extraction network to guide feature learning, which ignores the interference of unequal quality image information. In addition, the mainstream method fusion operation is always performed at a single level without considering the diversity and complementarity of multimodal features.

[0005] On this basis, most practices always directly fuse according to the depth of the fusion information. In the later stage of fusion, the basic pixel-level information may be directly ignored, which leads to low accuracy of crowd counting. Recently, Peng T, Li Q, Zhu P. RGB-T Crowd Counting from Drone: A Benchmark and MMCCN Network [C] / / Asian Conference on Computer Vision. Springer, Cham, 2021. DOI: 10.1007 / 978-3-030-69544-6_30. proposed an RGB-T crowd counting dataset for drones. The dataset images are as follows: Figure 1 As shown in the figure, the first row is RGB images and the second row is thermal images. A multimodal crowd counting network with a multi-scale feature learning module, a modality alignment module and an adaptive fusion module is proposed. The existing technologies all use methods similar to the above fusion operations. The flowchart of the conventional method is shown in the figure. Figure 3 shown.

[0006] In the current crowd counting technology, image information from different sources is introduced to solve problems such as lighting conditions and occlusion. On the one hand, the existing cross-modal crowd counting problem is to directly extract features from images, which may cause erroneous information to propagate and amplify in multiple levels. Since the model only relies on the specific features of this single modality, the later fusion effect of the model will not be significant enough. On the other hand, most methods directly fuse the aggregated features of each layer, ignoring the characteristics of low-level features and high-level features, which may lead to the direct neglect of basic pixel-level information in the later stage of fusion, resulting in low crowd counting accuracy. Summary of the invention

[0007] In view of the technical problems in the above-mentioned prior art, the present invention proposes a crowd counting method based on a multimodal hierarchical progressive interactive network. By designing an information aggregation module, data of different modalities are allowed to interact and fuse at each processing stage, ensuring that key information can be effectively transmitted to the high level. At the same time, low-level and high-level features are fused separately, which helps to retain important local information, thereby enhancing the adaptability and flexibility of the model and improving the accuracy of crowd counting.

[0008] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0009] A crowd counting method for a multimodal hierarchical progressive interaction network comprises the following steps:

[0010] Step 1: Preprocess the data set. First, save the two modal data to the specified path. Then randomly crop each image to the corresponding size of 352*352. Convert the real value annotation file to npy file and proceed to step 2.

[0011] Step 2: Determine whether the number of training rounds exceeds the preset maximum value. If so, proceed to step 7. Otherwise, read the training set path to start model training, save the model weight file, and proceed to step 3.

[0012] Step 3: Select N pairs of images from the training data in step 1, use the ResNet backbone network to extract features, and extract the first layer features of RGB images and thermal images to obtain Go to step 4;

[0013] Step 4: The first layer of features extracted in step 3 Enter the information aggregation module, feed the updated features back to the feature extractor corresponding to the next layer, and repeat the above step 4 for the remaining four layers until the features are extracted. Where i = 1, 2, .., 5); the features output by the information aggregation module are Where i=1,2,..,5); go to step 5;

[0014] Step 5: The output features of each layer of the information aggregation module in step 4 enter the information enhancement module, and obtain more comprehensive information F through dilated convolution and residual connection i m , go to step 6;

[0015] Step 6: Replace the F i m The first three layers and the last three layers of are divided into low-level features and high-level features, and the low-level features and high-level features are fused separately to obtain f l and f h , go to step 7;

[0016] Step 7: High-level fusion features f h Guide the low-level fusion feature f l Further refine the features and suppress background noise, generate the final crowd density map through the regression module, and proceed to step 8;

[0017] Step 8: Compare the number of people predicted by the crowd density map with the actual value, and adjust the model parameters through the gradient of the loss function, and proceed to step 9;

[0018] Step 9: When it is determined that the round number is greater than 50 and is a multiple of 5, read the test set image and the model weight file corresponding to the training, enter the test process, and go to step 10;

[0019] Step 10: Get the model training parameters corresponding to the optimal test results and end.

[0020] Further, as a preferred technical solution of the present invention, the specific steps of step 1 are as follows: Step 1-1: traverse all images according to the training set test set path, complete preprocessing operations such as cropping and data conversion, and enter step 1-2; Step 1-2: read the image storage path, convert the crowd coordinate information and image information in the .xml file where the true value is stored into an npy file and save it, and enter step 1-3; Step 1-3: end.

[0021] Further as a preferred technical solution of the present invention, the specific steps of step 4 are as follows:

[0022] Step 4-1: Extract the features from the first layer Input into the information aggregation module, first merge the two features by element-by-element multiplication to obtain Go to step 4-2;

[0023] Step 4-2: Then for the merged features Convolution, batch normalization and ReLU operations are performed to enhance the feature representation of the merged features, and then upsampling and maximum pooling operations are performed to obtain the enhanced feature representation. Go to step 4-3;

[0024] Step 4-3: Merge the features Respectively with the original features Add, upsample and pool to obtain updated feedback features Go to step 4-4;

[0025] Step 4-4: For the updated feedback features, add them to obtain the shared features expressed as Go to step 4-5;

[0026] Step 4-5: Repeat the above steps for the next four layers of features to obtain feedback features in turn Go to the next steps;

[0027] The formula for merging the extracted features in the above steps is as follows:

[0028]

[0029] in, Represents the RGB and thermal imaging features extracted by this layer, Represents element-by-element multiplication; the formula for obtaining the merged enhanced features is as follows:

[0030]

[0031] Among them, F CBR represents convolution, batch normalization and ReLU operations; the new feature formula fed back to the next layer by the merged feature update is expressed as follows:

[0032]

[0033] in, Represents an addition operation.

[0034] Further, as a preferred technical solution of the present invention, the information enhancement module used in step 5 is expressed by the following formula:

[0035]

[0036] Among them, CL j Represents a 3*3 convolution operation with j dilation rate, j=1,2,3,4.

[0037] Further as a preferred technical solution of the present invention, the specific steps of step 6 are as follows:

[0038] Step 6-1: Place F i m Divided into low-level features F 1 m , and advanced features Go to step 6-2;

[0039] Step 6-2: Perform feature layer fusion on high-level features and low-level features respectively. Taking low-level feature fusion as an example, first perform feature fusion on F 1 m Perform upsampling convolution and ReLU function operations, upsample again and pool to get F 1 m ', to reduce the dimension, go to step 6-3;

[0040] Step 6-3: Place F 1 m 'and Connect and process the merged features through a 3*3 convolution, batch normalization and ReLU function, and 1 m ′ Perform element-by-element multiplication to obtain To enhance feature compatibility and strengthen important features, proceed to step 6-4;

[0041] Step 6-4: and Connect and process through another 3*3 convolution, batch normalization and ReLU function, and use element-by-element multiplication to combine the output with Combined to get f l , to enhance feature alignment and expression, proceed to step 6-5;

[0042] Step 6-5: Repeat the above operation for high-level feature fusion to obtain f h , f l With f h The final output features are obtained by fusion and then proceed to the next step;

[0043] The formula for fusing the low-level two-level features in the above steps is as follows:

[0044]

[0045] The formula for fusing the low-level two-level features with the high-level features is expressed as:

[0046]

[0047] Further as a preferred technical solution of the present invention, the testing process in step 9 includes the following steps:

[0048] Step 9-1: Read the test set photos and corresponding data annotations according to the verification set path, read the weight file, and go to step 9-2;

[0049] Step 9-2: Verify whether all image pairs in the test set are input into the test model. If completed, proceed to step 9-3, otherwise proceed to step 9-1;

[0050] Step 9-3: Input the current test image into the test model, output the density map, sum the pixels of the estimated crowd density map to get the estimated number of people, and go to step 9-4;

[0051] Step 9-4: Calculate the absolute error and square error between the estimated number of people and the true value annotation, and proceed to the next step;

[0052] In the above steps, two main evaluation indicators are used in the error calculation: root mean square error RMSE and grid mean absolute error GAME; the definitions of RMSE and GAME are as follows:

[0053]

[0054] Where n represents the number of images, and denote the predicted value of the number of people in the jth region in the ith image and the basic facts in the corresponding GT image, respectively, and and p i are the predicted value and true value of the i-th image respectively; the lower the value of the indicator, the more accurate the prediction of the model.

[0055] The crowd counting method of a multimodal hierarchical progressive interactive network described in the present invention adopts the above technical solution and has the following technical effects compared with the prior art:

[0056] The present invention proposes a crowd counting method based on a multimodal hierarchical progressive interactive network. The modal information is aggregated and updated by an information aggregation module, and the merged features are divided into low-level features and high-level features as the input for feature extraction of the next layer, and the merged features are fused separately to avoid forgetting of detailed information and improve counting accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Images from the drone counting dataset showing different lighting and crowd density;

[0058] Figure 2 A framework diagram of the method proposed in the embodiment of the present invention;

[0059] Figure 3 A schematic diagram comparing the conventional method and the training phase flow chart of the present invention;

[0060] Figure 4 This is a framework diagram of the information aggregation module proposed in an embodiment of the present invention;

[0061] Figure 5 This is a framework diagram of the information enhancement module proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The present invention is further explained below in detail with reference to the accompanying drawings so that those skilled in the art can more deeply understand the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention.

[0063] The present invention proposes a crowd counting method based on a multimodal hierarchical progressive interactive network, aiming to solve the problems of low accuracy and poor robustness of the multimodal crowd counting method. The framework diagram of the method is as follows: Figure 2 shown. Figure 2 The proposed invention network architecture diagram, the framework mainly includes two modules: information enhancement and information aggregation, which respectively aggregate the extracted features and feed them back to the input of the next layer of feature extractor, and at the same time, based on the enhanced context information of the aggregated features, hierarchical guided aggregation decoding is performed.

[0064] Specifically, the present invention proposes an information aggregation module, which aims to update the extracted information of each layer of the image. The information aggregation module helps to integrate more fine-grained information and detect and correct defects in each layer in a timely manner. The framework diagram of the module is shown in the figure. Figure 4As shown in the figure, the information aggregation module framework mainly uses the features of modal fusion to reversely guide the adjustment and update of specific features of each modality. In addition, an information enhancement module is proposed to help obtain global context information, which helps to enhance the fusion effect of subsequent high-level and low-level information and help the model capture more complex and hierarchical scene structures. The module framework is shown in the figure Figure 5 As shown in the figure, the framework diagram of the information enhancement module mainly adopts a multi-branch approach to enable the model to capture a wide range of features from details to extensive contextual information.

[0065] The technical training process proposed by the present invention is as follows Figure 3 As shown, the conventional method and the flow chart of the training phase of the present invention are compared; before model training, the optimizer (Optimizer), weight (Weight), learning rate (Learningrate) and other parameters need to be initialized in advance. After the settings are completed, N crowd images will be selected from the training set data each time. The training process of the present invention is introduced as follows:

[0066] Step 1: Preprocess the data set. First, save the two modal data to the specified path. Then randomly crop each image to the corresponding size of 352*352. Convert the real value annotation file to npy file and proceed to step 2.

[0067] Step 2: Determine whether the number of training rounds exceeds the preset maximum value. If so, proceed to step 10. Otherwise, read the training set path to start model training, save the model weight file, and proceed to step 3.

[0068] Step 3: Select N pairs of images from the training data in step 1, use the ResNet backbone network to extract features, and extract the first layer features of RGB images and thermal images to obtain Go to step 4;

[0069] Step 4: The first layer of features extracted in step 3 Enter the information aggregation module, feed the updated features back to the feature extractor corresponding to the next layer, and repeat the above step 4 for the remaining four layers until the features are extracted. The features output by the information aggregation module are Go to step 5;

[0070] Step 5: The output features of each layer of the information aggregation module in step 4 enter the information enhancement module, and obtain more comprehensive information F through dilated convolution and residual connection i m , go to step 6;

[0071] Step 6: Replace the F i mThe first three layers and the last three layers of are divided into low-level features and high-level features, and the low-level features and high-level features are fused separately to obtain f l and f h , go to step 7;

[0072] Step 7: High-level fusion features f h Guide the low-level fusion feature f l Further refine the features and suppress background noise, generate the final crowd density map through the regression module, and proceed to step 8;

[0073] Step 8: Compare the number of people predicted by the crowd density map with the actual value, and adjust the model parameters through the gradient of the loss function, and proceed to step 9;

[0074] Step 9: When it is determined that the round number is greater than 50 and is a multiple of 5, read the test set image and the model weight file corresponding to the training, enter the test process, and go to step 10;

[0075] Step 10: Get the model training parameters corresponding to the optimal test results and end.

[0076] The dataset used in step 1 is captured from drone-mounted cameras, covering a wide range of scenes such as campuses, streets, parks, parking lots, playgrounds, and squares. The image pairs are captured under different lighting conditions such as darkness, dusk, and illumination. The images consist of 3,600 pairs of images. It is a multimodal drone view dataset with highly accurate cross-modal alignment and does not require any pre-processing or post-processing. It is also a large-scale dataset collected in many different scenes with 175,698 annotated instances. The steps of the data preprocessing stage are as follows:

[0077] Step 1-1: Traverse all images according to the training set and test set paths, complete preprocessing operations such as cropping and data conversion, and proceed to step 1-2;

[0078] Step 1-2: Read the image storage path, convert the crowd coordinate information and image information in the .xml file where the real value is stored into an npy file and save it, and then proceed to step 1-3;

[0079] Step 1-3: End.

[0080] In the present invention, step 3 uses the ResNet50 backbone network to extract features. The backbone allows the signal in the network to bypass one or more layers and pass directly, which helps to solve the gradient vanishing problem in deep networks. First, extract the first layer features of RGB and thermal imaging. Then, the feature aggregation module of the previous layer updates the features as new input to obtain the features of other layers.

[0081] The information aggregation module in step 4 effectively aggregates the feature representations shared across modalities and introduces feedback features. These feedback features are deeply integrated with the current layers of the RGB branch and the thermal imaging branch, which significantly improves the model's ability to learn discriminative features and enhances the distinguishability and expressiveness of the features. The specific steps are as follows:

[0082] Step 4-1: Extract the features from the first layer Input into the information aggregation module, first merge the two features by element-by-element multiplication to obtain Go to step 4-2;

[0083] Step 4-2: Then for the merged features Convolution, batch normalization and ReLU operations are performed to enhance the feature representation of the merged features, and then upsampling and maximum pooling operations are performed to obtain the enhanced feature representation. Go to step 4-3;

[0084] Step 4-3: Merge the features Respectively with the original features Add, upsample and pool to obtain updated feedback features Go to step 4-4;

[0085] Step 4-4: For the updated feedback features, add them to obtain the shared features expressed as Go to step 4-5;

[0086] Step 4-5: Repeat the above steps for the next four layers of features to obtain feedback features in turn Go to next steps.

[0087] The formula for merging the extracted features in step 4 is as follows:

[0088]

[0089] in, Represents the RGB and thermal imaging features extracted by this layer, Represents element-by-element multiplication. The formula for obtaining the merged enhanced features is as follows:

[0090]

[0091] Among them, F CBR Represents convolution, batch normalization and ReLU operations. The new feature formula fed back to the next layer by the merged feature update is expressed as follows:

[0092]

[0093] in, Represents an addition operation.

[0094] In step 5, the information enhancement module is used to mine the contextual information of the merged features. The multi-scale information of the input data is captured through four parallel branches with different expansion rates and a skip connection, which can extract features of different scales, which is particularly important for understanding objects of various sizes in the image (such as people at different distances in crowd counting). In addition, this module integrates multi-level features from details to abstraction through sequential convolution within the branch and fusion between branches, improving the representation ability of the features.

[0095] The formula of the information enhancement module is as follows:

[0096]

[0097] Among them, CL j Represents a 3*3 convolution operation with j dilation rate, j=1,2,3,4.

[0098] In step 6 of the present invention, a new method of hierarchical fusion is adopted. High-level features usually include abstract and global information, such as the overall distribution, shape and movement pattern of the crowd, while low-level features contain more detailed information, such as texture, edge and color changes. In a specific situation, some patterns may only be expressed through a specific feature level. The usual practice is to directly fuse the extracted five-layer fusion features, which may cause the omission of important information. Therefore, dividing the features into low-level and high-level features helps to understand the global and local details of the scene at the same time, avoiding the problem of especially detailed crowd information being ignored due to propagation, and reducing the impact of background noise. The specific steps are as follows:

[0099] Step 6-1: Place F i m Divided into low-level features F 1 m , and advanced features Go to step 6-2;

[0100] Step 6-2: Perform feature layer fusion on high-level features and low-level features respectively. Taking low-level feature fusion as an example, first perform feature fusion on F 1 m Perform upsampling convolution and ReLU function operations, upsample again and pool to get F 1 m ', to reduce the dimension, go to step 6-3;

[0101] Step 6-3: Place F 1 m 'and Connect and process the merged features through a 3*3 convolution, batch normalization and ReLU function, and 1 m ′ Perform element-by-element multiplication to obtain To enhance feature compatibility and strengthen important features, proceed to step 6-4;

[0102] Step 6-4: and Connect and process through another 3*3 convolution, batch normalization and ReLU function, and use element-by-element multiplication to combine the output with Combined to get f l , to enhance feature alignment and expression, proceed to step 6-5;

[0103] Step 6-5: Repeat the above operation for high-level feature fusion to obtain f h , f l With f h The final output features are obtained by fusion and then proceed to the next step.

[0104] The formula for fusing the low-level features in the low-level and high-level classifications in step 6 is as follows:

[0105]

[0106] The formula for fusing the low-level two-level features with the high-level features is expressed as:

[0107]

[0108] The specific steps of the test round in step 9 of the present invention are as follows:

[0109] Step 9-1: Read the test set photos and corresponding data annotations according to the verification set path, read the weight file, and go to step 9-2;

[0110] Step 9-2: Verify whether all image pairs in the test set are input into the test model. If completed, proceed to step 9-3, otherwise proceed to step 9-1;

[0111] Step 9-3: Input the current test image into the test model, output the density map, sum the pixels of the estimated crowd density map to get the estimated number of people, and go to step 9-4;

[0112] Step 9-4: Calculate the absolute error and square error between the estimated number of people and the true value label, and proceed to the next step.

[0113] There are two main evaluation indicators used in the error calculation: root mean square error (RMSE) and grid mean absolute error (GAME). The definitions of RMSE and GAME are as follows:

[0114]

[0115] Where n represents the number of images, and denote the predicted value of the number of people in the jth region in the ith image and the basic facts in the corresponding GT image, respectively, and and p i are the predicted value and true value of the i-th image respectively. The lower the value of these indicators, the more accurate the model's prediction is.

[0116] Existing crowd counting methods often fail to consider the differences in information at different levels when processing two modal extraction information. Direct fusion may cause low-level detail information to be forgotten when fused to the second half, resulting in low counting accuracy. To address this problem, the present invention proposes a crowd counting method based on a multimodal hierarchical progressive interactive network, which uses an information aggregation module to aggregate and update modal information and uses it as the input for feature extraction of the next layer. The merged features are divided into low-level features and high-level features for fusion, thereby avoiding the forgetting of detail information and improving counting accuracy.

[0117] Embodiment 1:

[0118] In this embodiment, the present invention aims to demonstrate the effectiveness of the present invention in practical application scenarios through a series of experiments. The dataset used in this example is the widely recognized DroneRGBT public dataset, which contains 3,600 pictures of different scenes taken by drones from different heights under different lighting conditions, such as campuses, streets, parks, parking lots, playgrounds, and squares. All images are randomly horizontally flipped with zero padding and randomly cropped to avoid overfitting. The optimizer used in the experiment is Adam, with the following hyperparameters: learning rate of 1e-5, step decay rate of 0.995, and batch size of 8. For fair comparison, the same parameter settings are used for each model, the same image processing input operation is used, and the Bayesian loss function is used to train according to the following steps.

[0119] S1: Take the batch size of images from the training set, perform forward propagation on the model and images, obtain the model output results, and calculate the error value with the ground truth value through the loss function.

[0120] S2: Perform back propagation and calculate the gradient. The optimizer updates the weights and bias parameters of the model according to the gradient of the loss function.

[0121] S3: At the end of each epoch, check whether the model meets the conditions for entering the test. If so, enter the test phase until the parameters of the best model are saved. If not, continue training until the maximum epoch value is reached.

[0122] The results of the invention and the comparison model are shown in Table 1 below:

[0123] Table 1: Experimental results of the method proposed in this invention and other methods

[0124]

[0125]

[0126] When evaluating the performance of the model, the present invention uses the root mean square error (RMSE) and the grid mean absolute error (GMAE) as the measurement criteria. Among these indicators, the lower the value of GAME(k), the smaller the gap between the model's prediction results and the actual values, which reflects that the model's prediction effect is better. After verification of experimental indicators, the proposed method has achieved an improvement in accuracy compared with other drone counting methods, proving its robustness and application value in practical applications.

[0127] Currently, the following two indicators are used in the direction of cross-modal crowd counting based on accuracy: root mean square error (RMSE) and grid mean absolute error (GAME). The smaller the values ​​of these two indicators, the higher the accuracy. The present invention uses the drone crowd counting dataset DroneRGBT for experiments. The dataset contains a large number of drone images taken under different lighting conditions. The same experimental conditions are used for comparison with other advanced methods. The experiments show that the GAME[0] of the BBSNet method is 8.24, GAME[1] is 10.32, GAME[2] is 12.96, GAME[3] is 17.31, and the RMSE is 12.96. The GAME[0] of the DEFNet method is 7.38, GAME[1] is 9.66, GAME[2] is 12.49, GAME[3] is 16.45, and the RMSE is 11.62. The GAME[0] of the present invention is 6.95, GAME[1] is 9.25, GAME[2] is 11.98, GAME[3] is 16.13, and the RMSE is 10.78. It is proved that the present method can improve the accuracy compared with other multimodal drone counting methods.

[0128] Embodiment 2:

[0129] This embodiment will introduce an applicable scenario of the present invention:

[0130] With the acceleration of urbanization, large-scale public events and festivals are becoming more and more frequent. In these occasions, crowd safety management and flow control have become important challenges for organizers and city managers. Drones can cruise on a predetermined flight path and collect data through cameras. Image data is transmitted to the ground control station, where deep learning models are used to analyze the images and automatically identify and count crowds. The model is specially trained to distinguish individuals in the crowd and maintain high accuracy even in dense crowds. Organizers can monitor crowd density and prevent overcrowding through real-time crowd counting data. If the number of people in a certain area exceeds the safety threshold, organizers can take quick measures, such as guiding the crowd to disperse to other areas or increasing security personnel to maintain order. In addition, drones can not only complete the counting function, but also provide an aerial perspective to monitor the entire activity area, helping security personnel to promptly detect abnormal situations, such as fighting, emergency medical situations or potential safety hazards. Therefore, the model of the present invention can be applied to the ground control station for crowd flow analysis and safety control.

[0131] First, multiple drones are used to take pictures above the crowds at large events and transmit the pictures back to the ground control station in a timely manner;

[0132] Secondly, the data is input into the model of the invention, and the crowd density estimation and distribution are output. If the density exceeds the crowd warning value, a warning is issued to relevant personnel in a timely manner;

[0133] Finally, the use of this model can improve the safety of large-scale events and provide organizers with valuable data support to help them better plan and manage events.

[0134] The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. A crowd counting method based on a multimodal hierarchical progressive interaction network, characterized in that: The following steps are involved: Step 1: Preprocess the data set. First, save the two modal data to the specified path. Then randomly crop each image to the corresponding size of 352*352. Convert the real value annotation file to npy file and proceed to step 2. Step 2: Determine whether the number of training rounds exceeds the preset maximum value. If so, proceed to step 7. Otherwise, read the training set path to start model training, save the model weight file, and proceed to step 3. Step 3: Select N pairs of images from the training data in step 1, use the ResNet backbone network to extract features, and extract the first layer features of RGB images and thermal images to obtain Go to step 4; Step 4: The first layer of features extracted in step 3 Enter the information aggregation module, feed the updated features back to the feature extractor corresponding to the next layer, and repeat the above step 4 for the remaining four layers until the features are extracted. Where i = 1, 2, .., 5); the features output by the information aggregation module are Where (i=1,2,..,5); go to step 5; Step 5: The output features of each layer of the information aggregation module in step 4 enter the information enhancement module to obtain more comprehensive information through dilated convolution and residual connection Go to step 6; Step 6: Replace the The first three layers and the last three layers of are divided into low-level features and high-level features, and the low-level features and high-level features are fused separately to obtain f l and f h , go to step 7; Step 7: High-level fusion features f h Guide the low-level fusion feature f l Further refine the features and suppress background noise, generate the final crowd density map through the regression module, and proceed to step 8; Step 8: Compare the number of people predicted by the crowd density map with the actual value, and adjust the model parameters through the gradient of the loss function, and proceed to step 9; Step 9: When it is determined that the round number is greater than 50 and is a multiple of 5, read the test set image and the model weight file corresponding to the training, enter the test process, and go to step 10; Step 10: Get the model training parameters corresponding to the optimal test results and end.

2. The method for counting people in a multimodal hierarchical progressive interactive network according to claim 1, characterized in that: The specific steps of step 1 are as follows: Step 1-1: Traverse all images according to the training set and test set paths, complete preprocessing operations such as cropping and data conversion, and proceed to step 1-2; Step 1-2: Read the image storage path, convert the crowd coordinate information and image information in the .xml file where the real value is stored into an npy file and save it, and then proceed to step 1-3; Step 1-3: End.

3. The method for counting people in a multimodal hierarchical progressive interactive network according to claim 2, characterized in that: The specific steps of step 4 are as follows: Step 4-1: Extract the features from the first layer Input into the information aggregation module, first merge the two features by element-by-element multiplication to obtain Go to step 4-2; Step 4-2: Then for the merged features Convolution, batch normalization and ReLU operations are performed to enhance the feature representation of the merged features, and then upsampling and maximum pooling operations are performed to obtain the enhanced feature representation. Go to step 4-3; Step 4-3: Merge the features Respectively with the original features Add, upsample and pool to obtain updated feedback features Go to step 4-4; Step 4-4: For the updated feedback features, add them to obtain the shared features expressed as Go to step 4-5; Step 4-5: Repeat the above steps for the next four layers of features to obtain feedback features in turn Go to the next steps; The formula for merging the extracted features in the above steps is as follows: in, Represents the RGB and thermal imaging features extracted by this layer, Represents element-by-element multiplication; the formula for obtaining the merged enhanced features is as follows: Among them, F CBR represents convolution, batch normalization and ReLU operations; the new feature formula fed back to the next layer by the merged feature update is expressed as follows: in, Represents an addition operation.

4. The method for counting people in a multimodal hierarchical progressive interactive network according to claim 3, characterized in that: The information enhancement module used in step 5 is expressed as follows: Among them, CL j Represents a 3*3 convolution operation with j dilation rate, j=1,2,3,4.

5. The method for counting people in a multimodal hierarchical progressive interactive network according to claim 4, characterized in that: The specific steps of step 6 are as follows: Step 6-1: Low-level features and advanced features Go to step 6-2; Step 6-2: Perform feature layering fusion on high-level features and low-level features respectively. Taking low-level feature fusion as an example, first perform feature fusion on Perform upsampling convolution and Relu function operations, upsample again and pool to obtain To reduce the dimension, go to step 6-3; Step 6-3: and Connect and process the merged features through a 3*3 convolution, batch normalization and Relu function, and Perform element-wise multiplication to get To enhance feature compatibility and strengthen important features, proceed to step 6-4; Step 6-4: and Connect and process through another 3*3 convolution, batch normalization and ReLU function, and use element-by-element multiplication to combine the output with Combined to get f l , to enhance feature alignment and expression, proceed to step 6-5; Step 6-5: Repeat the above operation for high-level feature fusion to obtain f h , f l With f h The final output features are obtained by fusion and then proceed to the next step; The formula for fusing the low-level two-level features in the above steps is as follows: The formula for fusing the low-level two-level features with the high-level features is expressed as:

6. The method for counting people in a multimodal hierarchical progressive interactive network according to claim 5, characterized in that: The testing process in step 9 includes the following steps: Step 9-1: Read the test set photos and corresponding data annotations according to the verification set path, read the weight file, and go to step 9-2; Step 9-2: Verify whether all image pairs in the test set are input into the test model. If completed, proceed to step 9-3, otherwise proceed to step 9-1; Step 9-3: Input the current test image into the test model, output the density map, sum the pixels of the estimated crowd density map to get the estimated number of people, and go to step 9-4; Step 9-4: Calculate the absolute error and square error between the estimated number of people and the true value annotation, and proceed to the next step; In the above steps, two main evaluation indicators are used in the error calculation: root mean square error RMSE and grid mean absolute error GAME; the definitions of RMSE and GAME are as follows: Where n represents the number of images, and denote the predicted value of the number of people in the jth region in the ith image and the basic facts in the corresponding GT image, respectively, and and p i are the predicted value and true value of the i-th image respectively; the lower the value of the indicator, the more accurate the prediction of the model.

Images