Multimodal crowd counting method based on modal attention gate recurrent unit

Abstract

The application discloses a multi-modal crowd counting method based on a modal attention gate recurrent unit, and the method comprises the following steps: preprocessing crowd data; inputting the preprocessed data into a Res2Net backbone network to extract feature output; inputting the extracted fifth layer features into a specific feature fusion module to extract a hidden state; performing feature fusion on the extracted five layer features and the hidden state based on a multi-layer gate recurrent unit; and inputting the fused features into a detail memory enhancement module to obtain an estimated crowd density map. The application improves the accuracy and the robustness of the model under low light.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a multimodal crowd counting method based on a modal attention-gated recurrent unit. Background Technology

[0002] Multimodal crowd counting methods effectively overcome the accuracy reduction problem of unimodal crowd counting under poor lighting conditions by combining traditional color (RGB) images and thermal imaging images. For example... Figure 1 As shown, the images are taken from surveillance footage. Color (RGB) images cannot accurately estimate the number of people in an image, but using thermal imaging significantly improves the accuracy of the count. This method is particularly useful in image analysis, especially in critical areas such as public safety, traffic management, and urban planning. For example, for monitoring events such as night markets, multimodal crowd counting can achieve more accurate crowd estimation, even when pedestrians are obscured or the viewing angle changes. By fusing RGB images and thermal imaging, this method improves the reliability and accuracy of crowd counting, especially in low-light or complex environments.

[0003] Early research on crowd counting methods primarily relied on traditional image processing techniques such as background subtraction, optical flow estimation, and pedestrian detection. Throughout the counting process, these methods typically used only color images as input. However, in real-world environments, color cameras often fail to capture clear crowd images under low-light or complex lighting conditions. For example, in a crowded outdoor scene, if only a color camera is used, images taken on a sunny day will be of better quality, while those taken on a cloudy day may be of lower quality. In such cases, variations in lighting significantly impact the performance of traditional crowd counting methods, leading to inaccurate counting results.

[0004] With technological advancements, an increasing number of methods employ completely symmetrical approaches to process multimodal information. This approach ignores the differences between multimodal features, leading to low accuracy and weak sensitivity to light intensity. These methods also exhibit poor robustness. Mainstream methods tend to integrate feature fusion into the feature extraction process, effectively achieving complementary fusion of multimodal information. These methods demonstrate excellent fusion performance, often employing top-down or bottom-up strategies. A novel multimodal crowd counting dataset has been proposed, utilizing a symmetrical plug-and-play module. Existing techniques, similar to the aforementioned methods, employ symmetrical approaches. However, observation reveals that data pairs in these datasets often fail to provide equivalent information, exhibiting omissions and misalignments in image pairs. Qualitative analysis of the final results reveals that these models perform poorly in low-light conditions. Meanwhile, most methods employ a progressive fusion strategy, in which high-level features are progressively transmitted and fused with shallow features from two streams. In this case, deep semantic information may be gradually diluted during the progressive fusion process, and low-quality depth images may also introduce noise into the fusion result.

[0005] While multimodal methods in current crowd counting techniques can integrate information from different modalities to some extent, they often overlook the inherent differences between modal features. Existing model frameworks tend to adopt a completely symmetrical approach, which not only limits the improvement of counting accuracy but also shows poor robustness under complex lighting conditions. Summary of the Invention

[0006] To address the aforementioned problems, this invention proposes a multimodal crowd counting method based on a modal attention-gated recurrent unit, specifically comprising the following steps:

[0007] Preprocess the population data;

[0008] The preprocessed data is input into the Res2Net backbone network to extract features and output them.

[0009] The extracted fifth-layer features are input into a specific feature fusion module to extract the hidden state;

[0010] The extracted five-layer features and hidden states are fused based on multi-layer gated recurrent units;

[0011] The fused features are input into the detail memory enhancement module to obtain the estimated population density map.

[0012] Furthermore, the crowd data includes: RGB images of the crowd and thermal images of the crowd.

[0013] Furthermore, the preprocessed data is input into the Res2Net backbone network to extract five layers of features. The specific steps include:

[0014] Features were extracted using the Res2Net backbone network. The extracted features are as follows: in, Represents the RGB image features of the i-th layer. Represents the image features of the i-th layer thermal imaging image;

[0015] The extracted features from each layer are passed through an attention module and a convolutional layer to obtain the output features, as shown in the following formula:

[0016] F=conv(channel(spital(E)))(1)

[0017] Here, spital(·) represents spatial attention mechanism, channel(·) represents channel attention, and conv(·) represents convolutional layer.

[0018] Furthermore, the extracted fifth-layer features are input into a specific feature fusion module to extract the hidden states. Specific steps include:

[0019] Output features of the fifth layer RGB image With thermal imaging image output features The concatenation is performed along the channel dimension, and the activation function is used as the input to the entire module, denoted as F. f ;

[0020] Features and characteristics Resize the device and perform pixel-level correlation calculations. The specific formula is as follows:

[0021]

[0022] in, Represents the RGB features of the input The result after resizing, K T Indicates thermal imaging features Each pixel in the algorithm is converted into its specific channel features, and conv(·) represents the convolution operation;

[0023] Next, spatial pooling is applied to the correlation results to transform them into pixel-wise spatial attention. The specific formula is as follows:

[0024] Att s =pool Spital (conv(P carr ))(3)

[0025] Among them, pool Spital(·) indicates a space pooling operation;

[0026] Features Correlation calculation is performed using channel attention processing; the specific formula is as follows:

[0027]

[0028] Att c =σ(Linear(T) max )+Linear(T avg ))(5)

[0029] in, Represents the input thermal imaging features The result after resizing, T max Indicates from T carr The channel max pooling obtained in T avg Indicates from T carr The channel average pooling obtained in the process is σ, which represents the Sigmoid activation function and Linear represents the fully connected layer.

[0030] Apply attention to F f The data is then concatenated through a connection operation and its size is transformed through a convolutional layer to obtain the output h0 of the feature fusion module as the hidden state. The specific formula is as follows:

[0031] h0 = conv(cat(F) f ⊙Att s ,F f ⊙Att c ))(6)

[0032] Here, ⊙ represents element-wise multiplication, and cat(·) represents concatenation operation.

[0033] Furthermore, the extracted five-layer features and hidden states are fused using multi-layer gated recurrent units, specifically including the following steps:

[0034] Each extracted feature layer is divided into low-level features and high-level features;

[0035] First, fuse the high-level features, and then gradually work down to the low-level features;

[0036] The hidden state of the previous level is sequentially passed through a specific modal gated recurrent unit along with the RGB image features and thermal imaging features of the current level. After the RGB features and thermal imaging features of each layer pass through the gated recurrent unit, the two fused results are passed through a connection layer. The specific formula is as follows:

[0037]

[0038] Among them, h i-1 This indicates the previous hidden state. SGRU is used for RGB mode fusion, and CGRU is used for thermal imaging mode fusion.

[0039] Furthermore, the fused features are input into the detail memory enhancement module to obtain the estimated crowd density map, specifically including the following steps:

[0040] The output hidden state of the higher-order part of the multi-gated recurrent unit (MLU) module is concatenated with all features input to the MLU module along the channel dimension; the shape of the concatenated features is then adjusted, and the output is h. out Finally, density estimation is performed to generate a density map.

[0041] According to one aspect of the present invention, a storage medium is provided, wherein the storage medium stores instructions that, when read by a computer, cause the computer to execute the multimodal crowd counting method based on a modal attention-gated recurrent unit as described above.

[0042] According to another aspect of the present invention, an electronic device is provided, comprising a processor and the aforementioned storage medium, wherein the processor executes instructions in the storage medium.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] 1. The present invention proposes a multimodal crowd counting method based on modal attention gated recurrent units. It uses specific modal attention and gated recurrent units to perform feature fusion, which improves accuracy and enhances the robustness of the model under low light conditions.

[0045] 2. The present invention adopts a detail memory enhancement strategy, which helps to more accurately represent and recover high-frequency details in fused features.

[0046] 3. This invention treats the information fusion process in multimodal population counting as a sequential process. This approach is similar to the process of progressively extracting and fusing features: first, high-level features are fused, and then gradually moved down to lower-level features. The update gate mechanism of the gated recurrent unit can effectively capture new fusion changes while retaining historical fusion information, thus enabling the model to retain past information and adapt to new fusion situations while continuously fusing features.

[0047] 4. The specific feature fusion module of this invention can help the model extract the spatial location of crowd information more effectively, avoiding interference from redundant information in multimodal features. This mechanism learns to generate a spatial descriptor that emphasizes the location of information, and uses a pixel-by-pixel similarity map as input to suppress irrelevant location information. Attached Figure Description

[0048] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0049] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0050] Figure 2 This is a network architecture diagram according to an embodiment of the present invention;

[0051] Figure 3 This is a model diagram of a multi-layer gated loop unit according to an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] like Figure 1 As shown, a multimodal crowd counting method based on a modal attention-gated recurrent unit is characterized by performing the following steps 1-11 for a crowd image to complete the crowd counting in the image:

[0054] Step 1: Preprocess the dataset. First, save the two modal data to the specified path. Then, randomly crop each image to the corresponding size of 256*256. Convert the ground truth annotation file into an npy file. Proceed to Step 2.

[0055] Step 2: Determine if the number of training rounds exceeds the preset maximum value. If it does, proceed to Step 7; otherwise, read the training set path to start model training, save the model weight file, and proceed to Step 3.

[0056] Step 3: Select N pairs of images from the training data in Step 1, extract features from them using the Res2Net backbone network, and pass each output through an attention module. Use convolutional layers to unify the number of channels, and proceed to Step 4.

[0057] Step 4: The fifth layer features extracted in Step 3 enter the specific feature fusion module to extract the spatial information of the crowd, and then proceed to Step 5;

[0058] Step 5: Divide each layer of features extracted in Step 3 into two stages: low-level and high-level. Use multi-layer gated recurrent units to perform feature fusion and proceed to Step 6.

[0059] Step 6: The two-level features fused layer by layer in Step 5 enter the detail memory enhancement module to enhance the detail representation of the fused features. After passing through the density map regression module, the estimated population density map is obtained, and then proceed to Step 7.

[0060] Step 7: Calculate the predicted value and the true value in Step 6, execute the backpropagation algorithm, adjust the model parameters by calculating the gradient of the loss function, and proceed to Step 8;

[0061] Step 8: Determine whether to start the verification round. If yes, verify the verification set data and proceed to step 9; otherwise, return to step 3.

[0062] Step 9: Verify if the indicator is optimal. If it is optimal, proceed to Step 10; otherwise, proceed to Step 3.

[0063] Step 10: Read the test set path and model weight file, start the test, output the metric data, and proceed to Step 11;

[0064] Step 11: Output the optimal model and end.

[0065] The specific steps for step 1 are as follows:

[0066] Step 1-1: Traverse all images in the training set, validation set, and test set, perform preprocessing operations on them, and then proceed to Step 1-2;

[0067] Step 1-2: Read the image path, modify the image file names of the two modalities according to the path and file name of the ground truth, and save them. Proceed to Step 1-3.

[0068] Steps 1-3: Read the actual value annotations, write them to the array Point, change the format of the annotation file to npy, and save the array Point to the specified path;

[0069] Steps 1-4: End.

[0070] The specific steps for step 3 are as follows:

[0071] Step 3-1: Extract features using the Res2Net backbone network. The extracted features are represented as follows: Proceed to step 3-2;

[0072] Step 3-2: Each feature output passes through an attention module, and finally through a convolutional layer to unify the number of channels before proceeding to the next step.

[0073] The attention module process described above is represented by the following formula:

[0074] F=conv(channel(spital(E)))(1)

[0075] Here, spital(·) represents spatial attention mechanism, channel(·) represents channel attention, and conv(·) represents convolutional layer.

[0076] The specific steps for step 4 are as follows:

[0077] Step 4-1: Process the features of the fifth layer of color image Features of thermal imaging images The vectors are concatenated along the channel dimension to form a new vector, which is then used as the input to the entire module via an activation function, denoted as F. f Proceed to step 4-2;

[0078] Step 4-2: For and Resize the image, perform pixel-level correlation calculations, and then apply spatial pooling to the correlation results to transform them into pixel-wise spatial attention. Proceed to step 4-3.

[0079] Step 4-3: Calculate the correlation of thermal imaging features, use channel attention processing, and proceed to step 4-4;

[0080] Step 4-4: The parts are joined together using a connection operation, and finally transformed to their initial state using a convolutional layer before proceeding to the next step.

[0081] The pixel-level correlation calculation in step 4 above is as follows:

[0082]

[0083] in, Represents the RGB features of the input The result after resizing, K T Indicates thermal imaging features Each pixel in the expression is transformed into its specific channel features, and conv(·) represents the convolution operation.

[0084] The spatial pooling transformation applied to the correlation results is expressed as a pixel-wise spatial attention formula, as follows:

[0085] Att s =pool Spital (conv(P carr ))(3)

[0086] Among them, pool Spital (·) indicates a space pooling operation;

[0087]

[0088] in, Represents the input thermal imaging features The result after resizing.

[0089] From T carr The channel maximum pooling T is obtained in the middle.max and channel average pooling T avg These two values ​​are used to generate channel attention, as shown in the following formula:

[0090] Att c =σ(Linear(T) max )+Linear(T avg ))(5)

[0091] Where σ represents the Sigmoid activation function and Linear represents a fully connected layer.

[0092] Apply attention to F f They are then concatenated together using a join operation and their dimensions are transformed using a convolutional layer, as shown in the following formula:

[0093] h0 = conv(cat(F) f ⊙Att s ,F f ⊙Att c ))(6)

[0094] Here, ⊙ represents element-wise multiplication, and cat(·) represents concatenation operation.

[0095] The specific steps for step 5 are as follows:

[0096] Step 5-1: Obtain the output h0 of the specific feature fusion module as the hidden state and the extracted multimodal features F, and proceed to step 5-2;

[0097] Step 5-2: Determine whether all high-level semantic features have entered the multi-layer gated recurrent unit. If yes, proceed to step 5-3; otherwise, proceed to step 5-1.

[0098] Step 5-3: Pass the hidden features of the previous level, the RGB image features of the current level, and the thermal imaging image features through a modal gated loop unit, and pass the two outputs through a connection layer to enter step 5-4;

[0099] Step 5-4: Determine whether all low-level semantic features have entered the multi-layer gated recurrent unit. If yes, proceed to step 5-5; otherwise, proceed to step 5-3.

[0100] Step 5-5: The hidden features of the previous level are sequentially combined with the RGB image features and thermal imaging image features of the current level through a specific modal gated loop unit. The two outputs are then passed through a connection layer to enter the next level, proceeding to step 5-6.

[0101] Steps 5-6: End, proceed to the next steps.

[0102] When information from a single modality is fused through a gated recurrent unit, feature loss or redundancy may occur, failing to reflect the complementary characteristics of the modalities. This problem can be effectively prevented by passing the RGB features and thermal imaging features from each layer through a gated recurrent unit and then connecting the fused results through a single layer. The formula is as follows:

[0103]

[0104] Among them, h i-1 This indicates the previous hidden state. SGRU is used for RGB mode fusion, and CGRU is used for thermal imaging mode fusion.

[0105] The loss function used in step 7 is the Bayesian loss function, and the formula is as follows:

[0106]

[0107] Where f is the distance function, and the L1 distance used in the experiment is E[c n The sum of the density maps is calculated using the posterior label probabilities and the estimated density map. When there are no objects in the training images, the sum of the density maps is directly set to zero. Since the loss function is differentiable, it can be easily applied to the backpropagation training algorithm.

[0108] The specific steps for the verification rounds in step 8 of this invention are as follows:

[0109] Step 8-1: Read the validation photos and corresponding data annotations according to the validation set path, read the weight file, and proceed to Step 8-2;

[0110] Step 8-2: Verify whether all image pairs in the validation set have been input into the validation model. If successful, proceed to step 8-3; otherwise, proceed to step 8-1.

[0111] Step 8-3: Input the currently verified image into the verification model, output the density map, sum the pixels of the estimated crowd density map to obtain the estimated number of people, and proceed to step 8-4;

[0112] Step 8-4: Calculate the absolute error and squared error between the estimated number of people and the actual number, and proceed to the next step.

[0113] As a preferred approach, two main evaluation metrics are used in the error calculation: Root Mean Square Error (RMSE) and Grid Mean Absolute Error (GAME). GAME(0) is equivalent to MAE, therefore RMSE and GAME(0) are used as the main evaluation metrics, while GAME[k] is used to evaluate the prediction accuracy of different image regions. Here, k determines the number of regions to be divided, which is 4. kThe mean absolute error (RMSE) for each region is calculated, and these results are then integrated. The definitions of RMSE and GAME are as follows:

[0114]

[0115] Where n represents the number of images. and Let represent the predicted number of people in the j-th region of the i-th image and the basic fact in the corresponding GT graph, respectively. and p i These are the predicted and actual values ​​for the i-th image, respectively. The lower the values ​​of these metrics, the more accurate the model's predictions.

[0116] The examples described herein are merely preferred embodiments of the invention and are not intended to limit the concept and scope of the invention. Any modifications and improvements made by those skilled in the art to the technical solutions of the invention without departing from the design concept of the invention should fall within the protection scope of the invention.

Claims

1. A multimodal crowd counting method based on a modal attention-gated recurrent unit, characterized in that, Includes the following steps: Preprocess the population data; The preprocessed data is input into the Res2Net backbone network to extract features and output them. The extracted fifth-layer features are input into a specific feature fusion module to extract the hidden state; The extracted five-layer features and hidden states are fused based on multi-layer gated recurrent units; The fused features are input into the detail memory enhancement module to obtain the estimated population density map; The extracted fifth-layer features are input into a specific feature fusion module to extract the hidden state. The specific steps include: Output features of the fifth layer RGB image With thermal imaging image output features The concatenation is performed along the channel dimension, and the activation function is used as the input to the entire module, denoted as... ; Features and characteristics Resize the device and perform pixel-level correlation calculations. The specific formula is as follows: (2) in, Represents the RGB features of the input The result after resizing Indicates thermal imaging features The result of converting each pixel into its specific channel features. Indicates the convolution operation; Next, spatial pooling is applied to the correlation results to transform them into pixel-wise spatial attention. The specific formula is as follows: (3) in, This represents the space pooling operation; Features Correlation calculation is performed using channel attention processing; the specific formula is as follows: (4) (5) in, Represents the input thermal imaging features The result after resizing Indicates from The channel max pooling obtained in the middle, Indicates from The channel average pooling obtained in the process This represents the Sigmoid activation function. Indicates a fully connected layer; Apply attention to The features are then concatenated through a connection operation and their sizes are transformed through a convolutional layer to obtain the output of the feature fusion module. As a hidden state, the specific formula is: (6) Where ⨀ represents element-wise multiplication, Indicates a splicing operation; The extracted five-layer features and hidden states are fused using multi-layer gated recurrent units, specifically including the following steps: Each extracted feature layer is divided into low-level features and high-level features; First, fuse the high-level features, and then gradually work down to the low-level features; The hidden state of the previous level is sequentially passed through a specific modal gated recurrent unit along with the RGB image features and thermal imaging features of the current level. After the RGB features and thermal imaging features of each layer pass through the gated recurrent unit, the two fused results are passed through a connection layer. The specific formula is as follows: (7) (8) (9) in, This indicates the previous hidden state. Used for RGB modal fusion Used for thermal imaging modal fusion.

2. The method according to claim 1, characterized in that, The crowd data includes: RGB images of the crowd and thermal images of the crowd.

3. The method according to claim 2, characterized in that, The preprocessed data is input into the Res2Net backbone network to extract five layers of features. The specific steps include: Features were extracted using the Res2Net backbone network. The extracted features are as follows: ,in, Represents the RGB image features of the i-th layer. Represents the image features of the i-th layer thermal imaging image; The extracted features from each layer are passed through an attention module and a convolutional layer to obtain the output features, as shown in the following formula: (1) in, This represents the spatial attention mechanism. Indicates channel attention. This indicates a convolutional layer.

4. The method according to claim 3, characterized in that, The fused features are input into the detail memory enhancement module to obtain the estimated crowd density map, which includes the following steps: The output hidden state of the higher-order part of the multi-gated recurrent unit (MLU) module is concatenated with all features input to the MLU module along the channel dimension; the shape of the concatenated features is then adjusted, and the output is... Finally, density estimation is performed to generate a density map.

5. A storage medium, characterized in that, The storage medium stores instructions that, when read by a computer, cause the computer to execute the multimodal crowd counting method based on a modal attention-gated recurrent unit as described in any one of claims 1-4.

6. An electronic device, characterized in that, It includes a processor and the storage medium of claim 5, wherein the processor executes instructions in the storage medium.

Images