Crowd counting system and method based on cross-modal feature registration and ghosting suppression

Abstract

The invention provides a crowd counting system and method based on cross-modal feature registration and ghosting suppression, and relates to the technical field of computer vision. The system comprises a visible light and infrared light feature extraction module, three cross-modal feature registration modules, four cross-modal ghosting suppression and fusion modules and a crowd density map estimation module. The visible light and infrared light feature extraction module extracts multi-level features from visible light and infrared light images of the same scene. The cross-modal feature registration module performs spatial registration on the visible light features based on the infrared light features so as to eliminate dislocation between modals; the ghosting suppression and fusion module carries out redundancy suppression and attention fusion on registered or unregistered features at all feature levels to generate multi-level fusion features; and the density map estimation module carries out crowd density estimation according to the multi-level fusion features to obtain the estimated value of the number of people in the scene to be counted. According to the method, the precision and robustness of crowd counting are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a crowd counting system and method based on cross-modal feature registration and ghost suppression. BACKGROUND

[0002] With the acceleration of urbanization and the continuous rise of population density, crowd counting as a key task in computer vision is increasingly becoming an indispensable part of smart city construction. It is not only widely used in industrial informationization scenarios such as public safety warning, disaster prevention and control, and traffic management, but also plays an important role in commercial analysis, urban planning, and crowd behavior analysis. Accurate and efficient counting results can reflect the regional crowd distribution characteristics, provide data support for urban management and business decision-making, and provide key references for traffic monitoring, public health and other practical applications.

[0003] In some exemplary technologies, crowd counting based on RGB images is widely studied due to its low cost and easy deployment, but in dense crowd, complex lighting or occlusion scenarios, a single modality is difficult to provide reliable features, affecting the counting accuracy. In contrast, cross-modal methods that fuse RGB and thermal images have been proven to significantly improve robustness and accuracy. However, in RGB images and thermal images, the positions of pedestrians in the image often appear to be misaligned to varying degrees, which is mainly due to factors such as spatial installation deviation between cameras, viewing angle difference, and inter-frame time asynchronization. This misalignment is very common in real-world scenarios, especially in dynamic environments or asynchronous acquisition conditions. The alignment error between modalities can interfere with the effective fusion of features, making it difficult for the model to accurately match the corresponding regions of the same target in different modalities, thereby affecting the overall counting accuracy and robustness. The result can lead to inaccurate identification of some crowd areas or introduce irrelevant background interference, reducing the practical application effect of cross-modal fusion methods in complex environments. In addition, due to the modal difference between visible light and infrared light, there will be feature ghosting when fusing cross-modal features. This ghosting usually manifests as multiple blurred or overlapping response regions of the same target in the fused feature map, obscuring the true distribution position of the crowd. Especially in high-density scenarios, this ambiguity can seriously interfere with the learning process of the density map, making it difficult for the model to accurately determine the position and number of each target, thereby reducing the final crowd counting accuracy. In addition, ghosting can also lead to feature redundancy or mislead the model to pay attention to non-target areas, further weakening the discriminative ability and stability of cross-modal fusion.

[0004] Therefore, the existing crowd counting scheme has the problems of cross-modal misalignment and feature ghosting in complex scenarios, resulting in poor accuracy and robustness of crowd counting. SUMMARY

[0005] This invention provides a crowd counting system and method based on cross-modal feature registration and ghosting suppression to address the problems of poor accuracy and robustness in existing crowd counting schemes.

[0006] In a first aspect, embodiments of the present invention provide a crowd counting system based on cross-modal feature registration and ghosting suppression, comprising: a visible light feature extraction module, an infrared light feature extraction module, three cross-modal feature registration modules, four cross-modal ghosting suppression and fusion modules, and a crowd density map estimation module; The infrared light feature extraction module is used to perform hierarchical feature extraction on the infrared light image corresponding to the scene to be counted, and obtain infrared light feature maps from the first level to the fifth level. The visible light feature extraction module is used to perform hierarchical feature extraction on the visible light images corresponding to the scenes to be counted, and obtain the first to fifth level visible light feature maps; Three cross-modal feature registration modules are used sequentially to register the first to third level visible light feature maps based on the first to third level infrared light feature maps, thereby obtaining the first to third level registered visible light feature maps; The first cross-modal ghosting suppression and fusion module and the second cross-modal ghosting suppression and fusion module are used to fuse the second-level and third-level infrared light feature maps and the registered visible light feature maps in turn to obtain first-level fusion features and second-level fusion features. The third and fourth cross-modal ghosting suppression and fusion modules are used to fuse the fourth and fifth level infrared and visible light feature maps, respectively, to obtain the third-level fusion feature and the fourth-level fusion feature. The crowd density map estimation module is used to estimate the crowd density based on the first-level to fourth-level fusion features, and obtain the estimated number of people in the scene to be counted.

[0007] In one possible implementation, the infrared light feature extraction module and the visible light feature extraction module have the same structure, and both the infrared light feature extraction module and the visible light feature extraction module include five convolutional blocks. In the infrared light feature extraction module and the visible light feature extraction module, the output channels of the first to the fifth convolutional blocks are 32, 64, 128, 320, and 512, respectively. The first convolutional block includes three 3×3 convolutional layers, the second convolutional block includes three 3×3 convolutional layers, one max pooling layer, one average pooling layer, and one 3×3 convolutional layer, the third convolutional block includes four 3×3 convolutional layers, one max pooling layer, one average pooling layer, and one 3×3 convolutional layer, the fourth convolutional block includes five 3×3 convolutional layers, one max pooling layer, one average pooling layer, and one 3×3 convolutional layer, and the fifth convolutional block includes six 3×3 convolutional layers, one max pooling layer, one average pooling layer, and one 3×3 convolutional layer. The stride of the pooling layers is 2.

[0008] In one possible implementation, the five convolutional blocks of the infrared light feature extraction module are connected sequentially; The first convolutional block of the infrared light feature extraction module is used to extract features from the infrared light image to obtain the first-level infrared light feature map; the first convolutional block of the visible light feature extraction module is used to extract features from the visible light image corresponding to the scene to be counted to obtain the first-level visible light feature map. The first cross-modal feature registration module is connected to the first convolutional block of the visible light feature extraction module and the first convolutional block of the infrared light feature extraction module. The first cross-modal feature registration module is used to register the first-level visible light feature map based on the first-level infrared light feature map to obtain the first-level registered visible light feature map. The second convolutional block of the infrared light feature extraction module is used to extract features from the first-level infrared light feature map to obtain the second-level infrared light feature map; the second convolutional block of the visible light feature extraction module is connected to the first cross-modal feature registration module, and the second convolutional block of the visible light feature extraction module is used to extract features from the first-level registered visible light feature map to obtain the second-level visible light feature map; The second cross-modal feature registration module is connected to the second convolutional block of the visible light feature extraction module and the second convolutional block of the infrared light feature extraction module. The second cross-modal feature registration module is used to register the second-level visible light feature map based on the second-level infrared light feature map to obtain the second-level registered visible light feature map. The third convolutional block of the infrared light feature extraction module is used to extract features from the second-level infrared light feature map to obtain the third-level infrared light feature map; the third convolutional block of the visible light feature extraction module is connected to the second cross-modal feature registration module, and the third convolutional block of the visible light feature extraction module is used to extract features from the second-level registered visible light feature map to obtain the third-level visible light feature map. The third cross-modal feature registration module is connected to the third convolutional block of the visible light feature extraction module and the third convolutional block of the infrared light feature extraction module. The third cross-modal feature registration module is used to register the third-level visible light feature map based on the third-level infrared light feature map to obtain the third-level registered visible light feature map. The fourth convolutional block of the infrared light feature extraction module is used to extract features from the third-level infrared light feature map to obtain the fourth-level infrared light feature map; the fourth convolutional block of the visible light feature extraction module is connected to the third cross-modal feature registration module, and the fourth convolutional block of the visible light feature extraction module is used to extract features from the third-level registered visible light feature map to obtain the fourth-level visible light feature map. The fifth convolutional block of the infrared light feature extraction module is used to extract features from the fourth-level infrared light feature map to obtain the fifth-level infrared light feature map; the fifth convolutional block of the visible light feature extraction module is used to extract features from the fourth-level visible light feature map to obtain the fifth-level visible light feature map.

[0009] In one possible implementation, the cross-modal feature registration module includes a prior knowledge calculation unit, a deformation field estimation unit, and a feature correction unit; The prior knowledge computation unit includes one subtraction operation, two convolutional layers, one ReLU layer, and one channel concatenation operation; among them, the kernel size of the two convolutional layers is 3×3, the stride is 1, and the output channels are 8 and 1 respectively; The deformation field estimation unit consists of 6 residual blocks and 1 convolutional block; each residual block includes 2 convolutional layers with a kernel size of 3×3, a stride of 1, and an output channel of 65, 1 ReLU layer, and an addition operation; the convolutional block has a kernel size of 1×1, a stride of 1, and an output channel of 2. The feature correction unit includes one normalized original sampling coordinate system generation operation, one displacement field normalization operation, one coordinate mapping addition operation, and one grid_sample feature resampling operation.

[0010] In one possible implementation, a prior knowledge computation unit is used to perform a subtraction operation on the visible light feature map and the infrared light feature map to obtain the registration cost; the registration cost is then processed sequentially through two convolutional layers and one ReLU layer to obtain the processing result; the processing result is then concatenated with the visible light feature map and the infrared light feature map through a channel stitching operation to obtain the prior knowledge. The deformation field estimation unit is used to transform prior knowledge through 6 residual blocks to obtain the feature transformation result; the feature transformation result is then input into a convolutional block for processing to obtain the estimated deformation field. The feature correction unit is used to generate a normalized original sampling coordinate system through a normalized original sampling coordinate system generation operation, normalize the estimated deformation field to obtain the normalized estimated deformation field; add the normalized estimated deformation field to the original sampling coordinate system through a displacement field normalization operation to obtain the deformed sampling coordinates; and perform spatial transformation of the visible light feature map according to the deformed sampling coordinates through a grid_sample feature resampling operation to obtain the registered visible light feature map.

[0011] In one possible implementation, the cross-modal ghosting suppression and fusion module includes: one subtraction operation, one sigmoid activation function, six multiplication operations, two addition operations, two channel concatenation operations, three convolutional blocks, two channel attention calculations, and two spatial attention calculations. The channel attention calculation includes one average pooling, one max pooling, two 1×1 convolutional layers, one sigmoid activation function, one ReLU layer, and one addition operation. The spatial attention calculation includes one 3×3 convolutional layer, one 5×5 convolutional layer, one 7×7 convolutional layer, one channel-dimensional average pooling, one channel-dimensional max pooling, one addition operation, and one sigmoid activation function.

[0012] In one possible implementation, the cross-modal ghosting suppression and fusion module is specifically used for: The visible light feature map and the infrared light feature map are subtracted, and the result of the subtraction operation is input into the Sigmoid activation function to obtain the feature mask; the visible light feature map is multiplied by 1-Mask to obtain the weighted visible light feature map; the infrared light feature map is multiplied by 1+Mask to obtain the weighted infrared light feature map; The weighted visible light feature map and the weighted infrared light feature map are concatenated by channels, and the channel concatenation result is passed in parallel through two convolutional blocks to generate two branch features. The two branch features are then added to the weighted visible light feature map and the weighted infrared light feature map respectively to obtain the updated visible light feature map and infrared light feature map. The updated visible light feature map and infrared light feature map are input into the channel attention module respectively to obtain the channel-weighted infrared light feature map and the channel-weighted infrared light feature map; The channel-weighted infrared light feature map and the channel-weighted infrared light feature map are respectively input into the spatial attention module to obtain the spatial-weighted infrared light feature map and the spatial-weighted infrared light feature map; The spatially weighted infrared feature map and the weighted visible light feature map are added together, and the spatially weighted visible light feature map and the weighted infrared light feature map are added together. The results of the two addition operations are then concatenated and input into the convolutional block to obtain the fused features.

[0013] In one possible implementation, the crowd density map estimation module includes: 6 feature downsampling operations, 6 feature fusion operations, 1 upsampling operation, 3 3×3 convolutional layers, and 3 ReLU layers; wherein the feature downsampling operation includes 1 average pooling operation, 1 3×3 convolutional layer, and 1 ReLU layer; the feature fusion operation includes 1 3×3 convolutional layer and 1 ReLU layer.

[0014] In one possible implementation, the crowd density map estimation module is specifically used for: The first-level fusion feature is downsampled, and the downsampled result is combined with the second-level fusion feature through channel concatenation and feature fusion to obtain the first intermediate fusion feature. The second-level fusion feature is downsampled, and the downsampled result is combined with the third-level fusion feature through channel concatenation and feature fusion to obtain the second intermediate fusion feature. The third-level fusion feature is downsampled, and the downsampled result is combined with the fourth-level fusion feature through channel concatenation and feature fusion to obtain the third intermediate fusion feature. The first intermediate fusion feature is downsampled, and the downsampled result is combined with the second intermediate fusion feature through channel concatenation and feature fusion to obtain the fourth intermediate fusion feature. The second intermediate fusion feature is downsampled, and the downsampled result is combined with the third intermediate fusion feature through channel concatenation and feature fusion to obtain the fifth intermediate fusion feature. The fourth intermediate fusion feature is downsampled, and the downsampled result is combined with the fifth intermediate fusion feature through channel concatenation and feature fusion to obtain the total fusion feature. Population density is estimated based on the total fusion characteristics, resulting in an estimated population density map; The estimated crowd density map is summed pixel by pixel to obtain the estimated number of people in the scene to be counted.

[0015] Secondly, embodiments of the present invention provide a crowd counting method based on cross-modal feature registration and ghosting suppression, applicable to the system in the first aspect or any possible implementation of the first aspect; the method includes: Feature extraction is performed on the infrared and visible light images corresponding to the scene to be counted to obtain the first-level infrared feature map and the first-level visible light feature map; Based on the first-level infrared light feature map, the first-level visible light feature map is registered to obtain the first-level registered visible light feature map. Feature extraction is performed on the first-level infrared light feature map and the first-level registered visible light feature map to obtain the second-level infrared light feature map and the second-level visible light feature map. Based on the second-level infrared light feature map, the second-level visible light feature map is registered to obtain the second-level registered visible light feature map. The first-level fused feature is obtained by fusing the second-level infrared feature map and the visible light feature map after second-level registration. Feature extraction is performed on the second-level infrared light feature map and the second-level registered visible light feature map to obtain the third-level infrared light feature map and the third-level visible light feature map; Based on the third-level infrared light feature map, the third-level visible light feature map is registered to obtain the third-level registered visible light feature map. By fusing the third-level infrared feature map and the third-level registered visible light feature map, a second-level fused feature is obtained; Feature extraction was performed on the third-level infrared light feature map and the third-level registered visible light feature map to obtain the fourth-level infrared light feature map and the fourth-level visible light feature map; By fusing the fourth-level infrared light feature map and the fourth-level visible light feature map, a third-level fused feature is obtained; Feature extraction was performed on the fourth-level infrared light feature map and the fourth-level visible light feature map to obtain the fifth-level infrared light feature map and the fifth-level visible light feature map; By fusing the fifth-level infrared light feature map and the fourth-level visible light feature map, a four-level fused feature map is obtained; Based on the first-level to fourth-level fusion features, the crowd density is estimated to obtain the estimated number of people in the scene to be counted.

[0016] In this embodiment of the invention, by employing a hierarchical visible and infrared feature extraction structure, and in conjunction with the collaborative work of a cross-modal feature registration module and a cross-modal ghosting suppression and fusion module, the system can efficiently process multimodal information at different scales. Specifically, the cross-modal feature registration module aligns visible and infrared features spatially and semantically at each feature level, effectively eliminating feature misalignment caused by sensor differences, viewing angle shifts, and different imaging conditions, thereby providing consistent and high-quality feature input for subsequent multimodal information fusion. The cross-modal ghosting suppression and fusion module, based on the aligned features, suppresses redundant and conflicting parts of different modal information and fully exploits their complementary characteristics to achieve effective feature fusion. By repeating the above registration and fusion operations at multiple feature levels, the system gradually generates multi-level fused features. The crowd density map estimation module obtains the estimated number of people in the scene to be counted based on the multi-level fused features, thereby accurately estimating the number of people while taking into account both global structure and local details. This hierarchical registration and fusion design maintains the complementary advantages of multimodal features while reducing the risk of information loss and fusion imbalance. Therefore, the solution proposed in this application can effectively register different modal features and suppress ghosting features, significantly improving the accuracy and robustness of crowd counting in complex scenes. Attached Figure Description

[0017] Figure 1 This is a structural diagram of a crowd counting system based on cross-modal feature registration and ghosting suppression provided in an embodiment of the present invention; Figure 2 This is a structural diagram of another crowd counting system based on cross-modal feature registration and ghosting suppression provided in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the implementation of the crowd counting method based on cross-modal feature registration and ghosting suppression provided in this embodiment of the invention.

[0018] Figure 4 This is a schematic diagram illustrating the effect of crowd counting based on cross-modal feature registration and ghosting suppression provided in an embodiment of the present invention. Detailed Implementation

[0019] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0020] Figure 1 This is a structural diagram of a crowd counting system based on cross-modal feature registration and ghosting suppression provided in an embodiment of the present invention. Figure 1 As shown, the crowd counting system based on cross-modal feature registration and ghosting suppression includes: visible light feature extraction module 1, infrared light feature extraction module 2, first cross-modal feature registration module 31, second cross-modal feature registration module 32, third cross-modal feature registration module 33, first cross-modal ghosting suppression and fusion module 41, second cross-modal ghosting suppression and fusion module 42, third cross-modal ghosting suppression and fusion module 43, fourth cross-modal ghosting suppression and fusion module 44, and crowd density map estimation module 5.

[0021] See Figure 1 The infrared light feature extraction module 1 is used to perform hierarchical feature extraction on the infrared light image corresponding to the scene to be counted, and obtain infrared light feature maps from the first level to the fifth level. Visible light feature extraction module 2 is used to perform hierarchical feature extraction on the visible light image corresponding to the scene to be counted, and obtain the first to fifth level visible light feature maps; The first cross-modal feature registration module 31, the second cross-modal feature registration module 32, and the third cross-modal feature registration module 33 are used sequentially to register the first to third level visible light feature maps based on the first to third level infrared light feature maps, thereby obtaining the first to third level registered visible light feature maps; The first cross-modal ghosting suppression and fusion module 41 and the second cross-modal ghosting suppression and fusion module 42 are used to fuse the second-level and third-level infrared light feature maps and the registered visible light feature maps in turn to obtain first-level fusion features and second-level fusion features. The third cross-modal ghosting suppression and fusion module 43 and the fourth cross-modal ghosting suppression and fusion module 44 are used to fuse the fourth-level and fifth-level infrared light feature maps and visible light feature maps in turn to obtain the third-level fusion feature and the fourth-level fusion feature; The crowd density map estimation module 5 is used to estimate the crowd density based on the first-level fusion features to the fourth-level fusion features, and obtain the estimated number of people in the scene to be counted.

[0022] In this embodiment, the infrared and visible light images are obtained by simultaneously capturing the same counting scene. In practical applications, the capturing device (e.g., a multispectral camera) simultaneously captures infrared and visible light images of the same scene to be counted, and inputs the infrared and visible light images corresponding to the scene to be counted into a crowd counting system based on cross-modal feature registration and ghosting suppression. Accordingly, the crowd counting system based on cross-modal feature registration and ghosting suppression outputs an estimated number of people in the scene to be counted through the synergistic effect of the above modules.

[0023] It should be noted that before using the crowd counting system based on cross-modal feature registration and ghosting suppression, the initial crowd counting system needs to be trained using multiple visible light and thermal infrared images from the same scene. The loss is calculated using the real crowd density map as supervision, and the Adam optimization algorithm is used to enable the system to learn the optimal parameters until the training results meet the requirements (counting accuracy reaches the required level, the number of training iterations reaches the maximum number of iterations, etc.), thus obtaining the final crowd counting system based on cross-modal feature registration and ghosting suppression.

[0024] Figure 2 This is a structural diagram of another crowd counting system based on cross-modal feature registration and ghosting suppression provided in an embodiment of the present invention. The following is in conjunction with... Figure 1 and Figure 2 This paper provides a detailed explanation of the specific implementation of a crowd counting system based on cross-modal feature registration and ghosting suppression.

[0025] In practical applications, the visible light and infrared images corresponding to the scene to be counted are input into a crowd counting system based on cross-modal feature registration and ghosting suppression. Through the synergistic effect of the aforementioned modules, the crowd counting system based on cross-modal feature registration and ghosting suppression outputs an estimated number of people in the scene to be counted.

[0026] Combination Figure 1 The infrared feature extraction module 1 and the visible light feature extraction module 2 each implement five levels of feature extraction. The difference is that the infrared feature extraction module 1 obtains the first to fifth level infrared feature maps through five consecutive feature extractions. The visible light feature extraction module 2 obtains the features by extracting features from the visible light feature map registered at the previous level in the second to fourth feature extraction operations.

[0027] Specifically, see Figure 2 In one possible implementation, the infrared light feature extraction module 1 and the visible light feature extraction module 2 have the same structure, and both the infrared light feature extraction module 1 and the visible light feature extraction module 2 include five convolutional blocks. In the infrared light feature extraction module 1 and the visible light feature extraction module 2, the output channels of the first to the fifth convolutional blocks are 32, 64, 128, 320, and 512, respectively. The first convolutional block includes three 3×3 convolutional layers, the second convolutional block includes three 3×3 convolutional layers, one max pooling layer, one average pooling layer, and one 3×3 convolutional layer, the third convolutional block includes four 3×3 convolutional layers, one max pooling layer, one average pooling layer, and one 3×3 convolutional layer, the fourth convolutional block includes five 3×3 convolutional layers, one max pooling layer, one average pooling layer, and one 3×3 convolutional layer, and the fifth convolutional block includes six 3×3 convolutional layers, one max pooling layer, one average pooling layer, and one 3×3 convolutional layer. The stride of the pooling layers is 2.

[0028] Specifically, see Figure 2 In one possible implementation, the five convolutional blocks of the infrared light feature extraction module 1 are connected in sequence; The first convolutional block of the infrared light feature extraction module 1 is used to extract features from the infrared light image to obtain a first-level infrared light feature map; the first convolutional block of the visible light feature extraction module 2 is used to extract features from the visible light image corresponding to the scene to be counted to obtain a first-level visible light feature map. The first cross-modal feature registration module 31 is connected to the first convolutional block of the visible light feature extraction module 2 and the first convolutional block of the infrared light feature extraction module 1. The first cross-modal feature registration module 31 is used to register the first-level visible light feature map based on the first-level infrared light feature map to obtain the first-level registered visible light feature map. The second convolutional block of the infrared light feature extraction module 1 is used to extract features from the first-level infrared light feature map to obtain the second-level infrared light feature map; the second convolutional block of the visible light feature extraction module 2 is connected to the first cross-modal feature registration module 31, and the second convolutional block of the visible light feature extraction module 2 is used to extract features from the first-level registered visible light feature map to obtain the second-level visible light feature map. The second cross-modal feature registration module 32 is connected to the second convolutional block of the visible light feature extraction module 2 and the second convolutional block of the infrared light feature extraction module 1. The second cross-modal feature registration module 32 is used to register the second-level visible light feature map based on the second-level infrared light feature map to obtain the second-level registered visible light feature map. The third convolutional block of the infrared light feature extraction module 1 is used to extract features from the second-level infrared light feature map to obtain the third-level infrared light feature map; the third convolutional block of the visible light feature extraction module 2 is connected to the second cross-modal feature registration module 32, and the third convolutional block of the visible light feature extraction module 2 is used to extract features from the second-level registered visible light feature map to obtain the third-level visible light feature map. The third cross-modal feature registration module 33 is connected to the third convolutional block of the visible light feature extraction module 2 and the third convolutional block of the infrared light feature extraction module 1. The third cross-modal feature registration module 33 is used to register the third-level visible light feature map based on the third-level infrared light feature map to obtain the third-level registered visible light feature map. The fourth convolutional block of the infrared light feature extraction module 1 is used to extract features from the third-level infrared light feature map to obtain the fourth-level infrared light feature map; the fourth convolutional block of the visible light feature extraction module 2 is connected to the third cross-modal feature registration module 33, and the fourth convolutional block of the visible light feature extraction module 2 is used to extract features from the third-level registered visible light feature map to obtain the fourth-level visible light feature map. The fifth convolutional block of the infrared light feature extraction module 1 is used to extract features from the fourth-level infrared light feature map to obtain the fifth-level infrared light feature map; the fifth convolutional block of the visible light feature extraction module 2 is used to extract features from the fourth-level visible light feature map to obtain the fifth-level visible light feature map.

[0029] Understandably, the infrared light feature extraction module 1 and the visible light feature extraction module 2 can extract feature maps of five levels from infrared light images and visible light images, achieving a comprehensive and rich representation from local details to global semantics, laying a solid foundation for the implementation of subsequent schemes.

[0030] Combination Figure 1 The first cross-modal feature registration module 31, the second cross-modal feature registration module 32, and the third cross-modal feature registration module 33 sequentially register the first to third level visible light feature maps based on the first to third level infrared light feature maps, thereby obtaining the first to third level registered visible light feature maps.

[0031] In one possible implementation, see Figure 2 The aforementioned cross-modal feature registration module includes a prior knowledge calculation unit, a deformation field estimation unit, and a feature correction unit. The prior knowledge computation unit includes one subtraction operation, two convolutional layers, one ReLU layer, and one channel concatenation operation; among them, the kernel size of the two convolutional layers is 3×3, the stride is 1, and the output channels are 8 and 1 respectively; The deformation field estimation unit consists of 6 residual blocks and 1 convolutional block; each residual block includes 2 convolutional layers with a kernel size of 3×3, a stride of 1, and an output channel of 65, 1 ReLU layer, and an addition operation; the convolutional block has a kernel size of 1×1, a stride of 1, and an output channel of 2. The feature correction unit includes one normalized original sampling coordinate system generation operation, one displacement field normalization operation, one coordinate mapping addition operation, and one grid_sample feature resampling operation.

[0032] In one possible implementation, the prior knowledge computation unit is used to perform a subtraction operation on the visible light feature map and the infrared light feature map to obtain the registration cost; the registration cost is processed sequentially through two convolutional layers and one ReLU layer to obtain the processing result; the processing result is combined with the visible light feature map and the infrared light feature map through a channel concatenation operation to obtain the prior knowledge. The deformation field estimation unit is used to transform prior knowledge through 6 residual blocks to obtain the feature transformation result; the feature transformation result is then input into a convolutional block for processing to obtain the estimated deformation field. The feature correction unit is used to generate a normalized original sampling coordinate system through a normalized original sampling coordinate system generation operation, normalize the estimated deformation field to obtain the normalized estimated deformation field; add the normalized estimated deformation field to the original sampling coordinate system through a displacement field normalization operation to obtain the deformed sampling coordinates; and perform spatial transformation of the visible light feature map according to the deformed sampling coordinates through a grid_sample feature resampling operation to obtain the registered visible light feature map.

[0033] Combination Figure 2 Specifically, the prior knowledge calculation unit of the first cross-modal feature registration module 31 performs a subtraction operation on the first-level visible light feature map and the first-level infrared light feature map to obtain the first registration cost; the first registration cost is processed sequentially through two convolutional layers and one ReLU layer to obtain the first processing result; the first processing result is combined with the first-level visible light feature map and the first-level infrared light feature map through channel splicing to obtain the first prior knowledge. The deformation field estimation unit of the first cross-modal feature registration module 31 performs feature transformation on the first prior knowledge through 6 residual blocks to obtain the first feature transformation result; and inputs the first feature transformation result into a convolutional block for processing to obtain the first estimated deformation field. The feature correction unit of the first cross-modal feature registration module 31 generates a normalized original sampling coordinate system through a normalized original sampling coordinate system generation operation, normalizes the first estimated deformation field, and obtains the normalized first estimated deformation field; through a displacement field normalization operation, the normalized first estimated deformation field is added to the original sampling coordinate system to obtain the deformed sampling coordinates; through a grid_sample feature resampling operation, the first visible light feature map is spatially transformed according to the deformed sampling coordinates to obtain the visible light feature map after the first-level registration.

[0034] Accordingly, the prior knowledge calculation unit of the second cross-modal feature registration module 32 obtains the second registration cost by performing a subtraction operation on the second-level visible light feature map and the second-level infrared light feature map; the second registration cost is processed sequentially through two convolutional layers and one ReLU layer to obtain the second processing result; the second processing result is combined with the second-level visible light feature map and the second-level infrared light feature map by channel concatenation to obtain the second prior knowledge. The deformation field estimation unit of the second cross-modal feature registration module 32 performs feature transformation on the second prior knowledge through 6 residual blocks to obtain the second feature transformation result; the second feature transformation result is input into a convolutional block for processing to obtain the second estimated deformation field; The feature correction unit of the second cross-modal feature registration module 32 generates a normalized original sampling coordinate system through a normalized original sampling coordinate system generation operation, normalizes the second estimated deformation field, and obtains the normalized second estimated deformation field; through a displacement field normalization operation, the normalized second estimated deformation field is added to the original sampling coordinate system to obtain the deformed sampling coordinates; through a grid_sample feature resampling operation, the second visible light feature map is spatially transformed according to the deformed sampling coordinates to obtain the visible light feature map after the second-level registration.

[0035] Accordingly, the prior knowledge calculation unit of the third cross-modal feature registration module 33 performs a subtraction operation on the third-level visible light feature map and the third-level infrared light feature map to obtain the third registration cost; the third registration cost is processed sequentially through two convolutional layers and one ReLU layer to obtain the third processing result; the third processing result is combined with the third-level visible light feature map and the third-level infrared light feature map through channel splicing to obtain the third prior knowledge. The deformation field estimation unit of the third cross-modal feature registration module 33 performs feature transformation on the third prior knowledge through 6 residual blocks to obtain the third feature transformation result; the third feature transformation result is input into a convolutional block for processing to obtain the third estimated deformation field; The feature correction unit of the third cross-modal feature registration module 33 generates a normalized original sampling coordinate system through a normalized original sampling coordinate system generation operation, normalizes the third estimated deformation field, and obtains the normalized third estimated deformation field; through a displacement field normalization operation, the normalized third estimated deformation field is added to the original sampling coordinate system to obtain the deformed sampling coordinates; through a grid_sample feature resampling operation, the third visible light feature map is spatially transformed according to the deformed sampling coordinates to obtain the visible light feature map after the third-level registration.

[0036] Understandably, by using three cross-modal feature registration modules to register the first three levels of visible light feature maps, spatial and semantic alignment of visible light and infrared light features can be achieved at each feature level. This effectively eliminates feature misalignment caused by sensor differences, viewing angle shifts, and different imaging conditions, thus providing a data foundation for accurate alignment for subsequent feature fusion.

[0037] Combination Figure 1 The first cross-modal ghosting suppression and fusion module 41 and the second cross-modal ghosting suppression and fusion module 42 sequentially fuse the second-level and third-level infrared light feature maps and the registered visible light feature maps to obtain first-level fusion features and second-level fusion features; the third cross-modal ghosting suppression and fusion module 43 and the fourth cross-modal ghosting suppression and fusion module 44 sequentially fuse the fourth-level and fifth-level infrared light feature maps and visible light feature maps to obtain third-level fusion features and fourth-level fusion features.

[0038] In one possible implementation, see Figure 2 The aforementioned cross-modal ghosting suppression and fusion module includes: 1 subtraction operation, 1 sigmoid activation function, 6 multiplication operations, 2 addition operations, 2 channel concatenation operations, 3 convolutional blocks, 2 channel attention calculations, and 2 spatial attention calculations. Among them, the channel attention calculation includes 1 average pooling, 1 max pooling, 2 1×1 convolutional layers, 1 sigmoid activation function, 1 ReLU layer, and 1 addition operation; the spatial attention includes 1 3×3 convolutional layer, 1 5×5 convolutional layer, 1 7×7 convolutional layer, 1 channel-dimensional average pooling, 1 channel-dimensional max pooling, 1 addition operation subunit, and 1 sigmoid activation function.

[0039] In one possible implementation, the aforementioned cross-modal ghosting suppression and fusion module is specifically used for: The visible light feature map and the infrared light feature map are subtracted, and the result of the subtraction operation is input into the Sigmoid activation function to obtain the feature mask; the visible light feature map is multiplied by 1-Mask to obtain the weighted visible light feature map; the infrared light feature map is multiplied by 1+Mask to obtain the weighted infrared light feature map; The weighted visible light feature map and the weighted infrared light feature map are concatenated by channels, and the channel concatenation result is passed in parallel through two convolutional blocks to generate two branch features. The two branch features are then added to the weighted visible light feature map and the weighted infrared light feature map respectively to obtain the updated visible light feature map and infrared light feature map. The updated visible light feature map and infrared light feature map are input into the channel attention module respectively to obtain the channel-weighted infrared light feature map and the channel-weighted infrared light feature map; The channel-weighted infrared light feature map and the channel-weighted infrared light feature map are respectively input into the spatial attention module to obtain the spatial-weighted infrared light feature map and the spatial-weighted infrared light feature map; The spatially weighted infrared feature map and the weighted visible light feature map are added together, and the spatially weighted visible light feature map and the weighted infrared light feature map are added together. The results of the two addition operations are then concatenated and input into the convolutional block to obtain the fused features.

[0040] Combination Figure 2 Specifically, the first cross-modal ghosting suppression and fusion module 41 is used for: The visible light feature map after second-level registration is subtracted from the second-level infrared feature map, and the result of the subtraction operation is input into the Sigmoid activation function to obtain the feature mask; the visible light feature map after second-level registration is multiplied with 1-Mask to obtain the weighted visible light feature map; the second-level infrared feature map is multiplied with 1+Mask to obtain the weighted infrared feature map; The weighted visible light feature map and the weighted infrared light feature map are concatenated by channels, and the channel concatenation result is passed in parallel through two convolutional blocks to generate two branch features. The two branch features are then added to the weighted visible light feature map and the weighted infrared light feature map respectively to obtain the updated visible light feature map and infrared light feature map. The updated visible light feature map and the second-level infrared light feature map are input into the channel attention module to obtain the channel-weighted infrared light feature map and the channel-weighted infrared light feature map, respectively. The channel-weighted infrared light feature map and the channel-weighted infrared light feature map are respectively input into the spatial attention module to obtain the spatial-weighted infrared light feature map and the spatial-weighted infrared light feature map; The spatially weighted infrared feature map and the weighted visible light feature map are added together, and the spatially weighted visible light feature map and the weighted infrared light feature map are added together. The results of the two addition operations are then concatenated and input into the convolutional block to obtain the first-level fused feature.

[0041] Combination Figure 2 Specifically, the second cross-modal ghosting suppression and fusion module 42 is used for: The visible light feature map after third-level registration is subtracted from the third-level infrared feature map, and the result of the subtraction operation is input into the Sigmoid activation function to obtain the feature mask; the visible light feature map after third-level registration is multiplied with 1-Mask to obtain the weighted visible light feature map; the third-level infrared feature map is multiplied with 1+Mask to obtain the weighted infrared feature map; The weighted visible light feature map and the weighted infrared light feature map are concatenated by channels, and the channel concatenation result is passed in parallel through two convolutional blocks to generate two branch features. The two branch features are then added to the weighted visible light feature map and the weighted infrared light feature map respectively to obtain the updated visible light feature map and infrared light feature map. The updated visible light feature map and the third-level infrared light feature map are input into the channel attention module to obtain the channel-weighted infrared light feature map and the channel-weighted infrared light feature map, respectively. The channel-weighted infrared light feature map and the channel-weighted infrared light feature map are respectively input into the spatial attention module to obtain the spatial-weighted infrared light feature map and the spatial-weighted infrared light feature map; The spatially weighted infrared feature map and the weighted visible light feature map are added together, and the spatially weighted visible light feature map and the weighted infrared light feature map are added together. The results of the two addition operations are then concatenated and input into the convolutional block to obtain the secondary fusion feature.

[0042] Combination Figure 2 Specifically, the third cross-modal ghosting suppression and fusion module 43 is used for: Subtract the fourth-level visible light feature map from the fourth-level infrared light feature map, and input the result of the subtraction operation into the Sigmoid activation function to obtain the feature mask; multiply the fourth-level visible light feature map with 1-Mask to obtain the weighted visible light feature map; multiply the fourth-level infrared light feature map with 1+Mask to obtain the weighted infrared light feature map; The weighted visible light feature map and the weighted infrared light feature map are concatenated by channels, and the channel concatenation result is passed in parallel through two convolutional blocks to generate two branch features. The two branch features are then added to the weighted visible light feature map and the weighted infrared light feature map respectively to obtain the updated visible light feature map and infrared light feature map. The updated visible light feature map and the fourth-level infrared light feature map are input into the channel attention module to obtain the channel-weighted infrared light feature map and the channel-weighted infrared light feature map, respectively. The channel-weighted infrared light feature map and the channel-weighted infrared light feature map are respectively input into the spatial attention module to obtain the spatial-weighted infrared light feature map and the spatial-weighted infrared light feature map; The spatially weighted infrared feature map and the weighted visible light feature map are added together, and the spatially weighted visible light feature map and the weighted infrared light feature map are added together. The results of the two addition operations are then concatenated and input into a convolutional block to obtain the three-level fused features.

[0043] Combination Figure 2 Specifically, the fourth cross-modal ghosting suppression and fusion module 44 is used for: The fifth-level visible light feature map and the fifth-level infrared light feature map are subtracted, and the result of the subtraction operation is input into the Sigmoid activation function to obtain the feature mask; the fifth-level visible light feature map is multiplied by 1-Mask to obtain the weighted visible light feature map; the fifth-level infrared light feature map is multiplied by 1+Mask to obtain the weighted infrared light feature map. The weighted visible light feature map and the weighted infrared light feature map are concatenated by channels, and the channel concatenation result is passed in parallel through two convolutional blocks to generate two branch features. The two branch features are then added to the weighted visible light feature map and the weighted infrared light feature map respectively to obtain the updated visible light feature map and infrared light feature map. The updated visible light feature map and the fourth-level infrared light feature map are input into the channel attention module to obtain the channel-weighted infrared light feature map and the channel-weighted infrared light feature map, respectively. The channel-weighted infrared light feature map and the channel-weighted infrared light feature map are respectively input into the spatial attention module to obtain the spatial-weighted infrared light feature map and the spatial-weighted infrared light feature map; The spatially weighted infrared feature map and the weighted visible light feature map are added together, and the spatially weighted visible light feature map and the weighted infrared light feature map are added together. The results of the two addition operations are then concatenated and input into a convolutional block to obtain a four-level fused feature.

[0044] Understandably, the four cross-modal ghosting suppression and fusion modules, based on the aligned features, suppress the redundant and conflicting parts of different modal information and fully explore their complementary characteristics to achieve effective feature fusion and obtain multi-level fused features.

[0045] Combination Figure 1 The crowd density estimation module 5 estimates the crowd density based on the first-level to fourth-level fusion features, and obtains the estimated number of people in the scene to be counted.

[0046] In one possible implementation, see Figure 2 The aforementioned population density map estimation module 5 includes: 6 feature downsampling operations, 6 feature fusion operations, 1 upsampling operation, 3 3×3 convolutional layers, and 3 ReLU layers; wherein, the feature downsampling operation includes 1 average pooling operation, 1 3×3 convolutional layer, and 1 ReLU layer; the feature fusion operation includes 1 3×3 convolutional layer and 1 ReLU layer.

[0047] In one possible implementation, see Figure 2 Crowd density map estimation module 5, specifically used for: First-level fusion features Perform feature downsampling and combine the downsampling results with the secondary fusion features. Perform channel splicing and feature fusion operations to obtain the first intermediate fused feature. ; Secondary fusion features Perform feature downsampling and combine the downsampling results with the three-level fused features. Perform channel splicing and feature fusion operations to obtain the second intermediate fused feature. ; Three-level fusion features Perform feature downsampling and combine the downsampling results with the fourth-level fused features. Perform channel splicing and feature fusion operations to obtain the third intermediate fused feature. ; The first intermediate fusion feature Perform feature downsampling and fuse the downsampling results with the second intermediate feature. Perform channel splicing and feature fusion operations to obtain the fourth intermediate fused feature. ; The second intermediate fusion feature Perform feature downsampling and fuse the downsampling results with the third intermediate feature. Perform channel splicing and feature fusion operations to obtain the fifth intermediate fused feature. ; The fourth intermediate fusion feature Perform feature downsampling and fuse the downsampling results with the fifth intermediate feature. Perform channel splicing and feature fusion operations to obtain the total fused features. ; Population density is estimated based on the total fusion characteristics, resulting in an estimated population density map; The estimated crowd density map is summed pixel by pixel to obtain the estimated number of people in the scene to be counted.

[0048] Specifically, the first-level fusion features Feature downsampling is performed, which involves sequentially passing the sample through an average pooling layer with a 2×2 kernel and a stride of 2, a convolutional layer with a 3×3 kernel and a stride of 1, and a ReLU activation layer. The result is then fused with the secondary feature. Channel concatenation is performed, followed by feature fusion, which involves passing the feature through a 3×3 convolutional layer with a stride of 1, and then a ReLU activation layer to obtain the first intermediate fused feature. .

[0049] Furthermore, the secondary fusion features Feature downsampling is performed, which involves sequentially passing the data through an average pooling layer with a 2×2 kernel and a stride of 2, a convolutional layer with a 3×3 kernel and a stride of 1, and a ReLU activation layer. The result is then fused with the three-level features. Channel concatenation is performed, followed by feature fusion, which involves passing the feature through a 3×3 convolutional layer with a stride of 1, and then a ReLU activation layer to obtain the second intermediate fused feature. .

[0050] Furthermore, the three-level fusion features Feature downsampling is performed, which involves sequentially passing the data through an average pooling layer with a 2×2 kernel and a stride of 2, a convolutional layer with a 3×3 kernel and a stride of 1, and a ReLU activation layer. The resulting data is then fused with the fourth-level features. Channel concatenation is performed, followed by feature fusion, which involves passing the feature through a 3×3 convolutional layer with a stride of 1, and then a ReLU activation layer to obtain the third intermediate fused feature. .

[0051] Furthermore, the first intermediate fusion feature Feature downsampling is performed, which involves sequentially passing the data through an average pooling layer with a 2×2 kernel and a stride of 2, a convolutional layer with a 3×3 kernel and a stride of 1, and a ReLU activation layer. The resulting data is then fused with the second intermediate feature. Channel concatenation is performed, followed by feature fusion, which involves passing the feature through a 3×3 convolutional layer with a stride of 1, and then a ReLU activation layer to obtain the fourth intermediate fused feature. .

[0052] Furthermore, the second intermediate fusion feature Feature downsampling is performed, which involves sequentially passing the data through an average pooling layer with a 2×2 kernel and a stride of 2, a convolutional layer with a 3×3 kernel and a stride of 1, and a ReLU activation layer. The resulting data is then fused with the third intermediate feature. Channel concatenation is performed, followed by feature fusion, which involves passing the feature through a 3×3 convolutional layer with a stride of 1, and then a ReLU activation layer to obtain the fifth intermediate fused feature. .

[0053] Furthermore, the fourth intermediate fusion feature Feature downsampling is performed, which involves sequentially passing the data through an average pooling layer with a 2×2 kernel and a stride of 2, a convolutional layer with a 3×3 kernel and a stride of 1, and a ReLU activation layer. The resulting data is then fused with the fifth intermediate feature. Channel concatenation is performed, followed by feature fusion, which involves passing the data through a 3×3 convolutional layer with a stride of 1, and then a ReLU activation layer to obtain the final fused features. .

[0054] Furthermore, the overall integration features An upsampling operation is performed, and the data is sequentially fed into a convolutional layer with a kernel size of 3×3, a stride of 1, and 128 output channels, a ReLU activation layer, a convolutional layer with a kernel size of 3×3, a stride of 1, and 64 output channels, a ReLU activation layer, and a convolutional layer with a kernel size of 3×3, a stride of 1, and 1 output channel, and a ReLU activation layer to obtain an estimated crowd density map. The estimated crowd density map is then summed pixel by pixel to obtain the estimated number of people in the scene to be counted.

[0055] Understandably, the crowd density map estimation module 5 obtains the estimated number of people in the scene to be counted based on multi-level fusion features, thereby accurately estimating the number of people while taking into account both the global structure and local details.

[0056] In this embodiment, by employing a hierarchical visible and infrared feature extraction structure, coupled with the collaborative work of a cross-modal feature registration module and a cross-modal ghosting suppression and fusion module, the system can efficiently process multimodal information at different scales. Specifically, the cross-modal feature registration module aligns visible and infrared features spatially and semantically at each feature level, effectively eliminating feature misalignment caused by sensor differences, viewing angle shifts, and varying imaging conditions, thus providing consistent and high-quality feature input for subsequent multimodal information fusion. The cross-modal ghosting suppression and fusion module, based on the aligned features, suppresses redundant and conflicting parts of different modal information and fully exploits their complementary characteristics to achieve effective feature fusion. By repeating the above registration and fusion operations at multiple feature levels, the system gradually generates multi-level fused features. The crowd density map estimation module obtains the estimated number of people in the scene to be counted based on the multi-level fused features, thereby accurately estimating the number of people while taking into account both global structure and local details. This hierarchical registration and fusion design maintains the complementary advantages of multimodal features while reducing the risk of information loss and fusion imbalance. Therefore, the solution in this embodiment can effectively register different modal features and suppress ghosting features, significantly improving the accuracy and robustness of crowd counting in complex scenes.

[0057] The following are method embodiments of the present invention. For details not described in detail, please refer to the corresponding system embodiments described above.

[0058] This embodiment provides a crowd counting method based on cross-modal feature registration and ghosting suppression, which can be applied to the crowd counting system based on cross-modal feature registration and ghosting suppression shown in any of the above embodiments.

[0059] Figure 3 This is a flowchart illustrating the implementation of the crowd counting method based on cross-modal feature registration and ghosting suppression provided in this embodiment of the invention. See also... Figure 3 Crowd counting methods based on cross-modal feature registration and ghosting suppression include: Step 601: Extract features from the infrared and visible light images corresponding to the scene to be counted to obtain the first-level infrared feature map and the first-level visible light feature map. Step 602: Based on the first-level infrared light feature map, register the first-level visible light feature map to obtain the first-level registered visible light feature map; Step 603: Extract features from the first-level infrared light feature map and the first-level registered visible light feature map to obtain the second-level infrared light feature map and the second-level visible light feature map. Step 604: Based on the second-level infrared light feature map, register the second-level visible light feature map to obtain the second-level registered visible light feature map; Step 605: Fuse the second-level infrared feature map and the second-level registered visible light feature map to obtain the first-level fused feature map; Step 606: Extract features from the second-level infrared feature map and the second-level registered visible light feature map to obtain the third-level infrared feature map and the third-level visible light feature map. Step 607: Based on the third-level infrared light feature map, register the third-level visible light feature map to obtain the third-level registered visible light feature map; Step 608: Fuse the third-level infrared feature map and the third-level registered visible light feature map to obtain the second-level fused feature map; Step 609: Extract features from the third-level infrared light feature map and the third-level registered visible light feature map to obtain the fourth-level infrared light feature map and the fourth-level visible light feature map. Step 610: Fuse the fourth-level infrared light feature map and the fourth-level visible light feature map to obtain the third-level fused feature map; Step 611: Extract features from the fourth-level infrared light feature map and the fourth-level visible light feature map respectively to obtain the fifth-level infrared light feature map and the fifth-level visible light feature map; Step 612: Fuse the fifth-level infrared light feature map and the fourth-level visible light feature map to obtain the four-level fused feature map; Step 613: Estimate the crowd density based on the first-level to fourth-level fusion features to obtain the estimated number of people in the scene to be counted.

[0060] Specifically, in one possible implementation, step 601 specifically includes: Step 6011: Input the visible light image and the infrared light image into the first convolutional block of the visible light feature extraction module 2 and the first convolutional block of the infrared feature extraction module 1, respectively. They are then passed through three convolutional layers with a kernel size of 3×3, a stride of 1, and an output channel of 32, respectively, to obtain the first-level visible light feature map and the first-level infrared light feature map with 32 channels.

[0061] Specifically, in one possible implementation, step 602 specifically includes steps 6021 to 6023.

[0062] Step 6021: Perform a subtraction operation on the first-level visible light feature map and the first-level infrared light feature map to obtain the first registration cost; process the first registration cost through two convolutional layers and one ReLU layer in sequence to obtain the first processing result; perform channel splicing processing on the first processing result and the first-level visible light feature map and the first-level infrared light feature map to obtain the first prior knowledge.

[0063] Step 6022: Perform feature transformation on the first prior knowledge through 6 residual blocks to obtain the first feature transformation result; input the first feature transformation result into a convolutional block for processing to obtain the first estimated deformation field.

[0064] Step 6023: Generate a normalized original sampling coordinate system through the normalized original sampling coordinate system generation operation, normalize the first estimated deformation field to obtain the normalized first estimated deformation field; add the normalized first estimated deformation field to the original sampling coordinate system through the displacement field normalization operation to obtain the deformed sampling coordinates; perform spatial transformation on the first visible light feature map according to the deformed sampling coordinates through the grid_sample feature resampling operation to obtain the visible light feature map after the first-level registration.

[0065] It should be noted that the specific implementation of steps 604 and 607 is similar to that of step 602, the difference being that the visible light feature maps of different levels are registered, which will not be elaborated on here.

[0066] Specifically, in one possible implementation, step 603 specifically includes: Step 6031: Input the first-level registered visible light feature map and the first-level infrared light feature map into the second convolutional block of the visible light feature extraction module and the second convolutional block of the infrared feature extraction module, respectively. They are sequentially passed through three convolutional layers with a kernel size of 3×3, a stride of 1, and 64 output channels; a max pooling layer with a kernel size of 2×2 and a stride of 2; an average pooling layer with a kernel size of 2×2 and a stride of 2; and one convolutional layer with a kernel size of 3×3, a stride of 1, and 64 output channels, to obtain a second-level visible light feature map and a second-level infrared light feature map with 64 channels.

[0067] Specifically, in one possible implementation, step 605 specifically includes steps 6051 to 6055.

[0068] Step 6051: Subtract the second-level registered visible light feature map from the second-level infrared feature map, and input the result of the subtraction operation into the Sigmoid activation function to obtain the feature mask; multiply the second-level registered visible light feature map with 1-Mask to obtain the weighted visible light feature map; multiply the second-level infrared feature map with 1+Mask to obtain the weighted infrared feature map; Step 6052: Perform channel concatenation on the weighted visible light feature map and the weighted infrared light feature map, and generate two branch features in parallel through two convolutional blocks; perform addition operations on the two branch features with the weighted visible light feature map and the weighted infrared light feature map respectively to obtain the updated visible light feature map and infrared light feature map; Step 6053: Input the updated visible light feature map and the second-level infrared light feature map into the channel attention module respectively to obtain the channel-weighted infrared light feature map and the channel-weighted infrared light feature map.

[0069] Specifically, in the channel attention module, the features are first passed through an adaptive average pooling layer and an adaptive max pooling layer to obtain two global channel description vectors. Each vector is then passed through a convolutional layer with a kernel size of 1×1, a stride of 1, and 16 output channels, followed by a ReLU activation layer, and then another convolutional layer with a kernel size of 1×1, a stride of 1, and 64 output channels, resulting in two sets of channel weight mappings. The two sets of mappings are then element-wise summed and normalized using a sigmoid activation function to obtain the final channel attention weights. Finally, the visible light feature map and the infrared light feature map are multiplied by their respective channel attention weights to obtain the channel-weighted infrared light feature map and the channel-weighted infrared light feature map, respectively.

[0070] Step 6054: Input the channel-weighted infrared light feature map and the channel-weighted infrared light feature map into the spatial attention module respectively to obtain the spatial-weighted infrared light feature map and the spatial-weighted infrared light feature map.

[0071] Specifically, the channel-weighted infrared feature map and the channel-weighted infrared feature map are input into the spatial attention module. The input features are first subjected to global average pooling and global max pooling in the channel dimension to obtain two single-channel feature maps, which are used to describe the response intensity of different channels in spatial location. Then, the two feature maps are concatenated in the channel dimension to form a dual-channel feature representation containing average and maximum information. This representation is fed in parallel with three convolutional layers: a 7×7 kernel with a stride of 1, a 5×5 kernel with a stride of 1, and a 3×3 kernel with a stride of 1. The outputs of these three layers are added together, and the result is input into the Sigmoid activation function to generate spatial attention weights in the range [0,1]. Then, the channel-weighted infrared feature map and the channel-weighted infrared feature map are multiplied by their respective spatial attention weights to obtain the spatially weighted infrared feature map and the spatially weighted infrared feature map.

[0072] Step 6055: Perform addition operations on the spatially weighted infrared light feature map and the weighted visible light feature map, and then concatenate the two addition results into a convolutional block to obtain the first-level fused feature. .

[0073] It should be noted that the specific implementation methods of steps 608, 610 and 612 are similar to those of step 605. The difference lies in the fusion of infrared light feature maps and visible light feature maps of different levels, which will not be elaborated on here.

[0074] Specifically, in one possible implementation, step 606 specifically includes: Step 6061: Input the second-level registered visible light feature map and the second-level infrared light feature map into the third convolutional block of the visible light feature extraction module and the third convolutional block of the infrared feature extraction module, respectively. They are sequentially passed through four convolutional layers with a kernel size of 3×3, a stride of 1, and 128 output channels, a max pooling layer with a kernel size of 2×2 and a stride of 2, an average pooling layer with a kernel size of 2×2 and a stride of 2, and one convolutional layer with a kernel size of 3×3, a stride of 1, and 128 output channels to obtain a third-level visible light feature map and a third-level infrared light feature map with 128 channels.

[0075] Specifically, in one possible implementation, step 609 specifically includes: Step 6091: Input the third-level registered visible light feature map and the third-level infrared light feature map into the fourth convolutional block of the visible light feature extraction module and the fourth convolutional block of the infrared feature extraction module, respectively. They are sequentially passed through 5 convolutional layers with a kernel size of 3×3, a stride of 1, and an output channel of 320; a max pooling layer with a kernel size of 2×2 and a stride of 2; an average pooling layer with a kernel size of 2×2 and a stride of 2; and 1 convolutional layer with a kernel size of 3×3, a stride of 1, and an output channel of 320, to obtain a fourth-level visible light feature map and a fourth-level infrared light feature map with 320 channels.

[0076] Specifically, in one possible implementation, step 611 specifically includes: Step 6111: Input the fourth-level visible light feature map and the fourth-level infrared light feature map into the fifth convolutional block of the visible light feature extraction module and the fifth convolutional block of the infrared feature extraction module, respectively. They are sequentially passed through 6 convolutional layers with a kernel size of 3×3, a stride of 1, and an output channel of 512; a max pooling layer with a pooling kernel of 2×2 and a stride of 2; an average pooling layer with a pooling kernel of 2×2 and a stride of 2; and a convolutional layer with a kernel size of 3×3, a stride of 1, and an output channel of 512, to obtain the fifth-level visible light feature map and the fifth-level infrared light feature map with 512 channels.

[0077] Optionally, in one possible implementation, step 613 includes steps 6131 to 6137.

[0078] Step 6131: Merge the first-level features Perform feature downsampling and combine the downsampling results with the secondary fusion features. Perform channel splicing and feature fusion operations to obtain the first intermediate fused feature. ; Step 6132: Merge the secondary features Perform feature downsampling and combine the downsampling results with the three-level fused features. Perform channel splicing and feature fusion operations to obtain the second intermediate fused feature. ; Step 6133: Merge the three-level features Perform feature downsampling and combine the downsampling results with the fourth-level fused features. Perform channel splicing and feature fusion operations to obtain the third intermediate fused feature. ; Step 6133: Merge the first intermediate features Perform feature downsampling and fuse the downsampling results with the second intermediate feature. Perform channel splicing and feature fusion operations to obtain the fourth intermediate fused feature. ; Step 6134: Merge the second intermediate features Perform feature downsampling and fuse the downsampling results with the third intermediate feature. Perform channel splicing and feature fusion operations to obtain the fifth intermediate fused feature. ; Step 6135: Merge the fourth intermediate features Perform feature downsampling and fuse the downsampling results with the fifth intermediate feature. Perform channel splicing and feature fusion operations to obtain the total fused features. ; Step 6136: Estimate the population density based on the total fusion features to obtain the estimated population density map; Step 6137: Add the estimated crowd density map pixel by pixel to obtain the estimated number of people in the scene to be counted.

[0079] The specific implementation methods of steps 6131 to 6135 are the same. Specifically, the feature downsampling operation sequentially passes through an average pooling layer with a 2×2 kernel and a stride of 2, a convolutional layer with a 3×3 kernel and a stride of 1, and a ReLU activation layer to obtain the downsampling result. Similarly, the feature fusion operation sequentially passes through a convolutional layer with a 3×3 kernel and a stride of 1, and a ReLU activation layer to obtain the fused features.

[0080] Specifically, in one possible implementation, step 6136 above includes: The overall integration features An upsampling operation is performed and the data is sequentially fed into a convolutional layer with a kernel size of 3×3, a stride of 1, and 128 output channels, a ReLU activation layer, a convolutional layer with a kernel size of 3×3, a stride of 1, and 64 output channels, a ReLU activation layer, a convolutional layer with a kernel size of 3×3, a stride of 1, and 1 output channel, and a ReLU activation layer to obtain the estimated crowd density map.

[0081] In this embodiment, by employing a hierarchical visible and infrared feature extraction structure, coupled with the collaborative work of a cross-modal feature registration module and a cross-modal ghosting suppression and fusion module, the system can efficiently process multimodal information at different scales. Specifically, the cross-modal feature registration module aligns visible and infrared features spatially and semantically at each feature level, effectively eliminating feature misalignment caused by sensor differences, viewing angle shifts, and varying imaging conditions, thus providing consistent and high-quality feature input for subsequent multimodal information fusion. The cross-modal ghosting suppression and fusion module, based on the aligned features, suppresses redundant and conflicting parts of different modal information and fully exploits their complementary characteristics to achieve effective feature fusion. By repeating the above registration and fusion operations at multiple feature levels, the system gradually generates multi-level fused features. The crowd density map estimation module obtains the estimated number of people in the scene to be counted based on the multi-level fused features, thereby accurately estimating the number of people while taking into account both global structure and local details. This hierarchical registration and fusion design maintains the complementary advantages of multimodal features while reducing the risk of information loss and fusion imbalance. Therefore, the solution in this embodiment can effectively register different modal features and suppress ghosting features, significantly improving the accuracy and robustness of crowd counting in complex scenes.

[0082] Figure 4 This is a schematic diagram illustrating the effect of crowd counting based on cross-modal feature registration and ghosting suppression provided in an embodiment of the present invention. See also Figure 4 When using the crowd counting system based on cross-modal feature registration and ghosting suppression provided by this invention to count crowds, the visible light image and thermal infrared image of the crowd in the target scene are input into the established cross-modal feature registration and ghosting suppression crowd counting system. The trained and optimized crowd counting system is used to estimate the crowd density, and the estimated crowd density map is added pixel by pixel to obtain the corresponding crowd counting result.

[0083] This invention addresses the misalignment problem between visible light and infrared images in existing cross-modal crowd counting methods. It proposes a cross-modal feature registration and ghosting suppression crowd counting system and method, effectively resolving this misalignment issue. By inputting visible light and infrared images into two branches respectively, a feature extraction module extracts features of different sizes from each modality. A feature registration module estimates the deformation field using visible light and infrared features, corrects the visible light features, and obtains registered visible light features. A cross-modal ghosting suppression and fusion module suppresses feature redundancy in the two modalities and then performs attention fusion to obtain fused features. The fused feature map is then processed by a convolutional neural network model to generate an estimated crowd density map. Because the feature extraction module can extract features of different sizes from different modalities; the cross-modal feature registration module can register visible light features; and the cross-modal ghosting suppression and fusion module can suppress cross-modal ghosting information and density map regression, the counting network can more accurately estimate the number of people in the image.

[0084] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A crowd counting system based on cross-modal feature registration and ghosting suppression, characterized in that, The system includes: a visible light feature extraction module, an infrared light feature extraction module, three cross-modal feature registration modules, four cross-modal ghosting suppression and fusion modules, and a crowd density map estimation module; The infrared light feature extraction module is used to perform hierarchical feature extraction on the infrared light image corresponding to the scene to be counted, and obtain infrared light feature maps of the first to fifth levels. The visible light feature extraction module is used to perform hierarchical feature extraction on the visible light image corresponding to the scene to be counted, and obtain the first to fifth level visible light feature maps; The three cross-modal feature registration modules are used in sequence to register the first to third level visible light feature maps based on the first to third level infrared light feature maps, so as to obtain the first to third level registered visible light feature maps; The first cross-modal ghosting suppression and fusion module and the second cross-modal ghosting suppression and fusion module are used to fuse the second-level and third-level infrared light feature maps and the registered visible light feature maps in turn to obtain first-level fusion features and second-level fusion features. The third and fourth cross-modal ghosting suppression and fusion modules are used to fuse the fourth and fifth level infrared and visible light feature maps, respectively, to obtain the third-level fusion feature and the fourth-level fusion feature. The crowd density map estimation module is used to estimate the crowd density based on the first-level to fourth-level fusion features to obtain the estimated number of people in the scene to be counted.

2. The crowd counting system based on cross-modal feature registration and ghosting suppression according to claim 1, characterized in that, The infrared light feature extraction module and the visible light feature extraction module have the same structure, and both the infrared light feature extraction module and the visible light feature extraction module include five convolutional blocks; In the infrared light feature extraction module and the visible light feature extraction module, the output channels of the first to the fifth convolutional blocks are 32, 64, 128, 320, and 512 respectively. The first convolutional block includes three 3×3 convolutional layers, the second convolutional block includes three 3×3 convolutional layers, one max pooling layer, one average pooling layer, and one 3×3 convolutional layer, the third convolutional block includes four 3×3 convolutional layers, one max pooling layer, one average pooling layer, and one 3×3 convolutional layer, the fourth convolutional block includes five 3×3 convolutional layers, one max pooling layer, one average pooling layer, and one 3×3 convolutional layer, and the fifth convolutional block includes six 3×3 convolutional layers, one max pooling layer, one average pooling layer, and one 3×3 convolutional layer. The stride of each pooling layer is 2.

3. The crowd counting system based on cross-modal feature registration and ghosting suppression according to claim 2, characterized in that, The five convolutional blocks of the infrared light feature extraction module are connected in sequence; The first convolutional block of the infrared light feature extraction module is used to extract features from the infrared light image to obtain a first-level infrared light feature map; the first convolutional block of the visible light feature extraction module is used to extract features from the visible light image corresponding to the scene to be counted to obtain a first-level visible light feature map. The first cross-modal feature registration module is connected to the first convolutional block of the visible light feature extraction module and the first convolutional block of the infrared light feature extraction module. The first cross-modal feature registration module is used to register the first-level visible light feature map based on the first-level infrared light feature map to obtain the first-level registered visible light feature map. The second convolutional block of the infrared light feature extraction module is used to extract features from the first-level infrared light feature map to obtain a second-level infrared light feature map; the second convolutional block of the visible light feature extraction module is connected to the first cross-modal feature registration module, and the second convolutional block of the visible light feature extraction module is used to extract features from the first-level registered visible light feature map to obtain a second-level visible light feature map; The second cross-modal feature registration module is connected to the second convolutional block of the visible light feature extraction module and the second convolutional block of the infrared light feature extraction module. The second cross-modal feature registration module is used to register the second-level visible light feature map based on the second-level infrared light feature map to obtain the second-level registered visible light feature map. The third convolutional block of the infrared light feature extraction module is used to extract features from the second-level infrared light feature map to obtain a third-level infrared light feature map; the third convolutional block of the visible light feature extraction module is connected to the second cross-modal feature registration module, and the third convolutional block of the visible light feature extraction module is used to extract features from the second-level registered visible light feature map to obtain a third-level visible light feature map; The third cross-modal feature registration module is connected to the third convolutional block of the visible light feature extraction module and the third convolutional block of the infrared light feature extraction module. The third cross-modal feature registration module is used to register the third-level visible light feature map based on the third-level infrared light feature map to obtain the third-level registered visible light feature map. The fourth convolutional block of the infrared light feature extraction module is used to extract features from the third-level infrared light feature map to obtain a fourth-level infrared light feature map; the fourth convolutional block of the visible light feature extraction module is connected to the third cross-modal feature registration module, and the fourth convolutional block of the visible light feature extraction module is used to extract features from the third-level registered visible light feature map to obtain a fourth-level visible light feature map. The fifth convolutional block of the infrared light feature extraction module is used to extract features from the fourth-level infrared light feature map to obtain a fifth-level infrared light feature map; the fifth convolutional block of the visible light feature extraction module is used to extract features from the fourth-level visible light feature map to obtain a fifth-level visible light feature map.

4. The crowd counting system based on cross-modal feature registration and ghosting suppression according to claim 1, characterized in that, The cross-modal feature registration module includes a prior knowledge calculation unit, a deformation field estimation unit, and a feature correction unit. The prior knowledge computation unit includes one subtraction operation, two convolutional layers, one ReLU layer, and one channel concatenation operation; wherein, the kernel size of the two convolutional layers is 3×3, the stride is 1, and the output channels are 8 and 1 respectively; The deformation field estimation unit includes 6 residual blocks and 1 convolutional block; each residual block includes 2 convolutional layers with a kernel size of 3×3, a stride of 1, and an output channel of 65, 1 ReLU layer, and an addition operation; the convolutional block has a kernel size of 1×1, a stride of 1, and an output channel of 2. The feature correction unit includes one normalized original sampling coordinate system generation operation, one displacement field normalization operation, one coordinate mapping addition operation, and one grid_sample feature resampling operation.

5. The crowd counting system based on cross-modal feature registration and ghosting suppression according to claim 4, characterized in that, The prior knowledge calculation unit is used to perform a subtraction operation on the visible light feature map and the infrared light feature map to obtain the registration cost; the registration cost is processed sequentially through the two convolutional layers and the one ReLU layer to obtain the processing result; the processing result is combined with the visible light feature map and the infrared light feature map through a channel concatenation operation to obtain the prior knowledge. The deformation field estimation unit is used to perform feature transformation on the prior knowledge through the six residual blocks to obtain the feature transformation result; and input the feature transformation result into the one convolutional block for processing to obtain the estimated deformation field. The feature correction unit is used to generate a normalized original sampling coordinate system through a normalized original sampling coordinate system generation operation, normalize the estimated deformation field to obtain a normalized estimated deformation field; add the normalized estimated deformation field to the original sampling coordinate system through a displacement field normalization operation to obtain the deformed sampling coordinates; and perform a grid_sample feature resampling operation to spatially transform the visible light feature map according to the deformed sampling coordinates to obtain the registered visible light feature map.

6. The crowd counting system based on cross-modal feature registration and ghosting suppression according to claim 1, characterized in that, The cross-modal ghosting suppression and fusion module includes: 1 subtraction operation, 1 sigmoid activation function, 6 multiplication operations, 2 addition operations, 2 channel concatenation operations, 3 convolutional blocks, 2 channel attention calculations, and 2 spatial attention calculations. The channel attention calculation includes 1 average pooling, 1 max pooling, 2 1×1 convolutional layers, 1 sigmoid activation function, 1 ReLU layer, and 1 addition operation. The spatial attention calculation includes 1 3×3 convolutional layer, 1 5×5 convolutional layer, 1 7×7 convolutional layer, 1 channel-dimensional average pooling, 1 channel-dimensional max pooling, 1 addition operation, and 1 sigmoid activation function.

7. The crowd counting system based on cross-modal feature registration and ghosting suppression according to claim 6, characterized in that, The cross-modal ghosting suppression and fusion module is specifically used for: The visible light feature map and the infrared light feature map are subtracted, and the result of the subtraction operation is input into the Sigmoid activation function to obtain the feature mask; the visible light feature map is multiplied by 1-Mask to obtain the weighted visible light feature map; the infrared light feature map is multiplied by 1+Mask to obtain the weighted infrared light feature map; The weighted visible light feature map and the weighted infrared light feature map are concatenated by channels, and the channel concatenation result is passed in parallel through two convolutional blocks to generate two branch features. The two branch features are added to the weighted visible light feature map and the weighted infrared light feature map respectively to obtain the updated visible light feature map and infrared light feature map. The updated visible light feature map and infrared light feature map are input into the channel attention module respectively to obtain the channel-weighted infrared light feature map and the channel-weighted infrared light feature map; The channel-weighted infrared light feature map and the channel-weighted infrared light feature map are respectively input into the spatial attention module to obtain the spatial weighted infrared light feature map and the spatial weighted infrared light feature map; The spatially weighted infrared feature map and the weighted visible light feature map are added together, and the spatially weighted visible light feature map and the weighted infrared light feature map are added together. The results of the two addition operations are then concatenated and input into the convolutional block to obtain the fused features.

8. The crowd counting system based on cross-modal feature registration and ghosting suppression according to claim 1, characterized in that, The crowd density map estimation module includes: 6 feature downsampling operations, 6 feature fusion operations, 1 upsampling operation, 3 3×3 convolutional layers, and 3 ReLU layers; wherein, the feature downsampling operation includes 1 average pooling operation, 1 3×3 convolutional layer, and 1 ReLU layer; the feature fusion operation includes 1 3×3 convolutional layer and 1 ReLU layer.

9. The crowd counting system based on cross-modal feature registration and ghosting suppression according to claim 8, characterized in that, The crowd density map estimation module is specifically used for: The first-level fusion feature is downsampled, and the downsampling result is combined with the second-level fusion feature through channel concatenation and feature fusion to obtain the first intermediate fusion feature. The secondary fusion feature is downsampled, and the downsampling result is combined with the tertiary fusion feature through channel concatenation and feature fusion to obtain the second intermediate fusion feature. The third-level fusion feature is downsampled, and the downsampling result is combined with the fourth-level fusion feature through channel splicing and feature fusion to obtain the third intermediate fusion feature. The first intermediate fusion feature is downsampled, and the downsampled result is combined with the second intermediate fusion feature through channel concatenation and feature fusion to obtain the fourth intermediate fusion feature. The second intermediate fusion feature is downsampled, and the downsampling result is combined with the third intermediate fusion feature through channel splicing and feature fusion to obtain the fifth intermediate fusion feature. The fourth intermediate fusion feature is downsampled, and the downsampling result is combined with the fifth intermediate fusion feature through channel concatenation and feature fusion to obtain the total fusion feature. Based on the total fusion features, the population density is estimated to obtain the estimated population density map; The estimated crowd density map is summed pixel by pixel to obtain the estimated number of people in the scene to be counted.

10. A crowd counting method based on cross-modal feature registration and ghosting suppression, characterized in that, A method applied to a crowd counting system based on cross-modal feature registration and ghosting suppression; the method includes: Feature extraction is performed on the infrared and visible light images corresponding to the scene to be counted to obtain the first-level infrared feature map and the first-level visible light feature map; Based on the first-level infrared light feature map, the first-level visible light feature map is registered to obtain the first-level registered visible light feature map. Feature extraction is performed on the first-level infrared light feature map and the first-level registered visible light feature map to obtain the second-level infrared light feature map and the second-level visible light feature map. Based on the second-level infrared light feature map, the second-level visible light feature map is registered to obtain the second-level registered visible light feature map. The second-level infrared feature map and the second-level registered visible light feature map are fused to obtain the first-level fused feature map. Feature extraction is performed on the second-level infrared light feature map and the second-level registered visible light feature map to obtain the third-level infrared light feature map and the third-level visible light feature map; Based on the third-level infrared light feature map, the third-level visible light feature map is registered to obtain the third-level registered visible light feature map. By fusing the third-level infrared feature map and the third-level registered visible light feature map, a second-level fused feature is obtained; Feature extraction is performed on the third-level infrared light feature map and the third-level registered visible light feature map respectively to obtain the fourth-level infrared light feature map and the fourth-level visible light feature map; By fusing the fourth-level infrared light feature map and the fourth-level visible light feature map, a three-level fused feature is obtained; Feature extraction is performed on the fourth-level infrared light feature map and the fourth-level visible light feature map respectively to obtain the fifth-level infrared light feature map and the fifth-level visible light feature map; By fusing the fifth-level infrared feature map and the fourth-level visible light feature map, a four-level fused feature is obtained; Based on the first-level to fourth-level fusion features, the crowd density is estimated to obtain the estimated number of people in the scene to be counted.

Images