A multimodal crowd counting method and system

Abstract

The present invention provides a multimodal crowd counting method and system. The method acquires images and extracts image features of the images, wherein the images include visible light modality images and thermal infrared modality images; the image features of the visible light modality images and thermal infrared modality images are layered through the spatial frequency attention in the spatial frequency guidance module to generate a target attention map; the image features of the visible light modality images and thermal infrared modality images at each level and the target attention map are fused through an adaptive dynamic fusion module to obtain various multimodal fusion features; the multimodal fusion features are passed through a multi-scale progressive fusion module to generate a predicted density map, wherein the multi-scale progressive fusion module is composed of a void space pyramid pooling and a Swin Transformer Block. The modules designed in the above steps can effectively solve the problems of large error and low counting accuracy in multimodal crowd counting.

Description

Technical Field

[0001] The present invention belongs to the technical field of crowd counting, and in particular relates to a multimodal crowd counting method and system. Background Art

[0002] In the multimodal (visible light-thermal infrared modality) crowd counting task, the first challenge to be solved is the multimodal fusion problem. This is because visible light modality images and thermal infrared modality images provide very different information under different lighting conditions.

[0003] By using the Sobel operator to extract gradient information from the two modal images, we found that in high-illuminance scenes, the visible light modality can easily distinguish between crowd areas and background noise, while the thermal infrared modality has relatively less texture detail information. In low-illuminance scenes, the visible light modality cannot well identify the crowd area, but the thermal infrared modality can easily determine the crowd's location based on human body temperature. Therefore, the multimodal fusion problem is to leverage the different imaging characteristics of visible light and thermal infrared images to fully explore the complementary information between multimodal data, thereby generating a "new image" that can fully and accurately describe the actual scene, thereby improving the model's counting accuracy.

[0004] Most existing crowd counting methods address this problem by directly fusing features from different modalities without any prior knowledge. While this strategy has achieved some success, it ignores the complex interactions between the two modalities. This can lead to a lack of fusion targets when faced with complex background interference, hindering the fusion of crowd areas. Furthermore, fusion results can fluctuate under varying lighting conditions, failing to fully capture the dynamic changes in light intensity in real-world scenarios. Summary of the Invention

[0005] Based on this, an embodiment of the present invention provides a multimodal crowd counting method and system, which aims to solve the problems of large errors and low counting accuracy in multimodal crowd counting in the prior art.

[0006] A first aspect of an embodiment of the present invention provides a multimodal crowd counting method, the method comprising:

[0007] Acquiring an image and extracting image features of the image, wherein the image includes a visible light modality image and a thermal infrared modality image;

[0008] Generate a target attention map by applying the image features of the visible light modality image and the thermal infrared modality image layer by layer through the spatial frequency attention in the spatial frequency guidance module;

[0009] fusing the image features of the visible light modality image and the thermal infrared modality image at each level and the target attention map through an adaptive dynamic fusion module to obtain multimodal fusion features;

[0010] Each multimodal fusion feature is passed through a multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features, and generate a predicted density map, wherein the multi-scale progressive fusion module is composed of void space pyramid pooling and Swin Transformer Block.

[0011] Furthermore, the spatial-frequency attention is composed of spatial-domain attention and frequency-domain attention. The processing results of the spatial-domain attention and the frequency-domain attention are fused to obtain the final target attention map.

[0012] Furthermore, in the step of fusing the processing results of the spatial domain attention and the frequency domain attention to obtain the final target attention map, the implementation process of the spatial domain attention is expressed as:

[0013]

[0014] in, represents the input features, represents 1×1 convolution, represents the Sigmoid activation function, represents the generated spatial attention weight, represents element-wise multiplication, Represents the features enhanced by spatial domain attention;

[0015] The implementation process of frequency domain attention is expressed as:

[0016]

[0017] in, represents the two-dimensional discrete cosine transform function, represents the fully connected layer, ReLU represents the rectified linear unit, represents the generated channel attention weight, Represents the features after frequency domain attention enhancement;

[0018] The fusion implementation process is expressed as:

[0019]

[0020] in, represents the element-by-element addition operation, Indicates that after 1×1 convolution, batch normalization and ReLU corrected linear unit operation, Indicates that after 1×1 convolution, batch normalization and Sigmoid activation operations, represents the target attention map.

[0021] Furthermore, the implementation process of the spatial frequency guidance module is expressed as follows:

[0022]

[0023] in, represents spatial frequency attention, represents the generated visible light modality attention map, represents the generated thermal infrared modality attention map, Indicates the maximum value operation of each element. The final generated guide Attention map for fusing hierarchical multimodal features.

[0024] Furthermore, the adaptive dynamic fusion module consists of three parts: foreground enhancement, cross attention, and dynamic weighting. The implementation process of the foreground enhancement is expressed as follows:

[0025]

[0026] in, represents element-wise addition operation, represents the element-wise multiplication operation, Represents the visible light mode After the enhancement of the level, Indicates thermal infrared mode Enhanced features of the hierarchy;

[0027] The implementation process of the cross attention is expressed as:

[0028]

[0029]

[0030] Among them, K r 、V r , Q r are three vectors, represents 1×1 convolution, and Respectively represent the transposed 、 vector, Indicates the number of feature channels after reduction in order to reduce computational overhead, represents the Softmax activation function, and Respectively represent the visible light modal features and thermal infrared modal features after interactive learning;

[0031] The implementation process of the dynamic weighting is expressed as follows:

[0032]

[0033] in, It represents the aggregated features after channel splicing of two different modal features. and They represent the feature fusion weights of the visible light modality and thermal infrared modality generated by 1×1 convolution and Softmax activation function respectively.

[0034] Furthermore, in the step of fusing the image features of the visible light modality image and the thermal infrared modality image at each level and the target attention map through an adaptive dynamic fusion module to obtain each multimodal fusion feature, the fusion process is expressed as:

[0035]

[0036] in, Indicates the backbone network Multimodal fusion features of hierarchical output.

[0037] Furthermore, the multimodal fusion features are passed through a multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features. In the step of generating a prediction density map, the generation process is expressed as follows:

[0038]

[0039] in, represents the atrous spatial pyramid pooling, Indicates 2x upsampling, Represents the concatenation of the dimensions of two tensors, represents Swin Transformer Block, Indicates the backbone network Multimodal fusion feature map of layer output.

[0040] A second aspect of an embodiment of the present invention provides a multimodal crowd counting system for implementing the multimodal crowd counting method described in the first aspect, the system comprising:

[0041] an acquisition module, configured to acquire images and extract image features of the images, wherein the images include visible light modality images and thermal infrared modality images;

[0042] a target attention map generation module, configured to generate a target attention map by applying the image features of the visible light modality image and the thermal infrared modality image layer by layer through the spatial frequency attention in the spatial frequency guidance module;

[0043] a fusion module, configured to fuse the image features of the visible light modality image and the thermal infrared modality image at each level and the target attention map through an adaptive dynamic fusion module to obtain multimodal fusion features;

[0044] The prediction density map generation module is used to pass each multimodal fusion feature through the multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features and generate a prediction density map, wherein the multi-scale progressive fusion module is composed of void spatial pyramid pooling and Swin Transformer Block.

[0045] A third aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal crowd counting method provided in the first aspect.

[0046] A fourth aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multimodal crowd counting method provided in the first aspect is implemented.

[0047] A multimodal crowd counting method and system provided in an embodiment of the present invention acquires images and extracts image features of the images, where the images include visible light modality images and thermal infrared modality images; the image features of the visible light modality images and thermal infrared modality images are layered through the spatial frequency attention in the spatial frequency guidance module to generate a target attention map; the image features of the visible light modality images and thermal infrared modality images at each level and the target attention map are fused through an adaptive dynamic fusion module to obtain various multimodal fusion features; the multimodal fusion features are passed through a multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features, and generate a predicted density map, wherein the multi-scale progressive fusion module is composed of void spatial pyramid pooling and Swin Transformer Block. The modules designed in the above steps can effectively solve the problems of large error and low counting accuracy in multimodal crowd counting. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of a multimodal crowd counting method according to the first embodiment of the present invention;

[0049] Figure 2 It is a structural diagram of the multimodal crowd counting method;

[0050] Figure 3 This is the spatial frequency attention structure diagram;

[0051] Figure 4 This is a schematic diagram of the spatial frequency guidance module structure;

[0052] Figure 5 It is a schematic diagram of the structure of the adaptive dynamic fusion module;

[0053] Figure 6 This is a structural block diagram of a multimodal crowd counting system provided in Example 2 of the present invention;

[0054] Figure 7 This is a structural block diagram of an electronic device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0055] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The drawings illustrate several embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present invention.

[0056] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0058] Example 1

[0059] According to an embodiment of the present invention, an embodiment of a multimodal crowd counting method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0060] In this embodiment, a multimodal crowd counting method is provided, which can be used in electronic devices, such as computers. Figure 1 and Figure 2 , Figure 1 FIG2 shows a flow chart of a multimodal crowd counting method according to the first embodiment of the present invention. Figure 2 Schematic diagram of the structure of the multimodal crowd counting method, which specifically includes steps S01 to S04.

[0061] Step S01 : acquiring an image and extracting image features of the image, wherein the image includes a visible light modality image and a thermal infrared modality image.

[0062] Among them, image features can be extracted through deep learning-based methods. Specifically, through structures such as convolutional layers, pooling layers, and fully connected layers, the feature representation of the image can be automatically learned.

[0063] Step S02 : The image features of the visible light modality image and the thermal infrared modality image are layered by layer through the spatial frequency attention in the spatial frequency guidance module to generate a target attention map.

[0064] Specifically, since different modalities have different target perception capabilities, for example, visible light images are more likely to provide low-frequency information in image processing, while thermal infrared images are better at capturing high-frequency information related to temperature distribution, a spatial frequency guidance module (SFGM) is proposed to use the attention mechanism to generate attention maps based on different modal characteristics to further guide the fusion of multimodal information. According to the different ways of applying attention weights, visual attention can be divided into two categories: spatial attention and channel attention. Spatial attention is to learn the spatial position information of the target, while channel attention focuses on effective task-related features by learning the importance of different channels in the feature map. In order to improve the generation quality of attention maps of different modalities, the respective advantages of these two types of attention are combined, and spatial-frequency attention (SFA) is proposed in SFGM, which includes spatial domain attention (SDA) acting on the feature space dimension and frequency domain attention (FDA) acting on the feature channel dimension. Its structure is as follows Figure 3 shown.

[0065] SDA allows the model to focus more on the crowd area in the image rather than other non-task targets, thereby suppressing the interference of complex backgrounds. To further reduce the computational overhead of the algorithm, SDA does not choose the classic visual spatial attention, but is implemented based on 1×1 convolution and Sigmoid function. The process can be expressed as follows:

[0066]

[0067] in, represents the input features, represents 1×1 convolution, represents the Sigmoid activation function, represents the generated spatial attention weight, represents element-wise multiplication, Represents the features enhanced by spatial domain attention.

[0068] In traditional visual channel attention, a global average pooling (GAP) operation is generally used to generate a global feature representation for each channel, thereby providing basic information for channel attention calculation. However, since the number of channels in a feature map is generally large, the mean of different channels may be the same based on GAP, but in fact the semantic content information they represent may be different, which will result in the inability to capture diverse feature information. Although the GAP-based channel attention retains the lowest frequency information during the feature learning process, it discards other equally important high-frequency information. Therefore, in order to overcome the above defects, FDA was introduced, which uses discrete cosine transform (DCT) to extract different frequency components from different channels from a frequency domain perspective to obtain richer crowd target feature information. The process can be expressed as follows:

[0069]

[0070] in, represents the two-dimensional discrete cosine transform function, represents the fully connected layer, ReLU represents the rectified linear unit, represents the generated channel attention weight, Represents the features after frequency domain attention enhancement.

[0071] In order to fully combine the respective advantages of spatial attention and channel attention, the features enhanced by SDA and FDA are fused to generate a hybrid attention map to further highlight the modal characteristics. The process can be expressed as follows:

[0072]

[0073] in, represents element-wise addition operation, Indicates that after 1×1 convolution, batch normalization and ReLU corrected linear unit operation, Indicates that after 1×1 convolution, batch normalization and Sigmoid activation operations, represents the target attention map.

[0074] Please note that Figure 4 and Figure 5 , Figure 4 This is a schematic diagram of the spatial frequency guidance module structure. Figure 5 Schematic diagram of the adaptive dynamic fusion module structure. In order to learn different modal characteristics, SFGM uses the hierarchical features of visible light images and red-hot images. and ( ) Use SFA to generate visible light modal attention maps and thermal infrared modal attention maps respectively, and use the element-by-element maximum operation to extract the most significant information of the two modal attentions to suppress the interference of background noise and provide the location information of the crowd area for the subsequent multimodal fusion process, thereby improving the attention to the crowd characteristics. The specific process of SFGM can be expressed as follows:

[0075]

[0076] in, represents spatial frequency attention, represents the generated visible light modality attention map, represents the generated thermal infrared modality attention map, Indicates the maximum value operation of each element. The final generated guide Attention map for fusing hierarchical multimodal features.

[0077] In step S03 , the image features of the visible light modality image and the thermal infrared modality image at each level and the target attention map are fused through an adaptive dynamic fusion module to obtain multimodal fusion features.

[0078] Specifically, crowd counting scenarios can be roughly divided into two categories based on illumination intensity: low-light and high-light scenarios. Different modalities play different roles in these two scenarios due to their unique characteristics. Therefore, an Adaptive Dynamic Fusion Module (ADFM) is proposed to dynamically fuse multimodal information based on the specific scenario to further improve fusion quality. To illustrate the rationale for dynamic fusion, the Sobel operator is used to extract gradient information from visible light and thermal infrared images in these two scenarios and visualize the results. It can be seen that in high-light scenarios, the visible light modality provides richer texture detail information than the thermal infrared modality. However, in low-light scenarios, the visible light modality has lower imaging quality and provides limited information. However, the thermal infrared modality, unaffected by illumination, provides relatively richer high-frequency gradient information. Therefore, the primary function of the ADFM is to dynamically learn the complementary information between the gradients of the two modalities. It consists of three components: foreground enhancement, cross-attention, and dynamic weighting.

[0079] In foreground enhancement, the main method is to generate a guided fusion attention map through SFGM. Hierarchical features of visible light mode and thermal infrared mode respectively and ( ) to enhance the features in order to be able to fuse them under the premise of knowing the location information of the crowd area. The process can be expressed as follows:

[0080]

[0081] in, represents the element-by-element addition operation, represents the element-wise multiplication operation, Represents the visible light mode After the enhanced features of the level, Indicates thermal infrared mode Enhanced features of the hierarchy.

[0082] In order to capture the global feature correlation between multimodal data, Cross-Attention is introduced to allow information to flow between different modalities. First, the enhanced visible light modal features are respectively and thermal infrared modal characteristics Mapping into three vectors in attention: key (Key, K), value (Value, V) and query (Query, Q), the process can be simplified as follows:

[0083]

[0084] In addition, unlike the self-attention mechanism, in the calculation process of cross-attention, the K vector and V vector come from the same modality, while the Q vector comes from another modality. The calculation process is as follows:

[0085]

[0086] Among them, Kr, Vr, and Qr are three vectors. represents 1×1 convolution, and Respectively represent the transposed 、 vector, Indicates the number of feature channels after reduction in order to reduce computational overhead, represents the Softmax activation function, and They respectively represent the visible light modal features and thermal infrared modal features after interactive learning.

[0087] Because different modalities provide different gradient information in different scenes, different fusion weights need to be generated according to the specific scene in the multimodal fusion process to fully learn the complementary characteristics between the modalities. For example, in high-illuminance scenes, the visible light modality should be the main one and the thermal infrared modality should be the auxiliary one, but in low-illuminance scenes, the opposite is true. Therefore, the visible light modality features learned based on cross-attention are and thermal infrared modal characteristics To dynamically weight the multimodal fusion process, the process can be expressed as follows:

[0088]

[0089] in, It represents the aggregated features after channel splicing of two different modal features. and They represent the feature fusion weights of the visible light modality and thermal infrared modality generated by 1×1 convolution and Softmax activation function respectively.

[0090] More specifically, in order to improve the quality of multimodal fusion and avoid the layer-by-layer attenuation of modal information, a design similar to the Residual Network (ResNet) is adopted. The fusion process is expressed as:

[0091]

[0092] in, Indicates the backbone network Multimodal fusion features of hierarchical output.

[0093] In step S04, each multimodal fusion feature is passed through a multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features, and generate a predicted density map, wherein the multi-scale progressive fusion module is composed of a dilated spatial pyramid pooling and a Swin Transformer Block.

[0094] Specifically, in real scenes, target objects may appear in different sizes under different environments, viewing angles, and other conditions. How to effectively identify and process target objects that appear in different sizes has always been a common problem in the field of computer vision - the multi-scale problem of targets, which is no exception in the multimodal crowd counting task. Therefore, ConvNeXt with hierarchical feature representation was selected as the backbone network. For an image with an input resolution of H×W, the sizes of the feature maps output by its four stages are , , and Although ConvNeXt's hierarchical structure helps capture multi-scale crowd information, in real-world scenarios, the scale of crowd objects often varies continuously, and a hierarchical backbone network alone cannot effectively handle this scale variation. To address this, we introduced the Multi-scale Progressive Fusion Module (MPFM) to better learn crowd object information at different scales.

[0095] MPFM is mainly composed of Atrous Spatial Pyramid Pooling (ASPP) and Swin Transformer Block. First, the fusion features of the output of different layers of the backbone network are learned through the dilated convolution of different dilation rates in ASPP. , to further learn target information at more diverse scales. Second, the sliding window attention in the Swin Transformer Block enables the network to have a global receptive field, effectively helping the model understand and fuse features from different image regions. Finally, due to the significant differences in features extracted from different layers of the backbone network, for example, shallow networks primarily capture image details (edges, textures, etc.), while deep networks include semantic information for a global understanding of the image. Crowd counting tasks require both semantic information to identify target objects in an image and detailed information to distinguish the boundaries of different target objects.

[0096] Therefore, MPFM designs a gradual fusion process from deep features to shallow features, aiming to fully learn semantic information and detail information to effectively improve the quality of multimodal fusion output feature maps. The specific process of MIAM can be expressed as follows:

[0097]

[0098] in, represents the atrous spatial pyramid pooling, Indicates 2x upsampling, Represents the concatenation of the dimensions of two tensors, represents Swin Transformer Block, Indicates the backbone network The multimodal fusion feature map of the layer output, that is, I 1 is the predicted density map.

[0099] In summary, the multimodal crowd counting method in the above embodiment of the present invention obtains images and extracts image features of the images, where the images include visible light modality images and thermal infrared modality images; the image features of the visible light modality images and thermal infrared modality images are layered through the spatial frequency attention in the spatial frequency guidance module to generate a target attention map; the image features of the visible light modality images and thermal infrared modality images at each level and the target attention map are fused through an adaptive dynamic fusion module to obtain various multimodal fusion features; the multimodal fusion features are passed through a multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features to generate a predicted density map, wherein the multi-scale progressive fusion module is composed of void spatial pyramid pooling and Swin Transformer Block. Through the modules designed in the above steps, the problems of large error and low counting accuracy in multimodal crowd counting can be effectively solved.

[0100] Example 2

[0101] See also Figure 6 , Figure 6 This is a block diagram of a multimodal crowd counting system provided in Example 2 of the present invention. The multimodal crowd counting system 200 is used to implement the above-mentioned embodiments and preferred implementations, and the details that have been described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0102] Specifically, the multimodal crowd counting system 200 includes: an acquisition module 21, a target attention map generation module 22, a fusion module 23, and a prediction density map generation module 24, wherein:

[0103] An acquisition module 21 is configured to acquire images and extract image features of the images, wherein the images include visible light modality images and thermal infrared modality images;

[0104] The target attention map generation module 22 is used to generate a target attention map by layer-by-layer using the spatial frequency attention in the spatial frequency guidance module for the image features of the visible light modality image and the thermal infrared modality image. The spatial frequency attention is composed of spatial domain attention and frequency domain attention. The processing results of the spatial domain attention and the processing results of the frequency domain attention are fused to obtain the final target attention map. The implementation process of the spatial domain attention is expressed as follows:

[0105]

[0106] in, represents the input features, represents 1×1 convolution, represents the Sigmoid activation function, represents the generated spatial attention weight, represents element-wise multiplication, Represents the features enhanced by spatial domain attention;

[0107] The implementation process of frequency domain attention is expressed as:

[0108]

[0109] in, represents the two-dimensional discrete cosine transform function, represents the fully connected layer, ReLU represents the rectified linear unit, represents the generated channel attention weight, Represents the features after frequency domain attention enhancement;

[0110] The fusion implementation process is expressed as:

[0111]

[0112] in, represents element-wise addition operation, Indicates that after 1×1 convolution, batch normalization and ReLU corrected linear unit operation, Indicates that after 1×1 convolution, batch normalization and Sigmoid activation operations, Represents the target attention map. In addition, the implementation process of the spatial frequency guidance module is expressed as:

[0113]

[0114] in, represents spatial frequency attention, represents the generated visible light modality attention map, represents the generated thermal infrared modality attention map, Indicates the maximum value operation of each element. The final generated guide Attention map for fusing hierarchical multimodal features;

[0115] The fusion module 23 is used to fuse the image features of the visible light modality image and the thermal infrared modality image at each level and the target attention map through an adaptive dynamic fusion module to obtain multimodal fusion features. The adaptive dynamic fusion module consists of three parts: foreground enhancement, cross attention, and dynamic weighting. The implementation process of the foreground enhancement is expressed as follows:

[0116]

[0117] in, represents element-wise addition operation, represents the element-wise multiplication operation, Represents the visible light mode After the enhancement of the level, Indicates thermal infrared mode Enhanced features of the hierarchy;

[0118] The implementation process of the cross attention is expressed as:

[0119]

[0120]

[0121] Among them, K r 、V r , Q r are three vectors, represents 1×1 convolution, and Respectively represent the transposed 、 vector, Indicates the number of feature channels after reduction in order to reduce computational overhead, represents the Softmax activation function, and Respectively represent the visible light modal features and thermal infrared modal features after interactive learning;

[0122] The implementation process of the dynamic weighting is expressed as follows:

[0123]

[0124] in, It represents the aggregated features after channel splicing of two different modal features. and They represent the feature fusion weights of the visible light modality and thermal infrared modality generated by 1×1 convolution and Softmax activation function respectively. In addition, the fusion process is expressed as:

[0125]

[0126] in, Indicates the backbone network Multimodal fusion features of hierarchical output;

[0127] The prediction density map generation module 24 is used to pass each multimodal fusion feature through the multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features, and generate a prediction density map. The multi-scale progressive fusion module is composed of void space pyramid pooling and Swin Transformer Block. The generation process is expressed as follows:

[0128]

[0129] in, represents the atrous spatial pyramid pooling, Indicates 2x upsampling, Represents the concatenation of the dimensions of two tensors, represents Swin Transformer Block, Indicates the backbone network Multimodal fusion feature map of layer output.

[0130] Example 3

[0131] Another aspect of the present invention provides an electronic device, see Figure 7 , shown is an electronic device in embodiment 3 of the present invention, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, the multimodal crowd counting method as described above is implemented.

[0132] In some embodiments, the processor 10 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, used to run program codes or process data stored in the memory 20, such as executing access restriction programs.

[0133] The memory 20 includes at least one type of readable storage medium, including flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 20 may be an internal storage unit of the electronic device, such as the hard disk of the electronic device. In other embodiments, the memory 20 may also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory 20 may include both an internal storage unit of the electronic device and an external storage device. The memory 20 can be used not only to store application software and various types of data of the electronic device, but also to temporarily store data that has been output or is about to be output.

[0134] It should be pointed out that Figure 7 The structure shown does not constitute a limitation to the electronic device. In other embodiments, the electronic device may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0135] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the multimodal crowd counting method as described above is implemented.

[0136] Those skilled in the art will appreciate that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device), or in conjunction with such instruction execution system, apparatus, or device. For purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device.

[0137] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting, or processing it in another suitable manner as necessary, and then storing it in a computer memory.

[0138] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following technologies known in the art may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0139] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0140] The above embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A multimodal crowd counting method, characterized in that: The method comprises: Acquiring an image and extracting image features of the image, wherein the image includes a visible light modality image and a thermal infrared modality image; Generate a target attention map by applying the image features of the visible light modality image and the thermal infrared modality image layer by layer through the spatial frequency attention in the spatial frequency guidance module; fusing the image features of the visible light modality image and the thermal infrared modality image at each level and the target attention map through an adaptive dynamic fusion module to obtain multimodal fusion features; Each multimodal fusion feature is passed through a multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features, and generate a predicted density map, wherein the multi-scale progressive fusion module is composed of a void space pyramid pooling and a SwinTransformer Block; The adaptive dynamic fusion module consists of three parts: foreground enhancement, cross attention and dynamic weighting. The implementation process of foreground enhancement is expressed as follows: in, represents element-wise addition operation, represents the element-wise multiplication operation, Represents the visible light mode After the enhancement of the level, Indicates thermal infrared mode Enhanced features of the hierarchy; The implementation process of the cross attention is expressed as: Among them, K r 、V r , Q r are three vectors, represents 1×1 convolution, and Respectively represent the transposed 、 vector, Indicates the number of feature channels after reduction in order to reduce computational overhead, represents the Softmax activation function, and Respectively represent the visible light modal features and thermal infrared modal features after interactive learning; The implementation process of the dynamic weighting is expressed as follows: in, It represents the aggregated features after channel splicing of two different modal features. and They represent the feature fusion weights of the visible light modality and thermal infrared modality generated by 1×1 convolution and Softmax activation function respectively.

2. The multimodal crowd counting method according to claim 1, characterized in that: The spatial-frequency attention consists of spatial-domain attention and frequency-domain attention. The processing results of the spatial-domain attention and the frequency-domain attention are fused to obtain the final target attention map.

3. The multimodal crowd counting method according to claim 2, characterized in that: In the step of fusing the processing results of spatial domain attention and frequency domain attention to obtain the final target attention map, the implementation process of spatial domain attention is expressed as: in, represents the input features, represents 1×1 convolution, represents the Sigmoid activation function, represents the generated spatial attention weight, represents element-wise multiplication, Represents the features enhanced by spatial domain attention; The implementation process of frequency domain attention is expressed as: in, represents the two-dimensional discrete cosine transform function, represents the fully connected layer, ReLU represents the rectified linear unit, represents the generated channel attention weight, Represents the features after frequency domain attention enhancement; The fusion implementation process is expressed as: in, represents element-wise addition operation, Indicates that after 1×1 convolution, batch normalization and ReLU corrected linear unit operation, Indicates that after 1×1 convolution, batch normalization and Sigmoid activation operations, represents the target attention map.

4. The multimodal crowd counting method according to claim 3, characterized in that: The implementation process of the spatial frequency guidance module is expressed as follows: in, represents spatial frequency attention, represents the generated visible light modality attention map, represents the generated thermal infrared modality attention map, Indicates the maximum value operation of each element. The final generated guide Attention map for fusing hierarchical multimodal features.

5. The multimodal crowd counting method according to claim 4, characterized in that: In the step of fusing the image features of the visible light modality image and the thermal infrared modality image at each level and the target attention map through an adaptive dynamic fusion module to obtain each multimodal fusion feature, the fusion process is expressed as follows: in, Indicates the backbone network Multimodal fusion features of hierarchical output.

6. The multimodal crowd counting method according to claim 5, characterized in that: The multi-modal fusion features are passed through a multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features. In the step of generating a prediction density map, the generation process is expressed as follows: in, represents the atrous spatial pyramid pooling, Indicates 2x upsampling, Represents the concatenation of the dimensions of two tensors, represents Swin Transformer Block, Indicates the backbone network Multimodal fusion feature map of layer output.

7. A multimodal crowd counting system, characterized in that: For implementing the multimodal crowd counting method according to any one of claims 1 to 6, the system comprises: an acquisition module, configured to acquire images and extract image features of the images, wherein the images include visible light modality images and thermal infrared modality images; a target attention map generation module, configured to generate a target attention map by applying the image features of the visible light modality image and the thermal infrared modality image layer by layer through the spatial frequency attention in the spatial frequency guidance module; a fusion module, configured to fuse the image features of the visible light modality image and the thermal infrared modality image at each level and the target attention map through an adaptive dynamic fusion module to obtain multimodal fusion features; The prediction density map generation module is used to pass each multimodal fusion feature through the multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features and generate a prediction density map, wherein the multi-scale progressive fusion module is composed of void spatial pyramid pooling and Swin Transformer Block.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multimodal crowd counting method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: The invention comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multimodal crowd counting method according to any one of claims 1 to 6 is implemented.

Images