Multi-mode crowd counting method and system

Abstract

The invention provides a multi-modal crowd counting method and system, and the method comprises the steps: obtaining an image, and extracting the image features of the image, the image comprising a visible light modal image and a thermal infrared modal image; image features of the visible light modal image and the thermal infrared modal image are subjected to attention through spatial frequency in a spatial frequency guiding module layer by layer to generate a target attention map; fusing the image features of the visible light modal image and the thermal infrared modal image of each level and the target attention map through an adaptive dynamic fusion module to obtain each multi-modal fusion feature; according to the multi-modal crowd counting method, a prediction density map is generated from each multi-modal fusion feature through a multi-scale progressive fusion module, the multi-scale progressive fusion module is composed of cavity space pyramid pooling and Swin Transform Block, and through the modules designed in the steps, the problems that multi-modal crowd counting errors are large and counting precision is low can be effectively solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of crowd counting, and particularly relates to a multi-modal crowd counting method and system. Background Art

[0002] In the multi-modal (visible light-thermal infrared modality) crowd counting task, the primary challenge to be addressed is the multi-modal fusion problem, because visible light modality images and thermal infrared modality images provide significantly different information under different lighting conditions.

[0003] By using the Sobel operator to extract the gradient information of the two modality images, it can be found that: in high-illumination scenes, the visible light modality can easily distinguish the crowd area from the background noise, while the texture detail information of the thermal infrared modality is relatively significantly less; in low-illumination scenes, the visible light modality cannot well identify the crowd area, but the thermal infrared modality can easily determine the crowd position based on the human body temperature. Therefore, the multi-modal fusion problem refers to how to utilize the different imaging characteristics of visible light images and thermal infrared images, fully exploit the complementary information between multi-modal data, so as to generate a "new image" that can comprehensively and accurately describe the actual scene, thereby improving the model counting accuracy.

[0004] When existing crowd counting methods solve this problem, most of them directly fuse features from different modalities without any prior knowledge. Although this strategy has achieved certain results, it ignores the intricate interaction between the two modalities. On the one hand, it easily lacks the fusion target when facing complex background interference and cannot well fuse the crowd area; on the other hand, the fusion result is sometimes good and sometimes bad under different lighting conditions, and does not fully learn the dynamic change of the lighting intensity in the real scene. Summary of the Invention

[0005] Based on this, embodiments of the present invention provide a multi-modal crowd counting method and system, aiming to solve the problems of large multi-modal crowd counting error and low counting accuracy in the prior art.

[0006] The first aspect of the embodiments of the present invention provides a multi-modal crowd counting method, and the method includes: Obtain an image and extract the image features of the image, where the image includes a visible light modality image and a thermal infrared modality image; Generate a target attention map layer by layer for the image features of the visible light modality image and the thermal infrared modality image through spatial frequency attention in the spatial frequency guidance module; Fuse the image features of the visible light modality image and the thermal infrared modality image at each layer and the target attention map through an adaptive dynamic fusion module to obtain each multi-modal fusion feature; The multi-modal fusion features are passed through a multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features, generating a predicted density map. Among them, the multi-scale progressive fusion module is composed of a dilated spatial pyramid pooling and a Swin Transformer Block.

[0007] Furthermore, the spatial-frequency attention is composed of spatial-domain attention and frequency-domain attention. The processing results of the spatial-domain attention and the frequency-domain attention are fused to obtain the final target attention map.

[0008] Furthermore, in the step of fusing the processing results of the spatial-domain attention and the frequency-domain attention to obtain the final target attention map, the implementation process of the spatial-domain attention is expressed as:

[0009] Among them, represents the input feature, represents a 1×1 convolution, represents the Sigmoid activation function, represents the generated spatial attention weight, represents element-wise multiplication, represents the feature enhanced by the spatial-domain attention; The implementation process of the frequency-domain attention is expressed as:

[0010] Among them, represents the two-dimensional discrete cosine transform function, represents the fully connected layer, represents the ReLU rectified linear unit, represents the generated channel attention weight, represents the feature enhanced by the frequency-domain attention; The implementation process of the fusion is expressed as:

[0011] Among them, represents the element-wise addition operation, represents the operation of passing through a 1×1 convolution, batch normalization, and ReLU rectified linear unit, represents the operation of passing through a 1×1 convolution, batch normalization, and Sigmoid activation, represents the target attention map.

[0012] Furthermore, the implementation process of the spatial-frequency guidance module is expressed as:

[0013] Among them, represents spatial frequency attention, represents the generated visible light modality attention map, represents the generated thermal infrared modality attention map, represents the element-wise maximum operation, represents the finally generated attention map for guiding the fusion of multi-modal features at the level.

[0014] Furthermore, the adaptive dynamic fusion module consists of three parts: foreground enhancement, cross-attention, and dynamic weighting. The implementation process of the foreground enhancement is expressed as:

[0015] Among them, represents the element-wise addition operation, represents the element-wise multiplication operation, represents the enhanced feature of the visible light modality at the level, represents the enhanced feature of the thermal infrared modality at the level; The implementation process of the cross-attention is expressed as:

[0016]

[0017] Among them, K r , V r , Q r are three vectors, represents a 1×1 convolution, and respectively represent the , vectors after transposition, represents the reduced number of feature channels for reducing the computational overhead, represents the Softmax activation function, and respectively represent the visible light modality features and thermal infrared modality features after interactive learning; The implementation process of the dynamic weighting is expressed as:

[0018] Among them, represents the aggregated feature after concatenating the channels of the two different modality features, and respectively represent the feature fusion weights of the visible light modality and the thermal infrared modality generated by the 1×1 convolution and the Softmax activation function.

[0019] Further, in the step of fusing the image features of the visible light modality image and the thermal infrared modality image at each level and the target attention map through the adaptive dynamic fusion module to obtain each multi-modal fusion feature, the fusion process is expressed as:

[0020] where represents the multi-modal fusion feature output by the backbone network at the level.

[0021] Further, in the step of passing each multi-modal fusion feature through the multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features and generate a predicted density map, the generation process is expressed as:

[0022] where represents atrous spatial pyramid pooling, represents upsampling by a factor of 2, represents the dimension concatenation of two tensors, represents the Swin Transformer Block, represents the multi-modal fusion feature map output by the backbone network at the level.

[0023] The second aspect of the embodiments of the present invention provides a multi-modal crowd counting system for implementing the multi-modal crowd counting method described in the first aspect. The system includes: An acquisition module, configured to acquire an image and extract the image features of the image, where the image includes a visible light modality image and a thermal infrared modality image; A target attention map generation module, configured to generate a target attention map by passing the image features of the visible light modality image and the thermal infrared modality image through the spatial frequency attention in the spatial frequency guidance module level by level; A fusion module, configured to fuse the image features of the visible light modality image and the thermal infrared modality image at each level and the target attention map through the adaptive dynamic fusion module to obtain each multi-modal fusion feature; A predicted density map generation module, configured to pass each multi-modal fusion feature through the multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features and generate a predicted density map, where the multi-scale progressive fusion module is composed of atrous spatial pyramid pooling and the Swin Transformer Block.

[0024] In a third aspect of the embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the multi-modal crowd counting method provided in the first aspect is implemented.

[0025] In a fourth aspect of the embodiments of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the multi-modal crowd counting method provided in the first aspect is implemented.

[0026] A multi-modal crowd counting method and system provided in the embodiments of the present invention obtain an image and extract image features of the image, where the image includes a visible light modality image and a thermal infrared modality image; generate a target attention map for the image features of the visible light modality image and the thermal infrared modality image layer by layer through spatial frequency attention in a spatial frequency guidance module; fuse the image features of the visible light modality image and the thermal infrared modality image at each layer and the target attention map through an adaptive dynamic fusion module to obtain multi-modal fusion features at each layer; pass the multi-modal fusion features at each layer through a multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features and generate a predicted density map. The multi-scale progressive fusion module is composed of an atrous spatial pyramid pooling and a Swin Transformer Block. Through the modules designed in the above steps, the problems of large multi-modal crowd counting error and low counting accuracy can be effectively solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flowchart of the implementation of a multi-modal crowd counting method provided in Embodiment 1 of the present invention; Figure 2 It is a structural schematic diagram of the multi-modal crowd counting method; Figure 3 It is a structural diagram of spatial frequency attention; Figure 4 It is a structural schematic diagram of the spatial frequency guidance module; Figure 5 It is a structural schematic diagram of the adaptive dynamic fusion module; Figure 6 It is a structural block diagram of a multi-modal crowd counting system provided in Embodiment 2 of the present invention; Figure 7 It is a structural block diagram of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] To facilitate the understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant accompanying drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the disclosure of the present invention more thorough and comprehensive.

[0029] It should be noted that when an element is referred to as being "fixedly provided on" another element, it can be directly on the other element or there may also be an intermediate element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be an intermediate element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are only for the purpose of illustration.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used herein in the description of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0031] Embodiment 1 According to an embodiment of the present invention, an embodiment of a multi-modal crowd counting method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0032] In the first embodiment, a multi-modal crowd counting method is provided, which can be used in an electronic device, such as a computer. Please refer to Figure 1 and Figure 2 , Figure 1 shows the implementation flowchart of a multi-modal crowd counting method provided by the first embodiment of the present invention. Figure 2 is a structural schematic diagram of the multi-modal crowd counting method, specifically including steps S01 to S04.

[0033] Step S01, obtain an image and extract the image features of the image, where the image includes a visible light modality image and a thermal infrared modality image.

[0034] Among them, the image features can be extracted by a method based on deep learning. Specifically, through structures such as convolutional layers, pooling layers, and fully connected layers, the feature representation of the image is automatically learned.

[0035] Step S02: Generate a target attention map layer by layer for the image features of the visible light modality image and the thermal infrared modality image through the spatial frequency attention in the spatial frequency guidance module.

[0036] Specifically, since different modalities have different target perception capabilities. For example, visible light images are more likely to provide low-frequency information in image processing, while thermal infrared images are better at capturing high-frequency information related to temperature distribution. Therefore, a Spatial Frequency Guidance Module (SFGM) is proposed to generate an attention map based on the characteristics of different modalities using the attention mechanism to further guide the fusion of multi-modal information. According to the different ways of applying attention weights, visual attention can be divided into two categories: spatial attention and channel attention. Spatial attention focuses on learning the spatial location information of the target, while channel attention focuses on the effective features related to the task by learning the importance of different channels in the feature map. To improve the generation quality of attention maps for different modalities, the respective advantages of these two types of attention are combined, and Spatial-Frequency Attention (SFA) is proposed in SFGM, which includes Spatial Domain Attention (SDA) acting on the spatial dimension of the feature and Frequency Domain Attention (FDA) acting on the channel dimension of the feature. Its structure is as Figure 3 shown.

[0037] SDA can enable the model to pay more attention to the crowd area in the image rather than other non-task targets to suppress the interference of complex backgrounds. To further reduce the computational overhead of the algorithm, SDA does not choose the classical visual spatial attention to implement, but is based on 1×1 convolution and the Sigmoid function. Its process can be expressed as follows:

[0038] Among them, represents the input feature, represents 1×1 convolution, represents the Sigmoid activation function, represents the generated spatial attention weight, represents element-wise multiplication, represents the feature enhanced by spatial domain attention.

[0039] In traditional visual channel attention, global average pooling (GAP) is generally used to generate global feature representations for each channel, providing basic information for channel attention calculation. However, since the number of channels in the feature map is usually large, based on GAP, the means of different channels may be the same, but in fact the semantic content information they represent may be different, which will lead to the inability to capture diverse feature information. Channel attention based on GAP retains the lowest frequency information during the feature learning process, but discards other equally important high-frequency information. Therefore, to overcome the above deficiencies, FDA is introduced. Using the discrete cosine transform (DCT), different frequency components are extracted from different channels from the frequency domain perspective to obtain richer population target feature information, and the process can be expressed as follows:

[0040] Among them, represents the two-dimensional discrete cosine transform function, represents the fully connected layer, represents the ReLU rectified linear unit, represents the generated channel attention weight, represents the feature enhanced by frequency domain attention.

[0041] To fully combine the respective advantages of spatial attention and channel attention, the features enhanced by SDA and FDA are fused to generate a hybrid attention map to further highlight the modal characteristics, and the process can be expressed as follows:

[0042] Among them, represents the element-wise addition operation, represents the operation of 1×1 convolution, batch normalization, and ReLU rectified linear unit, represents the operation of 1×1 convolution, batch normalization, and Sigmoid activation, represents the target attention map.

[0043] It should be noted that please refer to Figure 4 and Figure 5 , Figure 4 is the schematic diagram of the spatial frequency guidance module structure, Figure 5 is the schematic diagram of the adaptive dynamic fusion module structure. To learn different modal characteristics, SFGM targets the hierarchical features of visible light images and red and thermal infrared images and ( )Use SFA to generate the visible light modality attention map and the thermal infrared modality attention map respectively, and adopt the element-wise maximum operation to extract the most significant information of the two modalities of attention, so as to suppress the interference of background noise and provide the position information of the crowd area for the subsequent multi-modal fusion process, thereby improving the attention to crowd characteristics. The specific process of SFGM can be expressed as follows:

[0044] Among them, represents the spatial frequency attention, represents the generated visible light modality attention map, represents the generated thermal infrared modality attention map, represents the element-wise maximum operation, represents the finally generated attention map for guiding the fusion of multi-modal features at the th level.

[0045] Step S03, fuse the image features of the visible light modality image and the thermal infrared modality image at each level and the target attention map through the adaptive dynamic fusion module to obtain each multi-modal fusion feature.

[0046] Specifically, according to different illumination intensities, the crowd counting scenario can be roughly divided into two categories: low illumination scenario and high illumination scenario. Different modalities play different roles in these two scenarios due to their own characteristics. Therefore, an Adaptive Dynamic Fusion Module (ADFM) is proposed, which can perform multi-modal dynamic fusion according to the specific scenario to further improve the fusion quality. To vividly illustrate the reason for dynamic fusion, the gradient information of the visible light modality image and the thermal infrared modality image is extracted by the Sobel operator and visualized in these two scenarios respectively. It can be found that in the high illumination scenario, the visible light modality has richer texture detail information than the thermal infrared modality; however, in the low illumination scenario, the imaging quality of the visible light modality is low and the information provided is limited, but the thermal infrared modality provides relatively richer high-frequency gradient information because it is not affected by the illumination factor. Therefore, the main role of ADFM is to dynamically learn the complementary information of the gradients of the two modalities, which consists of three parts: foreground enhancement, cross attention, and dynamic weighting.

[0047] In foreground enhancement, mainly through the guiding fusion attention map generated by SFGM, respectively enhance the hierarchical features and ( ) of the visible light modality and the thermal infrared modality, in order to be able to perform fusion on the premise of knowing the position information of the crowd area. The process can be expressed as follows:

[0048] Among them, represents an element-wise addition operation, represents an element-wise multiplication operation, represents the enhanced feature of the th layer in the visible light modality, represents the enhanced feature of the th layer in the thermal infrared modality.

[0049] To capture the global feature correlations between multi-modal data, cross-attention is introduced to allow information to flow between different modalities. First, the enhanced visible light modality features and the thermal infrared modality features are respectively mapped into three vectors in the attention: key (K), value (V), and query (Q) through three different 1×1 convolutions. The process can be simply expressed as follows:

[0050] In addition, different from the self-attention mechanism, in the calculation process of cross-attention, the K vector and the V vector come from the same modality, while the Q vector comes from another modality. The calculation process is as follows:

[0051] Among them, Kr, Vr, and Qr are three vectors, represents a 1×1 convolution, and respectively represent the transposed , vectors, represents the reduced number of feature channels to reduce the computational overhead, represents the Softmax activation function, and respectively represent the visible light modality features and the thermal infrared modality features after interactive learning.

[0052] Because different modalities provide different gradient information in different scenarios, different fusion weights need to be generated according to the specific scenario during the multi-modal fusion process to fully learn the complementary characteristics between modalities. For example, in high-illumination scenarios, the visible light modality should be the main one and the thermal infrared modality should be the auxiliary one, but in low-illumination scenarios, it is the opposite. Therefore, based on the visible light modality features and the thermal infrared modality features Dynamically assign weights to the multi-modal fusion process, and the process can be expressed as follows:

[0053] Among them, represents the aggregated feature after concatenating the channels of two different modal features, and respectively represent the feature fusion weights of the visible light modality and the thermal infrared modality generated by passing through a 1×1 convolution and a Softmax activation function.

[0054] More specifically, in order to improve the multi-modal fusion quality and avoid the gradual attenuation of modal information, a design similar to the Residual Network (ResNet) is adopted. The fusion process is expressed as:

[0055] Among them, represents the multi-modal fusion feature output by the layer of the backbone network.

[0056] Step S04: Pass each multi-modal fusion feature through a multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features and generate a predicted density map. Among them, the multi-scale progressive fusion module is composed of an atrous spatial pyramid pooling and a Swin Transformer Block.

[0057] Specifically, in the real scene, the target object may appear in different sizes under different environments, perspectives, etc. How to effectively identify and process target objects presented in different sizes has always been a common problem in the field of computer vision - the target multi-scale problem, and this is no exception in the multi-modal crowd counting task. Therefore, ConvNeXt with hierarchical feature representation is selected as the backbone network. For an image with an input resolution of H×W, the sizes of the feature maps output by its 4 stages are respectively , , and . Although the hierarchical structure of ConvNeXt helps to capture crowd information at multiple scales, in the real scene, the scale change of the crowd target is often continuous, and the hierarchical backbone network alone cannot effectively handle this scale change problem. In view of this, a multi-scale progressive fusion module (Multi-scale Progressive Fusion Module, MPFM) is introduced, aiming to better learn crowd target information at different scales.

[0058] MPFM is mainly composed of Atrous Spatial Pyramid Pooling (ASPP) and Swin Transformer Block. First, dilated convolutions with different dilation rates in ASPP are used to learn the fused features of the outputs of different levels of the backbone network , so as to further learn the target information at more diverse scales; secondly, the sliding window attention in the Swin Transformer Block enables the network to have a global receptive field, effectively helping the model to understand and fuse the features from different image regions; finally, since there are obvious differences in the features extracted from different levels of the backbone network, for example, the shallow network mainly captures the detailed information of the image (edges, textures, etc.), while the deep network contains the semantic information for the global understanding of the image. The crowd counting task needs to utilize both semantic information to identify the target objects in the image and detailed information to distinguish the boundaries of different target objects.

[0059] Therefore, MPFM designs a progressive fusion process from deep features to shallow features, aiming to fully learn semantic information and detailed information to effectively improve the quality of the multi-modal fusion output feature map. The specific process of MIAM can be expressed by the following formula:

[0060] Among them, represents Atrous Spatial Pyramid Pooling, represents upsampling by a factor of 2, represents the dimensional concatenation of two tensors, represents Swin Transformer Block, represents the multi-modal fusion feature map output at the th level of the backbone network, that is, I 1 is the predicted density map.

[0061] In summary, the multi-modal crowd counting method in the above embodiments of the present invention obtains an image and extracts the image features of the image. The image includes a visible light modality image and a thermal infrared modality image. The image features of the visible light modality image and the thermal infrared modality image are successively passed through the spatial frequency attention in the spatial frequency guidance module to generate a target attention map. The image features of the visible light modality image and the thermal infrared modality image at each level and the target attention map are fused through an adaptive dynamic fusion module to obtain multi-modal fusion features. The multi-modal fusion features are passed through a multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features and generate a predicted density map. The multi-scale progressive fusion module is composed of an atrous spatial pyramid pooling and a Swin Transformer Block. Through the modules designed in the above steps, the problem of large error and low counting accuracy in multi-modal crowd counting can be effectively solved.

[0062] Embodiment 2 Please refer to Figure 6 , Figure 6 which is a structural block diagram of a multi-modal crowd counting system provided by Embodiment 2 of the present invention. The multi-modal crowd counting system 200 is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, the term "module" may be a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0063] Specifically, the multi-modal crowd counting system 200 includes: an acquisition module 21, a target attention map generation module 22, a fusion module 23, and a predicted density map generation module 24, where: The acquisition module 21 is configured to acquire an image and extract the image features of the image. The image includes a visible light modality image and a thermal infrared modality image; The target attention map generation module 22 is configured to generate a target attention map by successively passing the image features of the visible light modality image and the thermal infrared modality image through the spatial frequency attention in the spatial frequency guidance module. The spatial frequency attention is composed of a spatial domain attention and a frequency domain attention. The processing results of the spatial domain attention and the frequency domain attention are fused to obtain a final target attention map. The implementation process of the spatial domain attention is expressed as:

[0064] where, represents the input feature, represents a 1×1 convolution, represents a Sigmoid activation function, represents the generated spatial attention weights, represents element-wise multiplication, represents the feature enhanced by spatial domain attention; The implementation process of frequency domain attention is expressed as:

[0065] where, represents the two-dimensional discrete cosine transform function, represents the fully connected layer, represents the ReLU rectified linear unit, represents the generated channel attention weights, represents the feature enhanced by frequency domain attention; The implementation process of fusion is expressed as:

[0066] where, represents the element-wise addition operation, represents the operation of 1×1 convolution, batch normalization and ReLU rectified linear unit, represents the operation of 1×1 convolution, batch normalization and Sigmoid activation, represents the target attention map. In addition, the implementation process of the spatial frequency guidance module is expressed as:

[0067] where, represents spatial frequency attention, represents the generated visible light modality attention map, represents the generated thermal infrared modality attention map, represents the element-wise maximum operation, represents the finally generated attention map for guiding the fusion of multi-modal features at the layer; The fusion module 23 is used to fuse the image features of the visible light modality image and the thermal infrared modality image at each layer and the target attention map through the adaptive dynamic fusion module to obtain each multi-modal fusion feature. The adaptive dynamic fusion module consists of three parts: foreground enhancement, cross attention and dynamic weighting. The implementation process of the foreground enhancement is expressed as:

[0068] where, represents the element-wise addition operation, represents the element-wise multiplication operation, represents the enhanced feature of the visible light modality at the layer, Represents the enhanced features at the layer of the thermal infrared modality; The implementation process of the cross-attention is expressed as:

[0069]

[0070] Among them, K r , V r , Q r are three vectors, represents a 1×1 convolution, and respectively represent the , vectors after transposition, represents the reduced number of feature channels for reducing the computational overhead, represents the Softmax activation function, and respectively represent the visible light modality features and thermal infrared modality features after interactive learning; The implementation process of the dynamic weighting is expressed as:

[0071] Among them, represents the aggregated features after channel concatenation of two different modality features, and respectively represent the feature fusion weights of the visible light modality and thermal infrared modality generated by 1×1 convolution and the Softmax activation function. Additionally, the fusion process is expressed as:

[0072] Among them, represents the multi-modal fusion features output at the layer of the backbone network; The prediction density map generation module 24 is used to complete the progressive fusion process from deep features to shallow features for each multi-modal fusion feature through the multi-scale progressive fusion module, and generate a prediction density map. Among them, the multi-scale progressive fusion module is composed of an atrous spatial pyramid pooling and a Swin Transformer Block, and the generation process is expressed as:

[0073] Among them, represents the atrous spatial pyramid pooling, represents 2-fold upsampling, represents the dimension concatenation of two tensors, Denotes a Swin Transformer Block, Denotes the multimodal fusion feature map output at the layer of the backbone network.

[0074] Example 3 On the other hand, the present invention also proposes an electronic device. Please refer to Figure 7 . Shown is the electronic device in Example 3 of the present invention, including a memory 20, a processor 10, and a computer program 30 stored on the memory and executable on the processor. When the processor 10 executes the computer program 30, the above-mentioned multimodal crowd counting method is implemented.

[0075] Among them, in some embodiments, the processor 10 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips, and is used to run the program code stored in the memory 20 or process data, such as executing an access restriction program, etc.

[0076] Among them, the memory 20 includes at least one type of readable storage medium. The readable storage medium includes flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disc, etc. The memory 20 may be an internal storage unit of the electronic device in some embodiments, such as the hard disk of the electronic device. The memory 20 may also be an external storage device of the electronic device in other embodiments, such as a plug-in hard disk equipped on the electronic device, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 20 may also include both an internal storage unit and an external storage device of the electronic device. The memory 20 can be used not only to store application software and various types of data of the electronic device, but also to temporarily store data that has been output or will be output.

[0077] It should be noted that Figure 7 the structure shown does not constitute a limitation on the electronic device. In other embodiments, the electronic device may include fewer or more components than shown, or combine certain components, or have a different component arrangement.

[0078] The embodiment of the present invention also proposes a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned multimodal crowd counting method is implemented.

[0079] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0080] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection part with one or more wirings (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other suitable processing as necessary, and then stored in a computer memory.

[0081] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with suitable combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0082] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0083] The above embodiments merely illustrate several implementation manners of the present invention. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the scope of the patent for the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention shall be subject to the appended claims.

Claims

1. A multimodal crowd counting method, characterized in that: The method comprises: Acquire an image and extract image features of the image, wherein the image includes a visible light modality image and a thermal infrared modality image; The image features of the visible light modality image and the thermal infrared modality image are layered by layer through the spatial frequency attention in the spatial frequency guidance module to generate a target attention map; The image features of the visible light modality image and the thermal infrared modality image at each level and the target attention map are fused through an adaptive dynamic fusion module to obtain various multimodal fusion features; Each multimodal fusion feature is passed through a multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features, and generate a predicted density map, wherein the multi-scale progressive fusion module is composed of a void space pyramid pooling and a SwinTransformer Block.

2. The multimodal crowd counting method according to claim 1, characterized in that: The spatial frequency attention is composed of spatial domain attention and frequency domain attention. The processing results of the spatial domain attention and the processing results of the frequency domain attention are fused to obtain the final target attention map.

3. The multimodal crowd counting method according to claim 2, characterized in that: In the step of fusing the processing results of the spatial domain attention and the processing results of the frequency domain attention to obtain the final target attention map, the implementation process of the spatial domain attention is expressed as: in, represents the input features, represents 1×1 convolution, represents the Sigmoid activation function, represents the generated spatial attention weight, represents element-wise multiplication, Represents the features enhanced by spatial domain attention; The implementation process of frequency domain attention is expressed as: in, represents the two-dimensional discrete cosine transform function, represents the fully connected layer, ReLU represents the rectified linear unit, represents the generated channel attention weight, Represents the features after frequency domain attention enhancement; The fusion implementation process is expressed as: in, represents the element-by-element addition operation, It means that after 1×1 convolution, batch normalization and ReLU corrected linear unit operation, It means that after 1×1 convolution, batch normalization and Sigmoid activation operations, represents the target attention map.

4. The multimodal crowd counting method according to claim 3, characterized in that: The implementation process of the spatial frequency guidance module is expressed as follows: in, represents spatial frequency attention, represents the generated visible light modality attention map, represents the generated thermal infrared modality attention map, It means to take the maximum value of each element. The final generated guide Attention map for fusing hierarchical multimodal features.

5. The multimodal crowd counting method according to claim 4, characterized in that: The adaptive dynamic fusion module consists of three parts: foreground enhancement, cross attention and dynamic weighting. The implementation process of foreground enhancement is expressed as follows: in, represents the element-by-element addition operation, represents an element-wise multiplication operation, Represents the visible light mode After the enhancement of the level, Indicates the thermal infrared mode Enhanced features of the hierarchy; The implementation process of the cross attention is expressed as: Among them, K r 、V r , Q r are three vectors, represents 1×1 convolution, and Respectively represent the transposed , vector, Indicates the number of feature channels after reduction in order to reduce computational overhead, represents the Softmax activation function, and They represent the visible light modal features and thermal infrared modal features after interactive learning respectively; The implementation process of the dynamic weighting is expressed as follows: in, It represents the aggregated features after channel concatenation of two different modal features. and They respectively represent the feature fusion weights of the visible light modality and thermal infrared modality generated by 1×1 convolution and Softmax activation function.

6. The multimodal crowd counting method according to claim 5, characterized in that: In the step of fusing the image features of the visible light modality image and the thermal infrared modality image at each level and the target attention map through an adaptive dynamic fusion module to obtain each multimodal fusion feature, the fusion process is expressed as: in, Represents the backbone network Multimodal fusion features of hierarchical output.

7. The multimodal crowd counting method according to claim 6, characterized in that: In the step of passing each multimodal fusion feature through a multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features, and generating a prediction density map, the generation process is expressed as: in, represents the atrous spatial pyramid pooling, Indicates 2x upsampling, Represents the concatenation of the dimensions of two tensors, Represents Swin Transformer Block, Represents the backbone network Multimodal fusion feature map of layer output.

8. A multimodal crowd counting system, characterized in that: For implementing the multimodal crowd counting method according to any one of claims 1 to 7, the system comprises: An acquisition module, used to acquire images and extract image features of the images, wherein the images include visible light modality images and thermal infrared modality images; A target attention map generation module, used for generating a target attention map by layer-by-layer using the spatial frequency attention in the spatial frequency guidance module for the image features of the visible light modality image and the thermal infrared modality image; A fusion module, used for fusing the image features of the visible light modality image and the thermal infrared modality image at each level and the target attention map through an adaptive dynamic fusion module to obtain various multimodal fusion features; The prediction density map generation module is used to pass each multimodal fusion feature through a multi-scale progressive fusion module to complete the progressive fusion process from deep features to shallow features and generate a prediction density map, wherein the multi-scale progressive fusion module is composed of a void space pyramid pooling and a Swin Transformer Block.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multimodal crowd counting method as described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multimodal crowd counting method according to any one of claims 1 to 7 is implemented.

Images