Crowd counting system and method based on cross-modal feature extraction and adaptive fusion

Abstract

The invention discloses a cross-modal feature extraction and adaptive fusion crowd counting system and method, and belongs to the technical field of computer vision. The crowd counting system comprises a visible light image acquisition module and a thermal infrared image acquisition module which are parallel; a visible light double-layer feature extraction network layer and a visible light feature preference selection interaction layer are sequentially connected behind the visible light image acquisition module, and a thermal infrared double-layer feature extraction network layer and a thermal infrared feature preference selection interaction layer are sequentially connected behind the thermal infrared image acquisition module. A common lightweight cross-modal feature interaction fusion attention module and a crowd density map estimation module are connected behind the visible light feature preference selection interaction layer and the thermal infrared feature preference selection interaction layer, and crowd counting can be realized by using a crowd density map obtained by the system. The method can effectively solve the problem that the counting performance is limited due to the fact that different hierarchical features cannot be adaptively fused according to the current scene.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a crowd counting system and method for cross-modal feature extraction and adaptive fusion. Background Art

[0002] Crowd counting is an important research direction in the field of computer vision, and is of great significance to public safety, business analysis, smart security, crowd flow control, urban planning, traffic management, etc. Accurate and efficient crowd counting results can reflect the density of people in a certain area, provide diverse data support for urban planning and business decision-making, and provide useful information for other tasks, such as crowd behavior analysis and crowd positioning, and can be applied to applications such as traffic monitoring and public health.

[0003] At present, crowd counting algorithms based on visible light images have achieved satisfactory performance. However, there are unsolvable problems in designing network models based only on RGB images, because visible light images depend on the intensity of light. When the ambient lighting conditions are poor or the target object is blocked, the visible light images captured contain very little crowd information, and the crowd counting method using a single modality is often difficult to achieve the desired effect. In order to overcome the problem of missing crowds in dark scenes, thermal infrared images are integrated with visible light images to form a cross-modal crowd counting dataset to improve the performance and robustness of counting in complex scenes. However, there will be interference from the thermal cross phenomenon; on the contrary, visible light images have poor imaging effects under harsh lighting conditions but have higher resolution, which is conducive to alleviating the impact of the thermal cross phenomenon. Therefore, combining visible light images and thermal infrared images for cross-modal crowd counting can make full use of the complementary advantages of the two modalities and improve the accuracy of counting results and the robustness of the counting model. However, the effective crowd information that visible light images and thermal infrared images can provide in different scenarios is not the same. If fixed-weight addition or splicing operations are used, it may lead to information loss or redundancy, and it is impossible to preferentially fuse different levels of features according to the characteristics of specific tasks. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a crowd counting system and method for cross-modal feature extraction and adaptive fusion, which can effectively utilize features of different modalities and features of different levels in the same modality, thereby improving the crowd counting performance of the cross-modal crowd counting method in different scenarios.

[0005] To solve the above technical problems, the technical solution adopted by the present invention is: a crowd counting system for cross-modal feature extraction and adaptive fusion, comprising a parallel visible light image acquisition module and a thermal infrared image acquisition module, the visible light image acquisition module is sequentially connected with a visible light double-level feature extraction network layer and a visible light feature preference selection interaction layer, the thermal infrared image acquisition module is sequentially connected with a thermal infrared double-level feature extraction network layer and a thermal infrared feature preference selection interaction layer, and the visible light feature preference selection interaction layer and the thermal infrared feature preference selection interaction layer are connected with a common lightweight cross-modal feature interaction fusion attention module and a crowd density map estimation module.

[0006] The further improvement of the technical solution of the present invention is that: the visible light double-level feature extraction network layer and the thermal infrared double-level feature extraction network layer have the same structure, both of which include parallel local feature extraction layers and global feature extraction layers, wherein the local feature extraction layers are, from input to output, the first convolution processing block: 2 convolution layers, 1 maximum pooling layer, the second convolution processing block: 2 convolution layers, 1 maximum pooling layer, the third convolution processing block: 4 convolution layers, 1 maximum pooling layer, the fourth convolution processing block: 4 convolution layers, 1 maximum pooling layer and the fifth convolution processing block: 4 convolution layers; the global feature extraction layer includes 3 Transformer encoding blocks, namely the first, second and third encoding blocks, wherein each encoding block includes 1 shearing operation, 1 linear transformation operation, 1 normalization operation, 1 attention layer and 1 feedforward layer;

[0007] The visible light feature preference selection interaction layer and the thermal infrared feature preference selection interaction layer have the same structure, including one addition operation and two Sigmoid activation functions from input to output;

[0008] The lightweight cross-modal feature interaction fusion attention module includes two parallel intra-modal feature processing modules. The two intra-modal feature processing modules are respectively connected to the visible light feature preference selection interaction layer and the thermal infrared feature preference selection interaction layer, and there is inter-modal feature interaction within the two intra-modal feature processing modules. The intra-modal feature processing module includes one 1×W maximum pooling layer, one H×1 maximum pooling operation, one addition operation, one convolution operation, one multiplication operation, and two Sigmoid activation functions from input to output. The inter-modal feature interaction includes two 3×3 convolution layers, two global maximum pooling, two Softmax functions, two multiplication operations, and one splicing operation.

[0009] The crowd density map estimation module includes, from input to output, one convolution layer with a convolution kernel size of 3×3 and 256 output channels, one convolution layer with a convolution kernel size of 3×3 and 128 output channels, one convolution layer with a convolution kernel size of 1×1 and 1 output channel, and two ReLU layers.

[0010] A crowd counting method based on cross-modal feature extraction and adaptive fusion, the specific steps are as follows:

[0011] Step 1: Input the visible light image and the thermal infrared image into the local feature extraction layer for local feature extraction, and obtain a visible light local feature map and a thermal infrared local feature map respectively;

[0012] Step 2: Input the visible light image into the global feature extraction layer to extract global features and obtain a visible light global feature map;

[0013] Step 3: Input the thermal infrared image into the global feature extraction layer to extract global features and obtain a thermal infrared global feature map;

[0014] Step 4: Input the visible light local feature map and the visible light global feature map into the visible light feature preference selection interaction layer for self-learning preference selection and fusion to obtain a visible light comprehensive feature map;

[0015] Step 5: Input the thermal infrared local feature map and the thermal infrared global feature map into the thermal infrared feature preference selection interaction layer for self-learning preference selection and fusion to obtain a thermal infrared comprehensive feature map;

[0016] Step 6: Input the visible light comprehensive feature map and the thermal infrared comprehensive feature map into the lightweight cross-modal feature interactive fusion attention module for module fusion to obtain a cross-modal spatial attention map;

[0017] Step 7: Input the cross-modal spatial attention map into the crowd density map estimation layer to estimate the crowd density and obtain the estimated crowd density map;

[0018] Step 8: Add the estimated crowd density map pixel by pixel to obtain the estimated value of the number of people in the crowd image.

[0019] The further improvement of the technical solution of the present invention is that: the specific steps of step 1 are as follows:

[0020] Step 1.1: Input the visible light image and the thermal infrared image into the first convolution processing block to obtain the first visible light local feature map and the first thermal infrared local feature map

[0021] Step 1.2: Input the first visible light local feature map and the first thermal infrared local feature map into a second convolution processing block to obtain a second visible light local feature map and a second thermal infrared local feature map;

[0022] Step 1.3: Input the second visible light local feature map and the second thermal infrared local feature map into the third convolution processing block to obtain a third visible light local feature map and a third thermal infrared local feature map;

[0023] Step 1.4: Input the third visible light local feature map and the third thermal infrared local feature map into the fourth convolution processing block to obtain a fourth visible light local feature map and a fourth thermal infrared local feature map;

[0024] Step 1.5: Input the fourth visible light local feature map and the fourth thermal infrared local feature map into the fifth convolution processing block to obtain the final visible light local feature map and the thermal infrared local feature map.

[0025] The further improvement of the technical solution of the present invention is that the specific steps of step 2 are as follows:

[0026] Step 2.1.1: Perform a shearing operation on the input visible light image, and divide the visible light image into small blocks and input them into the first coding block;

[0027] Step 2.1.2: Perform a linear transformation on each block to convert it into a linear vector and generate a feature map;

[0028] Step 2.1.3: Normalize each linear vector in the feature map to make it standardized;

[0029] Step 2.1.4: Use the attention mechanism to extract the global relationship in the feature map and generate a self-attention map;

[0030] Step 2.1.5: Apply the feedforward network to perform nonlinear transformation on the self-attention map to obtain the first visible light global feature map;

[0031] Step 2.2: Perform a shearing operation on the input first visible light global feature map, divide the first visible light global feature map into small blocks and input the small blocks into the second coding block, and repeat the above steps to obtain a second visible light global feature map;

[0032] Step 2.3: Perform a shearing operation on the input second visible light global feature map, divide the second visible light global feature map into small blocks and input the small blocks into the third coding block; repeat the above steps to obtain the third visible light global feature map.

[0033] The further improvement of the technical solution of the present invention is that the specific steps of step 3 are as follows:

[0034] Step 3.1.1: Perform a shearing operation on the input thermal infrared image, and divide the thermal infrared image into small blocks and input them into the first coding block;

[0035] Step 3.1.2: Perform a linear transformation on each block to convert it into a linear vector and generate a feature map;

[0036] Step 3.1.3: Normalize each linear vector in the feature map to make it standardized;

[0037] Step 3.1.4: Use the attention mechanism to extract the global relationship in the feature map and generate a self-attention map;

[0038] Step 3.1.5: Apply the feedforward network to perform nonlinear transformation on the self-attention map to obtain the first thermal infrared global feature map;

[0039] Step 3.2: Perform a shearing operation on the input first thermal infrared global feature map, divide the first thermal infrared global feature map into small blocks and input the small blocks into the second coding block; repeat the above steps to obtain the second thermal infrared global feature map;

[0040] Step 3.3: Perform a shearing operation on the input second thermal infrared global feature map, divide the second thermal infrared global feature map into small blocks and input the small blocks into the third coding block; repeat the above steps to obtain the third thermal infrared global feature map.

[0041] The further improvement of the technical solution of the present invention is that the specific steps of steps 4 to 5 are as follows:

[0042] Step 4: multiply the variable weights by the visible light local feature map after Sigmoid transformation to obtain a weighted local feature map, multiply the variable weights by the visible light global feature map after Sigmoid transformation to obtain a weighted global feature map, add the weighted local feature map and the weighted global feature map to obtain a visible light comprehensive feature map;

[0043] Step 5: Multiply the variable weights by the thermal infrared local feature map after Sigmoid transformation to obtain a weighted local feature map. Multiply the variable weights by the thermal infrared global feature map after Sigmoid transformation to obtain a weighted global feature map. Add the weighted local feature map and the weighted global feature map to obtain a thermal infrared comprehensive feature map.

[0044] A further improvement of the technical solution of the present invention is that step 6 is specifically as follows:

[0045] Step 6.1: Process the visible light comprehensive feature map into intra-modal features;

[0046] Step 6.2: Process the thermal infrared comprehensive feature map into intra-modal features;

[0047] Step 6.3: Perform inter-modal feature fusion on the two modal processing results to obtain a cross-modal spatial attention map.

[0048] The further improvement of the technical solution of the present invention is that the specific steps of step 6.1 and step 6.2 are as follows:

[0049] Step 6.1.1: Design four parallel branches for the visible light comprehensive feature map. One is the original feature branch. The second branch is horizontally max-pooled through a 1×W max-pooling layer. The third branch is vertically max-pooled through an H×1 max-pooling layer. The above three branches are used for intra-modal feature fusion. The fourth branch is also the original feature branch and is used for subsequent inter-modal feature fusion.

[0050] Step 6.1.2: Add the feature information of the two dimensions and perform channel-level fusion through a 1×1 convolutional layer;

[0051] Step 6.1.3: Segment the fusion result horizontally and vertically and pass it through the Sigmoid function to obtain paired feature relationships in different dimensions;

[0052] Step 6.1.4: Perform matrix multiplication on the two-dimensional feature relationship and the original feature map to obtain the directional attention feature map;

[0053] Step 6.2.1: Design four parallel branches for the thermal infrared comprehensive feature map. One is the original feature branch. The second branch is subjected to the maximum pooling in the horizontal direction through the 1×W maximum pooling layer. The third branch is subjected to the maximum pooling in the vertical direction through the H×1 maximum pooling layer. The above three branches are used for intra-modal feature fusion. The fourth branch is also the original feature branch and is used for subsequent inter-modal feature fusion.

[0054] Step 6.2.2: Add the feature information of the two dimensions and perform channel-level fusion through a 1×1 convolutional layer;

[0055] Step 6.2.3: Segment the fusion result horizontally and vertically and pass it through the Sigmoid function to obtain paired feature relationships in different dimensions;

[0056] Step 6.2.4: Perform matrix multiplication on the two-dimensional feature relationship and the original feature map to obtain the directional attention feature map.

[0057] The further improvement of the technical solution of the present invention is that step 6.3 is specifically as follows:

[0058] Step 6.3.1: Perform global maximum pooling on the directional attention feature maps obtained in steps 6.1 and 6.2 and normalize them using the Sigmoid function;

[0059] Step 6.3.2: Perform matrix multiplication on the normalized directional attention feature map and the original feature map of the other modality after 3×3 convolution to obtain the interacted spatial attention map. After this operation, the two modalities respectively obtain two interacted spatial attention maps.

[0060] Step 6.3.3: Multiply the spatial attention map after interaction with the original feature map of the current modality, and add the result to the current modality to obtain the cross-modal spatial attention map. After this operation, two cross-modal spatial attention maps are obtained for the two modalities.

[0061] Step 6.3.4: Concatenate the cross-modal spatial attention maps of the two modalities in the channel dimension to obtain the final cross-modal spatial attention map.

[0062] Due to the adoption of the above technical solution, the technical progress achieved by the present invention is: by setting the local coding layer and the global coding layer, the local features and global features of the visible light image and the thermal infrared image can be obtained respectively, and the design of this module can improve the system's ability to understand the image content. By adopting the self-learning feature preference selection interaction strategy of the visible light feature preference selection interaction layer and the thermal infrared feature preference selection interaction layer, the problem of unbalanced feature extraction fusion that simple fixed weight fusion will lead to excessive extraction of some features and insufficient extraction of other features is solved, so that the model can select and fuse global-local features with preference according to different characteristics to achieve the best effect. In addition, by designing a lightweight cross-modal feature interaction fusion module, the system can simultaneously process features within and between modalities, fuse effective information between different modalities, and better adapt to target tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 It is a structural schematic diagram of the crowd counting system of the present invention;

[0065] Figure 2 is a flow chart of the crowd counting method of the present invention;

[0066] Figure 3 It is a schematic diagram of using the crowd counting method of the present invention to count people. DETAILED DESCRIPTION

[0067] The present invention is further described in detail below in conjunction with embodiments:

[0068] like Figure 1As shown, it is a structural diagram of a crowd counting system with cross-modal feature extraction and adaptive fusion. Before the crowd counting system is used, it is necessary to use multiple visible light images and thermal infrared images in the same scene to train the constructed crowd counting system, and use the real crowd density map as supervision to calculate the loss, and use the Adam optimization algorithm so that the system can learn the optimal parameters. It includes a parallel visible light image acquisition module and a thermal infrared image acquisition module. The visible light image acquisition module is sequentially connected with a visible light double-level feature extraction network layer and a visible light feature preference selection interaction layer. The thermal infrared image acquisition module is sequentially connected with a thermal infrared double-level feature extraction network layer and a thermal infrared feature preference selection interaction layer. The visible light feature preference selection interaction layer and the thermal infrared feature preference selection interaction layer are connected with a common lightweight cross-modal feature interaction fusion attention module and a crowd density map estimation module. The specific composition of each module is as follows:

[0069] The visible light dual-level feature extraction network layer and the thermal infrared dual-level feature extraction network layer have the same structure, both of which include parallel local feature extraction layers and global feature extraction layers. The local feature extraction layer is the remaining part of the VGG19 network after removing the fully connected layer and the Softmax layer, including 16 convolutional layers and 5 maximum pooling layers. From input to output, the first convolution processing block is: 2 3×3 convolutional layers, 1 maximum pooling layer, the second convolution processing block is: 2 3×3 convolutional layers, 1 maximum pooling layer, the third convolution processing block: 4 3×3 convolution layers, 1 maximum pooling layer, the fourth convolution processing block: 4 3×3 convolution layers, 1 maximum pooling layer and the fifth convolution processing block: 4 3×3 convolution layers. The number of feature map channels generated by each convolution layer is 64, 64, 128, 128, 256, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512 and 512 respectively, and the step size of the pooling layer is 2.

[0070] Global feature extraction layer: It includes three Transformer encoding blocks, namely the first, second and third encoding blocks, with output channels of 128, 256 and 512 respectively. Each encoding block includes 1 clipping operation, 1 linear transformation operation, 1 normalization operation, 1 attention layer and 1 feedforward layer.

[0071] The visible light feature preference selection interaction layer and the thermal infrared feature preference selection interaction layer have the same structure and together constitute the self-learning feature preference selection interaction layer, wherein the visible light feature preference selection interaction layer and the thermal infrared feature preference selection interaction layer both include one addition operation and two Sigmoid activation functions from input to output;

[0072] The lightweight cross-modal feature interaction fusion attention module includes two parallel intra-modal feature processing modules, which are respectively connected to the visible light feature preference selection interaction layer and the thermal infrared feature preference selection interaction layer, and there is inter-modal feature interaction within the two intra-modal feature processing modules. The intra-modal feature processing module includes a 1×W maximum pooling layer, a H×1 maximum pooling operation, an addition operation, a convolution operation with a convolution kernel of 1×1, a multiplication operation, and two Sigmoid activation functions. The inter-modal feature interaction includes two 3×3 convolution layers, two global maximum pooling, two Softmax functions, two multiplication operations, and a splicing operation.

[0073] The crowd density map estimation module includes, from input to output, one convolution layer with a convolution kernel size of 3×3 and 256 output channels, one convolution layer with a convolution kernel size of 3×3 and 128 output channels, one convolution layer with a convolution kernel size of 1×1 and 1 output channel, and two ReLU layers.

[0074] The crowd counting method using the above cross-modal feature extraction and adaptive fusion crowd counting is as follows: Figure 2 As shown, the specific method is as follows:

[0075] Step 1: Obtain corresponding visible light images and thermal infrared images through the visible light image acquisition module and the thermal infrared image acquisition module, input the visible light image and the thermal infrared image into the local feature extraction layer for local feature extraction, and obtain visible light local feature maps and thermal infrared local feature maps respectively;

[0076] Step 1.1: Input the visible light image and thermal infrared image into the first convolution processing block, and sequentially pass through two convolution layers with a convolution kernel size of 3×3, a step size of 1, and an output channel of 64, and a maximum pooling layer with a pooling kernel of 2×2 and a step size of 2, to obtain a visible light local feature map and a thermal infrared local feature map with 64 channels;

[0077] Step 1.2: Input the visible light local feature map and the thermal infrared local feature map with 64 channels into the second convolution processing block, and sequentially pass through two convolution layers with a convolution kernel size of 3×3, a step size of 1, and an output channel number of 128, and a maximum pooling layer with a pooling kernel of 2×2 and a step size of 2, to obtain a visible light local feature map and a thermal infrared local feature map with an output channel number of 128;

[0078] Step 1.3: Input the visible light local feature map and thermal infrared local feature map with 128 channels into the third convolution processing block, and sequentially pass through four convolutions and convolution layers with a size of 3×3, a step size of 1, and an output channel number of 256, and a maximum pooling layer with a pooling kernel of 2×2 and a step size of 2, to obtain the visible light local feature map and thermal infrared local feature map with an output channel number of 256;

[0079] Step 1.4: Input the visible light local feature map and thermal infrared local feature map with 256 channels into the fourth convolution processing block, and sequentially pass through four convolutions and convolution layers with a size of 3×3, a step size of 1, and an output channel number of 256, and a maximum pooling layer with a pooling kernel of 2×2 and a step size of 2, to obtain a visible light local feature map and a thermal infrared local feature map with an output channel number of 512;

[0080] Step 1.5: Input the visible light local feature map and thermal infrared local feature map with a channel number of 512 into the fifth convolution processing block, and successively pass through four convolutions and a convolution layer with a size of 3×3, a step size of 1, and an output channel number of 512, and a maximum pooling layer with a pooling kernel of 2×2 and a step size of 2 to obtain the visible light local feature map and thermal infrared local feature map with an output channel number of 512.

[0081] Step 2: Input the visible light image into the global feature extraction layer to extract global features and obtain a visible light global feature map;

[0082] Step 2.1: Input the visible light image into the first global encoding block to generate a first visible light global feature map;

[0083] Step 2.1.1: Perform a shearing operation on the input visible light image, and divide the visible light image into small blocks and input them into the first coding block;

[0084] Step 2.1.2: Perform a linear transformation on each block to convert it into a linear vector and generate a feature map;

[0085] Step 2.1.3: Normalize each linear vector in the feature map to make it standardized;

[0086] Step 2.1.4: Use the attention mechanism to extract the global relationship in the feature map and generate a self-attention map;

[0087] Step 2.1.5: Apply the feedforward network to perform nonlinear transformation on the self-attention map to obtain the first visible light global feature map;

[0088] Step 2.2: Input the first visible light global feature map into the second global encoding block to generate a second visible light global feature map;

[0089] Step 2.2.1: Perform a shearing operation on the input first visible light global feature map, and divide the first visible light global feature map into small blocks and input them into the second encoding block;

[0090] Step 2.2.2: Perform a linear transformation on each block to convert it into a linear vector and generate a feature map;

[0091] Step 2.2.3: Normalize each linear vector in the feature map to make it standardized;

[0092] Step 2.2.4: Use the attention mechanism to extract the global relationship in the feature map and generate a self-attention map;

[0093] Step 2.2.5: Apply the feedforward network to perform nonlinear transformation on the self-attention map to obtain the second visible light global feature map;

[0094] Step 2.3: input the second visible light global feature map into the third global encoding block to generate a final visible light global feature map;

[0095] Step 2.3.1: Perform a shearing operation on the input second visible light global feature map, and divide the second visible light global feature map into small blocks and input them into the third encoding block;

[0096] Step 2.3.2: Perform a linear transformation on each block to convert it into a linear vector and generate a feature map;

[0097] Step 2.3.3: Normalize each linear vector in the feature map to make it standardized;

[0098] Step 2.3.4: Use the attention mechanism to extract the global relationship in the feature map and generate a self-attention map;

[0099] Step 2.3.5: Apply the feedforward network to perform nonlinear transformation on the self-attention map to obtain the final visible light global feature map;

[0100] Step 3: Input the thermal infrared image into the global feature extraction layer to extract global features and obtain a thermal infrared global feature map;

[0101] Step 3.1: Input the thermal infrared image into the first global encoding block to generate a first thermal infrared global feature map;

[0102] Step 3.1.1: Perform a shearing operation on the input thermal infrared image, and divide the thermal infrared image into small blocks and input them into the first coding block;

[0103] Step 3.1.2: Perform a linear transformation on each block to convert it into a linear vector and generate a feature map;

[0104] Step 3.1.3: Normalize each linear vector in the feature map to make it standardized;

[0105] Step 3.1.4: Use the attention mechanism to extract the global relationship in the feature map and generate a self-attention map;

[0106] Step 3.1.5: Apply the feedforward network to perform nonlinear transformation on the self-attention map to obtain the first thermal infrared global feature map;

[0107] Step 3.2: Input the first thermal infrared global feature map into the second global encoding block to generate a second thermal infrared global feature map;

[0108] Step 3.2.1: Perform a shearing operation on the input first thermal infrared global feature map, and divide the first thermal infrared global feature map into small blocks and input them into the second encoding block;

[0109] Step 3.2.2: Perform a linear transformation on each block to convert it into a linear vector and generate a feature map;

[0110] Step 3.2.3: Normalize each linear vector in the feature map to make it standardized;

[0111] Step 3.2.4: Use the attention mechanism to extract the global relationship in the feature map and generate a self-attention map;

[0112] Step 3.2.5: Apply the feedforward network to perform nonlinear transformation on the self-attention map to obtain the second thermal infrared global feature map;

[0113] Step 3.3: input the second thermal infrared global feature map into the third global encoding block to generate a final thermal infrared global feature map;

[0114] Step 3.3.1: Perform a shearing operation on the input second thermal infrared global feature map, and divide the second thermal infrared global feature map into small blocks and input them into the third encoding block;

[0115] Step 3.3.2: Perform a linear transformation on each block to convert it into a linear vector and generate a feature map;

[0116] Step 3.3.3: Normalize each linear vector in the feature map to make it standardized;

[0117] Step 3.3.4: Use the attention mechanism to extract the global relationship in the feature map and generate a self-attention map;

[0118] Step 3.3.5: Apply the feedforward network to perform nonlinear transformation on the self-attention map to obtain the final thermal infrared global feature map;

[0119] Step 4: Input the visible light local feature map and the visible light global feature map into the visible light feature preference selection interaction layer for self-learning preference selection and fusion to obtain a visible light comprehensive feature map;

[0120] Step 4.1: Multiply the variable weights by the visible light local feature map after Sigmoid transformation to obtain a weighted local feature map, multiply the variable weights by the visible light global feature map after Sigmoid transformation to obtain a weighted global feature map, and add the weighted local feature map to the weighted global feature map to obtain a visible light comprehensive feature map.

[0121] Step 5: Input the thermal infrared local feature map and the thermal infrared global feature map into the thermal infrared feature preference selection interaction layer for self-learning preference selection and fusion to obtain a thermal infrared comprehensive feature map;

[0122] Step 5.1: Multiply the variable weights by the thermal infrared local feature map after Sigmoid transformation to obtain a weighted local feature map, multiply the variable weights by the thermal infrared global feature map after Sigmoid transformation to obtain a weighted global feature map, and add the weighted local feature map to the weighted global feature map to obtain a thermal infrared comprehensive feature map.

[0123] Step 6: Input the visible light comprehensive feature map and the thermal infrared comprehensive feature map into the lightweight cross-modal feature interactive fusion attention module for module fusion to obtain a cross-modal spatial attention map;

[0124] Step 6.1: Process the visible light comprehensive feature map into intra-modal features;

[0125] Step 6.1.1: Design four parallel branches for the visible light comprehensive feature map. One is the original feature branch. The second branch is horizontally max-pooled through a 1×W max-pooling layer. The third branch is vertically max-pooled through an H×1 max-pooling layer. The above three branches are used for intra-modal feature fusion. The fourth branch is also the original feature branch and is used for subsequent inter-modal feature fusion.

[0126] Step 6.1.2: Add the feature information of the two dimensions and perform channel-level fusion through a 1×1 convolutional layer;

[0127] Step 6.1.3: Segment the fusion result horizontally and vertically and pass it through the Sigmoid function to obtain paired feature relationships in different dimensions;

[0128] Step 6.1.4: Perform matrix multiplication on the two-dimensional feature relationship and the original feature map to obtain the directional attention feature map;

[0129] Step 6.2: Process the thermal infrared comprehensive feature map into intra-modal features;

[0130] Step 6.2.1: Design four parallel branches for the thermal infrared comprehensive feature map. One is the original feature branch. The second branch is subjected to the maximum pooling in the horizontal direction through the 1×W maximum pooling layer. The third branch is subjected to the maximum pooling in the vertical direction through the H×1 maximum pooling layer. The above three branches are used for intra-modal feature fusion. The fourth branch is also the original feature branch and is used for subsequent inter-modal feature fusion.

[0131] Step 6.2.2: Add the feature information of the two dimensions and perform channel-level fusion through a 1×1 convolutional layer;

[0132] Step 6.2.3: Segment the fusion result horizontally and vertically and pass it through the Sigmoid function to obtain paired feature relationships in different dimensions;

[0133] Step 6.2.4: Perform matrix multiplication on the two-dimensional feature relationship and the original feature map to obtain the directional attention feature map.

[0134] Step 6.3: Perform inter-modal feature fusion on the two modal processing results to obtain a cross-modal spatial attention map.

[0135] Step 6.3.1: Perform global maximum pooling on the directional attention feature maps obtained in steps 6.1 and 6.2 and normalize them using the Sigmoid function;

[0136] Step 6.3.2: Perform matrix multiplication on the normalized directional attention feature map and the original feature map of the other modality after 3×3 convolution to obtain the interacted spatial attention map. After this operation, the two modalities respectively obtain two interacted spatial attention maps.

[0137] Step 6.3.3: Multiply the spatial attention map after interaction with the original feature map of the current modality, and add the result to the current modality to obtain the cross-modal spatial attention map. After this operation, two cross-modal spatial attention maps are obtained for the two modalities.

[0138] Step 6.3.4: Concatenate the cross-modal spatial attention maps of the two modalities in the channel dimension to obtain the final cross-modal spatial attention map.

[0139] Step 7: Input the cross-modal spatial attention map into the crowd density map estimation layer to estimate the crowd density and obtain the estimated crowd density map;

[0140] Step 7.1: Input the cross-modal spatial attention map into the crowd density map estimation layer, and pass through a convolution layer with a convolution kernel size of 3×3 and an output channel of 256, a ReLU layer, a convolution layer with a convolution kernel size of 3×3 and an output channel of 128, a ReLU layer, a convolution layer with a convolution kernel size of 1×1 and an output channel of 1, and obtain the estimated crowd density map.

[0141] Step 8: Add the estimated crowd density map pixel by pixel to obtain the estimated value of the number of people in the crowd image.

[0142] Figure 3 Schematic diagram of crowd counting using the cross-modal feature extraction and adaptive fusion crowd counting of the present invention. Figure 3 When the crowd counting method of the present invention is used for crowd counting, the visible light image and thermal infrared image of the crowd in the target scene are input into the established cross-modal feature extraction and fusion crowd counting, and the trained and optimized crowd counting system is used to estimate the crowd density. The obtained estimated crowd density map is added pixel by pixel to obtain the corresponding crowd counting result.

[0143] The embodiments described above are merely descriptions of preferred implementation modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A crowd counting system with cross-modal feature extraction and adaptive fusion, comprising a parallel visible light image acquisition module and a thermal infrared image acquisition module, characterized in that: The visible light image acquisition module is sequentially connected with a visible light two-level feature extraction network layer and a visible light feature preference selection interaction layer; the thermal infrared image acquisition module is sequentially connected with a thermal infrared two-level feature extraction network layer and a thermal infrared feature preference selection interaction layer; the visible light feature preference selection interaction layer and the thermal infrared feature preference selection interaction layer are connected with a common lightweight cross-modal feature interaction fusion attention module and a crowd density map estimation module.

2. A crowd counting system for cross-modal feature extraction and adaptive fusion according to claim 1, characterized in that: The visible light double-level feature extraction network layer and the thermal infrared double-level feature extraction network layer have the same structure, both of which include parallel local feature extraction layers and global feature extraction layers, where the local feature extraction layers are, from input to output, the first convolution processing block: 2 convolution layers, 1 maximum pooling layer, the second convolution processing block: 2 convolution layers, 1 maximum pooling layer, the third convolution processing block: 4 convolution layers, 1 maximum pooling layer, the fourth convolution processing block: 4 convolution layers, 1 maximum pooling layer and the fifth convolution processing block: 4 convolution layers, 1 maximum pooling layer; the global feature extraction layer includes 3 Transformer encoding blocks, namely the first, second and third encoding blocks, each of which includes 1 shearing operation, 1 linear transformation operation, 1 normalization operation, 1 attention layer and 1 feedforward layer; The visible light feature preference selection interaction layer and the thermal infrared feature preference selection interaction layer have the same structure, including one addition operation and two Sigmoid activation functions from input to output; The lightweight cross-modal feature interaction fusion attention module includes two parallel intra-modal feature processing modules, which are respectively connected to the visible light feature preference selection interaction layer and the thermal infrared feature preference selection interaction layer, and there is inter-modal feature interaction within the two intra-modal feature processing modules. The intra-modal feature processing module includes one 1×W maximum pooling layer, one H×1 maximum pooling operation, one addition operation, one convolution operation, one multiplication operation, and two Sigmoid activation functions from input to output. The inter-modal feature interaction includes two 3×3 convolution layers, two global maximum pooling, two Softmax functions, two multiplication operations, and one concatenation operation. The crowd density map estimation module includes, from input to output, one convolution layer with a convolution kernel size of 3×3 and 256 output channels, one convolution layer with a convolution kernel size of 3×3 and 128 output channels, one convolution layer with a convolution kernel size of 1×1 and 1 output channel, and two ReLU layers.

3. A crowd counting method for cross-modal feature extraction and adaptive fusion, applied to the counting system as claimed in claim 1 or 2, characterized in that: The specific steps are as follows: Step 1: Input the visible light image and the thermal infrared image into the local feature extraction layer for local feature extraction, and obtain a visible light local feature map and a thermal infrared local feature map respectively; Step 2: Input the visible light image into the global feature extraction layer to extract global features and obtain a visible light global feature map; Step 3: Input the thermal infrared image into the global feature extraction layer to extract global features and obtain a thermal infrared global feature map; Step 4: Input the visible light local feature map and the visible light global feature map into the visible light feature preference selection interaction layer for self-learning preference selection and fusion to obtain a visible light comprehensive feature map; Step 5: Input the thermal infrared local feature map and the thermal infrared global feature map into the thermal infrared feature preference selection interaction layer for self-learning preference selection and fusion to obtain a thermal infrared comprehensive feature map; Step 6: Input the visible light comprehensive feature map and the thermal infrared comprehensive feature map into the lightweight cross-modal feature interactive fusion attention module for module fusion to obtain a cross-modal spatial attention map; Step 7: Input the cross-modal spatial attention map into the crowd density map estimation layer to estimate the crowd density and obtain the estimated crowd density map; Step 8: Add the estimated crowd density map pixel by pixel to obtain the estimated value of the number of people in the crowd image.

4. The method for crowd counting based on cross-modal feature extraction and adaptive fusion according to claim 3 is characterized in that: Step 1 The specific steps are as follows: Step 1.1: Input the visible light image and the thermal infrared image into the first convolution processing block to obtain the first visible light local feature map and the first thermal infrared local feature map Step 1.2: Input the first visible light local feature map and the first thermal infrared local feature map into a second convolution processing block to obtain a second visible light local feature map and a second thermal infrared local feature map; Step 1.3: Input the second visible light local feature map and the second thermal infrared local feature map into the third convolution processing block to obtain a third visible light local feature map and a third thermal infrared local feature map; Step 1.4: Input the third visible light local feature map and the third thermal infrared local feature map into the fourth convolution processing block to obtain a fourth visible light local feature map and a fourth thermal infrared local feature map; Step 1.5: Input the fourth visible light local feature map and the fourth thermal infrared local feature map into the fifth convolution processing block to obtain the final visible light local feature map and the thermal infrared local feature map.

5. The method for crowd counting based on cross-modal feature extraction and adaptive fusion according to claim 3, characterized in that: Step 2 The specific steps are as follows: Step 2.1.1: Perform a shearing operation on the input visible light image, and divide the visible light image into small blocks and input them into the first coding block; Step 2.1.2: Perform a linear transformation on each block to convert it into a linear vector and generate a feature map; Step 2.1.3: Normalize each linear vector in the feature map to make it standardized; Step 2.1.4: Use the attention mechanism to extract the global relationship in the feature map and generate a self-attention map; Step 2.1.5: Apply the feedforward network to perform nonlinear transformation on the self-attention map to obtain the first visible light global feature map; Step 2.2: Perform a shearing operation on the input first visible light global feature map, divide the first visible light global feature map into small blocks and input the small blocks into the second coding block, and repeat the above steps to obtain a second visible light global feature map; Step 2.3: Perform a shearing operation on the input second visible light global feature map, divide the second visible light global feature map into small blocks and input the small blocks into the third coding block; repeat the above steps to obtain the third visible light global feature map.

6. The method for crowd counting based on cross-modal feature extraction and adaptive fusion according to claim 5, characterized in that: Step 3 The specific steps are as follows: Step 3.1.1: Perform a shearing operation on the input thermal infrared image, and divide the thermal infrared image into small blocks and input them into the first coding block; Step 3.1.2: Perform a linear transformation on each block to convert it into a linear vector and generate a feature map; Step 3.1.3: Normalize each linear vector in the feature map to make it standardized; Step 3.1.4: Use the attention mechanism to extract the global relationship in the feature map and generate a self-attention map; Step 3.1.5: Apply the feedforward network to perform nonlinear transformation on the self-attention map to obtain the first thermal infrared global feature map; Step 3.2: Perform a shearing operation on the input first thermal infrared global feature map, divide the first thermal infrared global feature map into small blocks and input the small blocks into the second coding block; repeat the above steps to obtain the second thermal infrared global feature map; Step 3.3: Perform a shearing operation on the input second thermal infrared global feature map, divide the second thermal infrared global feature map into small blocks and input the small blocks into the third coding block; repeat the above steps to obtain the third thermal infrared global feature map.

7. The method for crowd counting based on cross-modal feature extraction and adaptive fusion according to claim 3, characterized in that: The specific steps of steps 4 to 5 are as follows: Step 4: multiply the variable weights by the visible light local feature map after Sigmoid transformation to obtain a weighted local feature map, multiply the variable weights by the visible light global feature map after Sigmoid transformation to obtain a weighted global feature map, add the weighted local feature map and the weighted global feature map to obtain a visible light comprehensive feature map; Step 5: Multiply the variable weights by the thermal infrared local feature map after Sigmoid transformation to obtain a weighted local feature map. Multiply the variable weights by the thermal infrared global feature map after Sigmoid transformation to obtain a weighted global feature map. Add the weighted local feature map and the weighted global feature map to obtain a thermal infrared comprehensive feature map.

8. The method for crowd counting based on cross-modal feature extraction and adaptive fusion according to claim 3, characterized in that: Step 6 is as follows: Step 6.1: Process the visible light comprehensive feature map into intra-modal features; Step 6.2: Process the thermal infrared comprehensive feature map into intra-modal features; Step 6.3: Perform inter-modal feature fusion on the two modal processing results to obtain a cross-modal spatial attention map.

9. The method for crowd counting based on cross-modal feature extraction and adaptive fusion according to claim 8, characterized in that: The specific steps of step 6.1 and step 6.2 are as follows: Step 6.1.1: Design four parallel branches for the visible light comprehensive feature map. One is the original feature branch. The second branch is horizontally max-pooled through a 1×W max-pooling layer. The third branch is vertically max-pooled through an H×1 max-pooling layer. The above three branches are used for intra-modal feature fusion. The fourth branch is also the original feature branch and is used for subsequent inter-modal feature fusion. Step 6.1.2: Add the feature information of the two dimensions and perform channel-level fusion through a 1×1 convolutional layer; Step 6.1.3: Segment the fusion result horizontally and vertically and pass it through the Sigmoid function to obtain paired feature relationships in different dimensions; Step 6.1.4: Perform matrix multiplication on the two-dimensional feature relationship and the original feature map to obtain the directional attention feature map; Step 6.2.1: Design four parallel branches for the thermal infrared comprehensive feature map. One is the original feature branch. The second branch is subjected to the maximum pooling in the horizontal direction through the 1×W maximum pooling layer. The third branch is subjected to the maximum pooling in the vertical direction through the H×1 maximum pooling layer. The above three branches are used for intra-modal feature fusion. The fourth branch is also the original feature branch and is used for subsequent inter-modal feature fusion. Step 6.2.2: Add the feature information of the two dimensions and perform channel-level fusion through a 1×1 convolutional layer; Step 6.2.3: Segment the fusion result horizontally and vertically and pass it through the Sigmoid function to obtain paired feature relationships in different dimensions; Step 6.2.4: Perform matrix multiplication on the two-dimensional feature relationship and the original feature map to obtain the directional attention feature map.

10. The method for crowd counting based on cross-modal feature extraction and adaptive fusion according to claim 8, characterized in that: Step 6.3 is as follows: Step 6.3.1: Perform global maximum pooling on the directional attention feature maps obtained in steps 6.1 and 6.2 and normalize them using the Sigmoid function; Step 6.3.2: Perform matrix multiplication on the normalized directional attention feature map and the original feature map of the other modality after 3×3 convolution to obtain the interacted spatial attention map. After this operation, the two modalities respectively obtain two interacted spatial attention maps. Step 6.3.3: Multiply the spatial attention map after interaction with the original feature map of the current modality, and add the result to the current modality to obtain the cross-modal spatial attention map. After this operation, two cross-modal spatial attention maps are obtained for the two modalities. Step 6.3.4: Concatenate the cross-modal spatial attention maps of the two modalities in the channel dimension to obtain the final cross-modal spatial attention map.

Images