A two-stage feature alignment fusion counting system and method for cross-modal spatial dislocation

Abstract

The present invention discloses a two-stage feature alignment fusion counting system and method for cross-modal spatial dislocation, which relates to the field of computer vision technology, including: obtaining a visible light image and a thermal infrared image of a scene to be counted with cross-modal spatial dislocation; inputting the visible light image and the thermal infrared image into the two-stage feature alignment fusion counting system to obtain an estimated crowd density map; wherein the two-stage feature alignment fusion counting system includes a shared weight initial feature extraction layer, a parallel two-level feature alignment stage, a low-frequency attention filtering fusion stage and a crowd density map estimation layer; the estimated crowd density map is added pixel by pixel to obtain an estimated value of the number of people in the scene crowd image. The present invention can effectively solve the problem of large counting errors caused by the cross-modal spatial dislocation phenomenon of visible light-thermal infrared images.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a dual-stage feature alignment fusion counting system and method for cross-modal spatial dislocation. Background Art

[0002] Crowd counting is to predict the number of people in a single image or video frame. It has important applications in crowd status monitoring, urban traffic management, and other intelligent security and public safety. Therefore, the research on crowd counting theory, methods, and key technologies has received extensive attention in academia and industry.

[0003] At present, crowd counting using visible light data has achieved satisfactory performance. However, in real-world applications where illumination is uncontrollable, the accuracy of crowd counting using only single visible light data imaged by illumination is greatly limited. Therefore, the industry has turned to crowd counting research based on visible light-thermal infrared cross-modal fusion. However, since visible light images and thermal infrared images are derived from scene data acquisition devices with completely different imaging principles, they have an unavoidable spatial misalignment problem. Spatial misalignment refers to the same scene target (the same pedestrian individual in this task) being in different spatial positions in paired visible light images and thermal infrared images. This spatial misalignment phenomenon makes the existing cross-modal crowd counting method based on the assumption of complete alignment of visible light and thermal infrared modalities have great limitations in extracting crowd features in cross-modal scenes, which seriously affects the accuracy of crowd number prediction results. Therefore, it is urgent to carry out new cross-modal crowd counting method technology research that is robust to the visible light-thermal infrared cross-modal spatial misalignment problem. In order to eliminate the impact of spatial misalignment on cross-modal crowd counting. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a two-stage feature alignment fusion counting system and method for cross-modal spatial misalignment, which can effectively eliminate the limitation of visible light-thermal infrared cross-modal spatial misalignment problem on the feature representation effect in the counting process and then the negative impact on the final scene counting accuracy, thereby improving the counting performance of the cross-modal crowd counting model.

[0005] To solve the above technical problems, the technical solution adopted by the present invention is: a two-stage feature alignment fusion counting system for cross-modal spatial dislocation, including a shared weight initial feature extraction layer, a parallel two-level feature alignment stage, a low-frequency attention filtering fusion stage, and a crowd density map estimation layer connected in sequence, wherein the parallel two-level feature alignment stage includes a spatial feature alignment unit, a semantic feature alignment unit and a two-level feature fusion unit.

[0006] The further improvement of the technical solution of the present invention is that the specific structure of each layer is as follows:

[0007] The shared weight initial feature extraction layer includes 2 convolution layers, 1 maximum pooling layer, 2 convolution layers, 1 maximum pooling layer, 4 convolution layers, 1 maximum pooling layer, 4 convolution layers, 1 maximum pooling layer, and 4 convolution layers in sequence. The convolution kernel size of each convolution layer is 3×3, the step size is 1, and the number of channels of the generated feature maps is 64, 64, 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512, and 512 from input to output, and the step size of the maximum pooling layer is 2;

[0008] The parallel two-level feature alignment stage includes a spatial feature alignment unit, a semantic feature alignment unit, and a two-level feature fusion unit:

[0009] The spatial feature alignment unit sequentially includes 1 channel splicing operation, 1 first convolution processing block for downsampling operation, 1 second convolution processing block for downsampling operation, 1 third convolution processing block for downsampling operation, and 1 fourth convolution processing block for generating transformation parameters; the structures of the first convolution processing block, the second convolution processing block, and the third convolution processing block are sequentially including 1 convolution layer with a convolution kernel size of 3×3, an output channel number of 128, and a step size of 2, 1 batch normalization, 1 ReLU activation function, 1 convolution kernel size of 1×1, an output channel number of 128, and a step size of 2, 1 batch normalization, and 1 ReLU activation function; the fourth convolution processing block sequentially includes 1 adaptive average pooling layer with a window size of 1×1, 1 convolution kernel size of 1×1, an output channel number of 6, and a step size of 2, and 1 bilinear interpolation spatial sampling operation;

[0010] The semantic feature alignment unit includes 1 channel splicing operation, 1 convolution layer with a convolution kernel size of 3×3 and 18 output channels, 1 convolution layer with a convolution kernel size of 3×3 and 512 output channels, 1 ReLU activation function, and 1 addition operation.

[0011] The two-level feature fusion unit sequentially includes one channel splicing operation, one convolution layer with a convolution kernel size of 1×1 and an output channel number of 512;

[0012] The low-frequency attention filter fusion stage includes 1 discrete wavelet transform operation, 1 low-frequency component attention unit, 1 multiplication operation, 1 channel splicing operation, and 1 inverse wavelet transform operation. The low-frequency component attention unit includes 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 512, 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 512, and 1 Sigmoid activation function.

[0013] The crowd density map estimation layer sequentially includes 1 bilinear interpolation upsampling layer with a sampling rate of 2, 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 256, 1 ReLU activation function, 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 128, 1 ReLU activation function, 1 convolution layer with a convolution kernel size of 1×1 and an output channel number of 1. The purpose of the last convolution layer with an output channel number of 1 is to generate an estimated crowd density map.

[0014] A two-stage feature alignment fusion counting method for cross-modal spatial misalignment, the steps are as follows:

[0015] Step 1: Input the visible light image and thermal infrared image into the shared weight initial feature extraction layer to extract initial features of different modalities, and obtain the visible light initial feature map and thermal infrared initial feature map respectively;

[0016] Step 2: Input the visible light initial feature map and the thermal infrared initial feature map into the parallel two-level feature alignment stage for spatial-semantic two-level feature alignment to obtain a cross-modal aligned feature map;

[0017] Step 3: Input the cross-modal aligned feature map into the low-frequency attention filtering fusion stage for frequency domain feature alignment fusion to obtain a dual-stage aligned fusion feature map;

[0018] Step 4: Input the dual-stage aligned fusion feature map into the crowd density map estimation layer to estimate the crowd density and obtain the estimated crowd density map;

[0019] Step 5: Add the estimated crowd density map pixel by pixel to obtain the estimated value of the number of people in the crowd image.

[0020] The further improvement of the technical solution of the present invention is that step 2 is specifically as follows:

[0021] Step 2.1: Input the visible light initial feature map and the thermal infrared initial feature map into the spatial feature alignment unit to obtain a spatial alignment feature map;

[0022] Step 2.2: Input the visible light initial feature map and the thermal infrared initial feature map into the semantic feature alignment unit to obtain a semantic alignment feature map;

[0023] Step 2.3: Input the spatial alignment feature map and the semantic alignment feature map into the two-level feature fusion unit to obtain the visible light alignment feature map;

[0024] Step 2.4: Add the visible light alignment feature map to the thermal infrared initial feature map to obtain the cross-modal alignment feature map.

[0025] The further improvement of the technical solution of the present invention is that step 2.1 is specifically as follows:

[0026] Step 2.1.1: Perform channel splicing on the visible light initial feature map and the thermal infrared initial feature map to obtain a cross-modal feature map;

[0027] Step 2.1.2: Input the cross-modal feature map into the first convolution processing block, and pass through the convolution layer with a convolution kernel size of 3×3, an output channel number of 128, a stride of 2, batch normalization, ReLU activation function, a convolution kernel size of 1×1, an output channel number of 128, a stride of 2, batch normalization, and ReLU activation function of the first convolution processing block in turn to obtain the first spatial alignment feature map;

[0028] Step 2.1.3: Input the first spatial alignment feature map into the second convolution processing block, and sequentially pass through the convolution layer with a convolution kernel size of 3×3, an output channel number of 128, a step size of 2, batch normalization, ReLU activation function, a convolution kernel size of 1×1, an output channel number of 128, a step size of 2, batch normalization, and ReLU activation function of the second convolution processing block to obtain the second spatial alignment feature map;

[0029] Step 2.1.4: Input the second spatial alignment feature map into the third convolution processing block, and sequentially pass through the convolution layer with a convolution kernel size of 3×3, an output channel number of 128, a step size of 2, batch normalization, ReLU activation function, a convolution kernel size of 1×1, an output channel number of 128, a step size of 2, batch normalization, and ReLU activation function of the third convolution processing block to obtain the third spatial alignment feature map;

[0030] Step 2.1.5: Input the third spatially aligned feature map into the fourth convolution processing block, and pass through the fourth convolution processing block's adaptive average pooling layer with a window size of 1×1, a convolution layer with a convolution kernel size of 1×1, an output channel number of 6, a step size of 2, and a bilinear interpolation spatial sampling operation to obtain a spatially aligned feature map.

[0031] The further improvement of the technical solution of the present invention is that step 2.2 is specifically as follows:

[0032] Step 2.2.1: perform channel splicing on the initial visible light feature map and the initial thermal infrared feature map to obtain a cross-modal feature map;

[0033] Step 2.2.2: Input the cross-modal feature map into a convolutional layer with a convolution kernel size of 3×3 and an output channel number of 18 to obtain an offset feature map;

[0034] Step 2.2.3: Add the offset feature map and the cross-modal feature map and input them into the convolution layer with a convolution kernel size of 3×3 and an output channel number of 512, and the ReLU activation function to obtain a preliminary semantic alignment feature map;

[0035] Step 2.2.4: Add the initial visible light feature map to the preliminary semantic alignment feature map to obtain the semantic alignment feature map.

[0036] The further improvement of the technical solution of the present invention is that step 2.3 is specifically as follows:

[0037] Step 2.3.1: Concatenate the spatial alignment feature map and the semantic alignment feature map to obtain a preliminary visible light alignment feature map;

[0038] Step 2.3.2: Input the preliminary visible light alignment feature map into a convolution layer with a convolution kernel size of 1×1 and an output channel number of 512 to obtain a visible light alignment feature map.

[0039] The further improvement of the technical solution of the present invention is that step 3 is specifically as follows:

[0040] Step 3.1: Perform discrete wavelet transform on the cross-modal alignment feature map to obtain low-frequency components, diagonal high-frequency components, vertical high-frequency components, and horizontal high-frequency components;

[0041] Step 3.2: The low-frequency components are sequentially passed through two convolution layers with a convolution kernel size of 3×3 and an output channel number of 512 and a Sigmoid activation function of the low-frequency component attention unit to perform attention calculation to obtain a low-frequency attention map;

[0042] Step 3.3: Multiply the low-frequency component with the low-frequency attention map to obtain the filtered low-frequency component;

[0043] Step 3.4: After channel splicing of the filtered low-frequency components and the diagonal high-frequency components, vertical high-frequency components, and horizontal high-frequency components, a discrete inverse wavelet transform is performed to obtain a two-stage aligned fusion feature map.

[0044] The further improvement of the technical solution of the present invention is that step 4 is specifically as follows:

[0045] Step 4.1: Input the two-stage aligned fusion feature map into the crowd density map estimation layer, and then pass through the bilinear interpolation upsampling layer with a sampling rate of 2 to obtain the preliminary crowd feature map;

[0046] Step 4.2: The preliminary crowd feature map is sequentially passed through a convolution layer with a convolution kernel size of 3×3 and an output channel number of 256, a ReLU activation function, a convolution layer with a convolution kernel size of 3×3 and an output channel number of 128, a ReLU activation function, and a convolution layer with a convolution kernel size of 1×1 and an output channel number of 1 to obtain an estimated crowd density map.

[0047] Due to the adoption of the above technical scheme, the technical progress achieved by the present invention is: using a shared weight feature extraction layer to perform initial feature extraction on the visible light modality and thermal infrared modality input images, and extracting complementary scene features of different modality images; by designing a spatial and semantic dual-level joint perception alignment strategy, the spatial feature and semantic feature level joint alignment of the visible light-thermal infrared cross-modal spatial misalignment is achieved; based on the research fact that the alignment artifacts mainly exist in the low-frequency component part, the low-frequency attention filtering and fusion stage is designed, which will be able to effectively eliminate the feature fusion ghosting artifact phenomenon caused by the spatial misalignment problem. The counting core design of the above shared feature extraction encoding, dual-level feature alignment and low-frequency attention filtering fusion will effectively and powerfully alleviate the negative impact of the visible light-thermal infrared modal spatial misalignment, improve the accuracy and richness of the cross-modal crowd complementary feature representation, and thus significantly improve the accuracy of cross-modal crowd counting. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 It is a schematic diagram of the structure of the dual-stage feature alignment fusion counting system of the present invention;

[0050] Figure 2 This is a flow chart of the dual-stage feature alignment fusion counting method of the present invention;

[0051] Figure 3 is a schematic diagram of using the cross-modal counting method of the present invention to count people; DETAILED DESCRIPTION

[0052] The present invention is further described in detail below in conjunction with embodiments:

[0053] like Figure 1 As shown in the figure, it is a structural diagram of a two-stage feature alignment and fusion counting system for cross-modal spatial dislocation. Before the crowd counting system is used, it is necessary to train the constructed crowd counting system using multiple visible light images and thermal infrared images of the same scene, calculate the loss with the real crowd density map as supervision, and optimize the crowd counting system using the Adam algorithm to obtain the optimal parameters of the crowd counting system.

[0054] A two-stage feature alignment fusion counting system for cross-modal spatial dislocation is sequentially connected from input to output with a shared weight initial feature extraction layer, a parallel two-level feature alignment stage, a low-frequency attention filtering fusion stage, and a crowd density map estimation layer, wherein the parallel two-level feature alignment stage includes a spatial feature alignment unit, a semantic feature alignment unit, and a two-level feature fusion unit. The specific structure of each layer is as follows:

[0055] The shared weight initial feature extraction layer includes 2 convolution layers, 1 maximum pooling layer, 2 convolution layers, 1 maximum pooling layer, 4 convolution layers, 1 maximum pooling layer, 4 convolution layers, 1 maximum pooling layer, and 4 convolution layers in sequence. The convolution kernel size of each convolution layer is 3×3, the step size is 1, and the number of channels of the generated feature maps is 64, 64, 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512, and 512 from input to output, and the step size of the maximum pooling layer is 2;

[0056] The parallel two-level feature alignment stage includes a spatial feature alignment unit, a semantic feature alignment unit, and a two-level feature fusion unit:

[0057] The spatial feature alignment unit sequentially includes 1 channel splicing operation, 1 first convolution processing block for downsampling operation, 1 second convolution processing block for downsampling operation, 1 third convolution processing block for downsampling operation, and 1 fourth convolution processing block for generating transformation parameters; the structures of the first convolution processing block, the second convolution processing block, and the third convolution processing block are sequentially including 1 convolution layer with a convolution kernel size of 3×3, an output channel number of 128, and a step size of 2, 1 batch normalization, 1 ReLU activation function, 1 convolution kernel size of 1×1, an output channel number of 128, and a step size of 2, 1 batch normalization, and 1 ReLU activation function; the fourth convolution processing block sequentially includes 1 adaptive average pooling layer with a window size of 1×1, 1 convolution kernel size of 1×1, an output channel number of 6, and a step size of 2, and 1 bilinear interpolation spatial sampling operation;

[0058] The semantic feature alignment unit includes 1 channel splicing operation, 1 convolution layer with a convolution kernel size of 3×3 and 18 output channels, 1 convolution layer with a convolution kernel size of 3×3 and 512 output channels, 1 ReLU activation function, and 1 addition operation.

[0059] The two-level feature fusion unit sequentially includes one channel splicing operation, one convolution layer with a convolution kernel size of 1×1 and an output channel number of 512;

[0060] The low-frequency attention filter fusion stage includes 1 discrete wavelet transform operation, 1 low-frequency component attention unit, 1 multiplication operation, 1 channel splicing operation, and 1 inverse wavelet transform operation. The low-frequency component attention unit includes 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 512, 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 512, and 1 Sigmoid activation function.

[0061] The crowd density map estimation layer sequentially includes 1 bilinear interpolation upsampling layer with a sampling rate of 2, 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 256, 1 ReLU activation function, 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 128, 1 ReLU activation function, 1 convolution layer with a convolution kernel size of 1×1 and an output channel number of 1. The purpose of the last convolution layer with an output channel number of 1 is to generate an estimated crowd density map.

[0062] The cross-modal crowd counting method using the above two-stage feature alignment fusion counting method for cross-modal spatial misalignment is as follows: Figure 2 As shown, the specific steps are as follows:

[0063] Step 1: Input the visible light image and thermal infrared image into the shared weight initial feature extraction layer to extract initial features of different modalities, and obtain the visible light initial feature map and thermal infrared initial feature map respectively;

[0064] Step 2: Input the visible light initial feature map and the thermal infrared initial feature map into the parallel two-level feature alignment stage for spatial-semantic two-level feature alignment to obtain a cross-modal aligned feature map;

[0065] Step 2.1: Input the visible light initial feature map and the thermal infrared initial feature map into the spatial feature alignment unit to obtain a spatial alignment feature map;

[0066] Step 2.1.1: Perform channel splicing on the visible light initial feature map and the thermal infrared initial feature map to obtain a cross-modal feature map;

[0067] Step 2.1.2: Input the cross-modal feature map into the first convolution processing block, and pass through the convolution layer with a convolution kernel size of 3×3, an output channel number of 128, a stride of 2, batch normalization, ReLU activation function, a convolution kernel size of 1×1, an output channel number of 128, a stride of 2, batch normalization, and ReLU activation function of the first convolution processing block in turn to obtain the first spatial alignment feature map;

[0068] Step 2.1.3: Input the first spatial alignment feature map into the second convolution processing block, and sequentially pass through the convolution layer with a convolution kernel size of 3×3, an output channel number of 128, a step size of 2, batch normalization, ReLU activation function, a convolution kernel size of 1×1, an output channel number of 128, a step size of 2, batch normalization, and ReLU activation function of the second convolution processing block to obtain the second spatial alignment feature map;

[0069] Step 2.1.4: Input the second spatial alignment feature map into the third convolution processing block, and sequentially pass through the convolution layer with a convolution kernel size of 3×3, an output channel number of 128, a step size of 2, batch normalization, ReLU activation function, a convolution kernel size of 1×1, an output channel number of 128, a step size of 2, batch normalization, and ReLU activation function of the third convolution processing block to obtain the third spatial alignment feature map;

[0070] Step 2.1.5: Input the third spatial alignment feature map into the fourth convolution processing block, and sequentially pass through the adaptive average pooling layer with a window size of 1×1, a convolution layer with a convolution kernel size of 1×1, an output channel number of 6, a step size of 2, and a bilinear interpolation spatial sampling operation of the fourth convolution processing block to obtain a spatial alignment feature map;

[0071] Step 2.2: Input the visible light initial feature map and the thermal infrared initial feature map into the semantic feature alignment unit to obtain a semantic alignment feature map;

[0072] Step 2.2.1: perform channel splicing on the initial visible light feature map and the initial thermal infrared feature map to obtain a cross-modal feature map;

[0073] Step 2.2.2: Input the cross-modal feature map into a convolutional layer with a convolution kernel size of 3×3 and an output channel number of 18 to obtain an offset feature map;

[0074] Step 2.2.3: Add the offset feature map and the cross-modal feature map and input them into the convolution layer with a convolution kernel size of 3×3 and an output channel number of 512, and the ReLU activation function to obtain a preliminary semantic alignment feature map;

[0075] Step 2.2.4: Add the initial visible light feature map to the preliminary semantic alignment feature map to obtain the semantic alignment feature map;

[0076] Step 2.3: Input the spatial alignment feature map and the semantic alignment feature map into the two-level feature fusion unit to obtain the visible light alignment feature map;

[0077] Step 2.3.1: Concatenate the spatial alignment feature map and the semantic alignment feature map to obtain a preliminary visible light alignment feature map;

[0078] Step 2.3.2: Input the preliminary visible light alignment feature map into a convolution layer with a convolution kernel size of 1×1 and an output channel number of 512 to obtain a visible light alignment feature map;

[0079] Step 2.4: Add the visible light alignment feature map to the thermal infrared initial feature map to obtain a cross-modal alignment feature map;

[0080] Step 3: Input the cross-modal aligned feature map into the low-frequency attention filtering fusion stage for frequency domain feature alignment fusion to obtain a dual-stage aligned fusion feature map;

[0081] Step 3.1: Perform discrete wavelet transform on the cross-modal alignment feature map to obtain low-frequency components, diagonal high-frequency components, vertical high-frequency components, and horizontal high-frequency components;

[0082] Step 3.2: The low-frequency components are sequentially passed through two convolution layers with a convolution kernel size of 3×3 and an output channel number of 512 and a Sigmoid activation function of the low-frequency component attention unit to perform attention calculation to obtain a low-frequency attention map;

[0083] Step 3.3: Multiply the low-frequency component with the low-frequency attention map to obtain the filtered low-frequency component;

[0084] Step 3.4: After channel splicing of the filtered low-frequency component and the diagonal high-frequency component, the vertical high-frequency component, and the horizontal high-frequency component, a discrete inverse wavelet transform is performed to obtain a two-stage aligned fusion feature map;

[0085] Step 4: Input the dual-stage aligned fusion feature map into the crowd density map estimation layer to estimate the crowd density and obtain the estimated crowd density map;

[0086] Step 4.1: Input the two-stage aligned fusion feature map into the crowd density map estimation layer, and then pass through the bilinear interpolation upsampling layer with a sampling rate of 2 to obtain the preliminary crowd feature map;

[0087] Step 4.2: The preliminary crowd feature map is sequentially passed through a convolution layer with a convolution kernel size of 3×3 and an output channel number of 256, a ReLU activation function, a convolution layer with a convolution kernel size of 3×3 and an output channel number of 128, a ReLU activation function, and a convolution layer with a convolution kernel size of 1×1 and an output channel number of 1 to obtain an estimated crowd density map;

[0088] Step 5: Add the estimated crowd density map pixel by pixel to obtain the estimated value of the number of people in the crowd image.

[0089] Figure 3 Schematic diagram of crowd counting using the cross-modal crowd counting method of the present invention. Figure 3When the cross-modal crowd counting method of the present invention is used for counting, the visible light image and thermal infrared image of the scene to be counted are input into the established two-stage feature alignment fusion counting model, and the trained and optimized crowd counting model is used to estimate the crowd density map. The estimated crowd density map is added pixel by pixel to obtain the corresponding crowd quantity estimation result.

[0090] Aiming at the spatial misalignment problem of the existing visible light-thermal infrared cross-modal crowd counting, the present invention designs a two-stage feature alignment fusion counting system and method for cross-modal spatial misalignment, which can effectively solve the problem of decreased counting accuracy caused by spatial misalignment of visible light-thermal infrared cross-modal images. Specifically, a shared weight initial feature extraction network is used to extract the initial crowd features of the two modal input images respectively to obtain complementary scene features; the parallel two-level feature alignment stage performs spatial-semantic two-level feature alignment on the extracted initial features of different modalities; the low-frequency attention filtering and fusion stage performs frequency domain feature decomposition and low-frequency attention filtering on the feature map after the spatial-semantic two-level alignment to obtain a two-stage aligned fusion feature map; the two-stage aligned fusion feature map is passed through the crowd density map estimation layer to generate an estimated crowd density map. Since the shared weight backbone network can effectively extract complementary features of different modalities, the parallel two-level feature alignment stage can realize the two-level joint perception alignment of spatial and semantic features, and the low-frequency attention filtering fusion stage can effectively eliminate the artifacts produced by the feature fusion process, the counting system has the ability to extract robust features for the spatial misalignment problem of cross-modal images, thereby accurately estimating the number of people in the scene.

[0091] The embodiments described above are merely descriptions of preferred implementation modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A two-stage feature alignment fusion counting system for cross-modal spatial dislocation, characterized by: It includes a shared weight initial feature extraction layer, a parallel two-level feature alignment stage, a low-frequency attention filter fusion stage, and a crowd density map estimation layer, wherein the parallel two-level feature alignment stage includes a spatial feature alignment unit, a semantic feature alignment unit, and a two-level feature fusion unit; the specific structure of each layer is as follows: The shared weight initial feature extraction layer includes 2 convolution layers, 1 maximum pooling layer, 2 convolution layers, 1 maximum pooling layer, 4 convolution layers, 1 maximum pooling layer, 4 convolution layers, 1 maximum pooling layer, and 4 convolution layers in sequence. The convolution kernel size of each convolution layer is 3×3, the step size is 1, and the number of channels of the generated feature maps is 64, 64, 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512, and 512 from input to output, and the step size of the maximum pooling layer is 2; The spatial feature alignment unit sequentially includes 1 channel splicing operation, 1 first convolution processing block for downsampling operation, 1 second convolution processing block for downsampling operation, 1 third convolution processing block for downsampling operation, and 1 fourth convolution processing block for generating transformation parameters; the structures of the first convolution processing block, the second convolution processing block, and the third convolution processing block are sequentially including 1 convolution layer with a convolution kernel size of 3×3, an output channel number of 128, and a step size of 2, 1 batch normalization, 1 ReLU activation function, 1 convolution kernel size of 1×1, an output channel number of 128, and a step size of 2, 1 batch normalization, and 1 ReLU activation function; the fourth convolution processing block sequentially includes 1 adaptive average pooling layer with a window size of 1×1, 1 convolution kernel size of 1×1, an output channel number of 6, and a step size of 2, and 1 bilinear interpolation spatial sampling operation; The semantic feature alignment unit includes 1 channel splicing operation, 1 convolution layer with a convolution kernel size of 3×3 and 18 output channels, 1 convolution layer with a convolution kernel size of 3×3 and 512 output channels, 1 ReLU activation function, and 1 addition operation. The two-level feature fusion unit sequentially includes one channel splicing operation, one convolution layer with a convolution kernel size of 1×1 and an output channel number of 512; The low-frequency attention filter fusion stage includes 1 discrete wavelet transform operation, 1 low-frequency component attention unit, 1 multiplication operation, 1 channel splicing operation, and 1 inverse wavelet transform operation. The low-frequency component attention unit includes 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 512, 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 512, and 1 Sigmoid activation function. The crowd density map estimation layer sequentially includes 1 bilinear interpolation upsampling layer with a sampling rate of 2, 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 256, 1 ReLU activation function, 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 128, 1 ReLU activation function, 1 convolution layer with a convolution kernel size of 1×1 and an output channel number of 1. The purpose of the last convolution layer with an output channel number of 1 is to generate an estimated crowd density map.

2. A two-stage feature alignment fusion counting method for cross-modal spatial misalignment, characterized by Applied to the counting system as claimed in claim 1, the steps are as follows: Step 1: Input the visible light image and thermal infrared image into the shared weight initial feature extraction layer to extract initial features of different modalities, and obtain the visible light initial feature map and thermal infrared initial feature map respectively; Step 2: Input the visible light initial feature map and the thermal infrared initial feature map into the parallel two-level feature alignment stage for spatial-semantic two-level feature alignment to obtain a cross-modal alignment feature map; Step 2 is as follows: Step 2.1: Input the visible light initial feature map and the thermal infrared initial feature map into the spatial feature alignment unit to obtain a spatial alignment feature map; Step 2.1 is as follows: Step 2.1.1: Perform channel splicing on the visible light initial feature map and the thermal infrared initial feature map to obtain a cross-modal feature map; Step 2.1.2: Input the cross-modal feature map into the first convolution processing block, and pass through the convolution layer with a convolution kernel size of 3×3, an output channel number of 128, a stride of 2, batch normalization, ReLU activation function, a convolution kernel size of 1×1, an output channel number of 128, a stride of 2, batch normalization, and ReLU activation function of the first convolution processing block in turn to obtain the first spatial alignment feature map; Step 2.1.3: Input the first spatial alignment feature map into the second convolution processing block, and sequentially pass through the convolution layer with a convolution kernel size of 3×3, an output channel number of 128, a step size of 2, batch normalization, ReLU activation function, a convolution kernel size of 1×1, an output channel number of 128, a step size of 2, batch normalization, and ReLU activation function of the second convolution processing block to obtain the second spatial alignment feature map; Step 2.1.4: Input the second spatial alignment feature map into the third convolution processing block, and sequentially pass through the convolution layer with a convolution kernel size of 3×3, an output channel number of 128, a step size of 2, batch normalization, ReLU activation function, a convolution kernel size of 1×1, an output channel number of 128, a step size of 2, batch normalization, and ReLU activation function of the third convolution processing block to obtain the third spatial alignment feature map; Step 2.1.5: Input the third spatial alignment feature map into the fourth convolution processing block, and sequentially pass through the adaptive average pooling layer with a window size of 1×1, a convolution layer with a convolution kernel size of 1×1, an output channel number of 6, a step size of 2, and a bilinear interpolation spatial sampling operation of the fourth convolution processing block to obtain a spatial alignment feature map; Step 2.2: Input the visible light initial feature map and the thermal infrared initial feature map into the semantic feature alignment unit to obtain a semantic alignment feature map; Step 2.2 is as follows: Step 2.2.1: perform channel splicing on the initial visible light feature map and the initial thermal infrared feature map to obtain a cross-modal feature map; Step 2.2.2: Input the cross-modal feature map into a convolutional layer with a convolution kernel size of 3×3 and an output channel number of 18 to obtain an offset feature map; Step 2.2.3: Add the offset feature map and the cross-modal feature map and input them into the convolution layer with a convolution kernel size of 3×3 and an output channel number of 512, and the ReLU activation function to obtain a preliminary semantic alignment feature map; Step 2.2.4: Add the initial visible light feature map to the preliminary semantic alignment feature map to obtain the semantic alignment feature map; Step 2.3: Input the spatial alignment feature map and the semantic alignment feature map into the two-level feature fusion unit to obtain the visible light alignment feature map; Step 2.4: Add the visible light alignment feature map to the thermal infrared initial feature map to obtain a cross-modal alignment feature map; Step 3: Input the cross-modal aligned feature map into the low-frequency attention filter fusion stage for frequency domain feature alignment fusion to obtain a dual-stage aligned fusion feature map; Step 4: Input the dual-stage aligned fusion feature map into the crowd density map estimation layer to estimate the crowd density and obtain the estimated crowd density map; Step 5: Add the estimated crowd density map pixel by pixel to obtain the estimated value of the number of people in the crowd image.

3. According to claim 2, a two-stage feature alignment fusion counting method for cross-modal spatial dislocation is characterized by: Step 2.3 is as follows: Step 2.3.1: Concatenate the spatial alignment feature map and the semantic alignment feature map to obtain a preliminary visible light alignment feature map; Step 2.3.2: Input the preliminary visible light alignment feature map into a convolution layer with a convolution kernel size of 1×1 and an output channel number of 512 to obtain a visible light alignment feature map.

4. According to claim 2, a two-stage feature alignment fusion counting method for cross-modal spatial dislocation is characterized by: Step 3 is as follows: Step 3.1: Perform discrete wavelet transform on the cross-modal alignment feature map to obtain low-frequency components, diagonal high-frequency components, vertical high-frequency components, and horizontal high-frequency components; Step 3.2: The low-frequency components are sequentially passed through two convolution layers with a convolution kernel size of 3×3 and an output channel number of 512 and a Sigmoid activation function of the low-frequency component attention unit to perform attention calculation to obtain a low-frequency attention map; Step 3.3: Multiply the low-frequency component with the low-frequency attention map to obtain the filtered low-frequency component; Step 3.4: After channel splicing of the filtered low-frequency components and the diagonal high-frequency components, vertical high-frequency components, and horizontal high-frequency components, a discrete inverse wavelet transform is performed to obtain a two-stage aligned fusion feature map.

5. According to claim 2, a two-stage feature alignment fusion counting method for cross-modal spatial dislocation is characterized by: Step 4 is as follows: Step 4.1: Input the two-stage aligned fusion feature map into the crowd density map estimation layer, and then pass through the bilinear interpolation upsampling layer with a sampling rate of 2 to obtain the preliminary crowd feature map; Step 4.2: The preliminary crowd feature map is sequentially passed through a convolution layer with a convolution kernel size of 3×3 and an output channel number of 256, a ReLU activation function, a convolution layer with a convolution kernel size of 3×3 and an output channel number of 128, a ReLU activation function, and a convolution layer with a convolution kernel size of 1×1 and an output channel number of 1 to obtain an estimated crowd density map.

Images