Foggy day pedestrian image enhancement method based on CNN multi-scale feature fusion

Abstract

The invention discloses a foggy-day pedestrian image enhancement method based on CNN multi-scale feature fusion, and the method comprises the steps: constructing a foggy-day pedestrian image enhancement model which is of a U-shaped network structure and comprises an encoder module, a feature fusion module and a decoder module; in the encoder module and the decoder module, lightweight FastCNN modules are adopted, so that the parameter quantity and the calculation quantity of the model can be effectively reduced; the feature fusion module comprises a CSP-ADD module, and the CSP-ADD module enhances the feature extraction capability by stacking a plurality of residual blocks, effectively transmits gradients in a deep neural network, keeps detail information of input features, and improves the focusing capability of the network on important information, thereby further enhancing the effects of tasks such as defogging and denoising. The model constructed by the method has the advantage of light weight, and meanwhile, a relatively good image restoration effect can be ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and relates to a method for enhancing pedestrian images in foggy weather based on CNN multi-scale feature fusion. Background Art

[0002] Image restoration is one of the most basic visual tasks, and its purpose is to restore a clean image from a degraded image. In pedestrian detection scenarios in bad weather such as foggy days, unclear images will directly affect the accuracy of pedestrian detection. Therefore, it is necessary to defog and enhance pedestrian images to provide clear images for pedestrian detection.

[0003] Traditional image enhancement methods can be roughly divided into three categories: spatial domain methods, which directly process pixel values, such as histogram equalization and gamma transform; frequency domain methods, which operate in a certain transform domain, such as wavelet transform; and hybrid domain methods, which are a combination of spatial domain methods and frequency domain methods. These methods are relatively simple and fast, but they do not consider the contextual information in the image, so the effect is not good.

[0004] In recent years, convolutional neural networks (CNNs) have made great breakthroughs in low-level computer vision tasks such as image super-resolution, deblurring, defogging, and denoising. Through supervised learning, the mapping relationship between the original image and the target image is learned to obtain the enhanced image. Compared with traditional methods, the image enhancement quality is greatly improved.

[0005] With the development of deep learning, a variety of CNN-based frameworks have been proposed based on clever modules or borrowed units, such as encoder-decoder architecture, dilated convolution, dense connection, attention mechanism, etc. These models provide long-range pixel interaction and adaptive capabilities with respect to input features, and have significant image restoration performance, but they also have problems with large number of parameters and computation, and there is still room for further improvement in image restoration performance. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to provide a method for enhancing pedestrian images in foggy weather based on CNN multi-scale feature fusion, which has good image restoration effect, small number of parameters and calculation amount.

[0007] Technical solution: The method for enhancing pedestrian images in foggy weather based on CNN multi-scale feature fusion according to the present invention comprises:

[0008] (1) Construct a pedestrian image enhancement model in foggy weather;

[0009] The foggy pedestrian image enhancement model is a U-shaped network structure, including an encoder module, a feature fusion module and a decoder module;

[0010] The encoder module includes a first Conv module, a second Conv module, a third Conv module, a first FastCNN module, a second FastCNN module, a first Conv-S module and a second Conv-S module; the feature fusion module includes a CSP-ADD module; the decoder module includes a fourth Conv module, a fifth Conv module, a sixth Conv module, a seventh Conv module, an eighth Conv module, a third FastCNN module and a fourth FastCNN module;

[0011] The original image is used as the input of the first Conv module, the first Conv-S module and the second Conv-S module respectively. The output of the first Conv module is used as the input of the first FastCNN module. One output of the first FastCNN module is processed by wavelet downsampling and then spliced ​​with the output of the first Conv-S module, and then used as the input of the second Conv module; the output of the second Conv module is used as the input of the second FastCNN module, one output of the second FastCNN module is processed by wavelet downsampling and then spliced ​​with the output of the second Conv-S module, and then used as the input of the third Conv module; the output of the third Conv module is used as the input of the CSP-ADD module;

[0012] One output of the CSP-ADD module is used as the input of the fourth Conv module, and the output of the fourth Conv module is the first picture enhancement output; another output of the CSP-ADD module is spliced ​​with another output of the second FastCNN module after upsampling processing, and then used as the input of the fifth Conv module; the output of the fifth Conv module is used as the input of the third FastCNN module, one output of the third FastCNN module is used as the input of the sixth Conv module, and the output of the sixth Conv module is the second picture enhancement output; another output of the third FastCNN module is spliced ​​with another output of the first FastCNN module after upsampling processing, and then used as the input of the seventh Conv module; the output of the seventh Conv module is used as the input of the fourth FastCNN module, the output of the fourth FastCNN module is used as the input of the eighth Conv module, and the output of the eighth Conv module is added to the original image to obtain the final enhanced picture;

[0013] (2) Train a pedestrian image enhancement model for foggy days, and use the trained pedestrian image enhancement model to enhance pedestrian images in foggy days.

[0014] Furthermore, the FastCNN module includes a lightweight Pconv, four Convs and an MSM attention mechanism module. The input passes through Pconv, Conv1, and Conv2 in sequence. The convolution output is added to the initial input. After this operation is repeated n times, the output is passed to the convolution block Conv3, and then passes through the MSM attention mechanism module to emphasize the information frequency component by re-weighing the weight of the high-pass filter; then the result is input into the convolution block Conv4, and finally the result is output after passing through a feedforward network.

[0015] Furthermore, in the CSP-ADD module, the input tensor first passes through two convolution blocks Conv1 and Conv2, and then passes through a feedforward network. After this operation is repeated n times, the output is passed to the convolution block Conv3, and then passes through the MSM attention mechanism module, and then the result is input into the convolution block Conv4, and then the result is input into the CGA attention mechanism module; the output of the CGA attention mechanism module is multiplied by the output obtained by the n-times cycle operation, and finally passes through a feedforward network to output the result of feature fusion.

[0016] Furthermore, the MSM attention mechanism module uses different average pooling to convert the input tensor into different feature spaces; the input tensor is connected using three average pooling layer branches, and the downsampling rate is set to {8, 4, 2}; in each branch, the features generated after the MSA module are merged into the next branch through the addition operator; then, the output tensors of the three branches are upsampled by 2, 4, and 8 times respectively, so that the size of the tensors is unified to the original input size and added; finally, the output result is output after Conv convolution;

[0017] For the i-th branch, the output feature X formula is:

[0018]

[0019] Where i∈{1, 2, 3}, X0=0; Indicates downsampling to 2 4-i The average pooling layer of ;↑2 represents the interpolation with bilinear upsampling rate of 2;

[0020] The whole process of MSM is formally expressed as:

[0021]

[0022] Among them, Conv 3x3 Represents a convolution with a 3×3 kernel.

[0023] Furthermore, the MSA module is composed of a dilated square attention module DSA and a dilated rectangular attention module DRA in parallel.

[0024] Furthermore, in the DSA module, the input tensor first passes through a Conv, and then the output is input into the global average pooling layer to generate attention weights that are adaptive to the input features, and then aggregated through the convolution Conv operation; after that, the hyperbolic tangent function is used to avoid the limitation of the low-pass filter and enhance the role of the high-pass filter in the attention weight; then, the result is decomposed into low / high-pass filters, and then the high-pass filter is re-evaluated using the rectangular FM; the output result is multiplied and added with the initial input tensor and the result is output; finally, the attention weight integrates information on the dilated pixels of the feature, improving the performance of multi-scale learning;

[0025] Attention weight generation process is expressed as:

[0026] A DSA =Tanh(Conv(GAP(Conv(X))))(3)

[0027] Among them, A DSA represents the attention weight in the DSA module, Tanh represents the hyperbolic tangent function, and GAP represents the global average pooling;

[0028] Low-pass component in DSA module Extracted from the DSA module via:

[0029]

[0030] in, G represents the number of groups, K×K represents the absolute area size of the integration, It means a ring or a particular space;

[0031] Then, the high-pass component can be seen as the complementary part of the low-pass component:

[0032]

[0033] Furthermore, in the DRA module, the input tensor first passes through the global average pooling layer to generate attention weights that are adaptive to the input features, and then is aggregated through the convolution Conv operation; after that, the hyperbolic tangent function is used to avoid the limitation of the low-pass filter and enhance the role of the high-pass filter in the attention weight; subsequently, the result is decomposed into low / high-pass filters, and then the high-pass filter is re-evaluated using the rectangular FM; the output result is multiplied and added with the starting input tensor and the result is output; the input tensor is operated twice, respectively, in the horizontal and vertical directions of the tensor, and finally the processed result is output;

[0034] The output characteristic expression of each group of horizontal units is:

[0035]

[0036]

[0037] in, represents the input feature, C represents the number of channels, H and W represent the height and width of the input feature respectively; g, h, w represent the group, height, and width; d is the expansion rate; K represents the width of the absolute area; G represents the number of groups, Indicates a ring or a specific space; A H represents the attention map of the horizontal unit in the DRA module; RFM refers to the attention weights of applying FM to the rectangle; GAP represents global average pooling, and Tanh represents the hyperbolic tangent function.

[0038] Furthermore, the CGA attention mechanism module includes spatial attention, channel attention and channel shuffling modules; for the input tensor, there are three branches; the first branch performs a global maximum pooling layer operation on the tensor, the second branch performs a global average pooling layer operation on the tensor, and then the output results of the first and second branches are spliced, and finally the spliced ​​result is convolved through Conv1; the third branch inputs the tensor into the global average pooling layer, and then outputs the result after convolution operations through Conv2 and Conv3; then the tensor results output by Conv1 and Conv3 are added, and the result after addition is spliced ​​with the initial input tensor; the spliced ​​result passes through the channel confusion module and Conv4, and then passes through the Sigmoid activation function to output the result.

[0039] Furthermore, the Conv-S module is composed of 4 stacked Conv convolutions.

[0040] Furthermore, the NH-HAZE dataset is used for model training.

[0041] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0042] In the encoder module, the FastCNN module reduces the number of model parameters and computational complexity, making the network more flexible in expressing complex features, and easier to expand and adjust. It can effectively improve the network's feature extraction capabilities, especially when processing images with complex spatial structures. It is very suitable for complex image processing tasks such as image defogging and denoising. The Conv-S module is used to extract shallow features from low-resolution degraded images. In addition, wavelet downsampling is used for downsampling, which can reduce information uncertainty and improve the accuracy of target detection. HWD uses wavelet transform to maintain feature details, which can better preserve spatial information compared to traditional downsampling methods.

[0043] The feature fusion module uses the CSP-ADD module, which enhances the feature extraction capability by stacking multiple residual blocks and effectively transfers gradients in the deep neural network while maintaining the detailed information of the input features. By stacking multiple residual blocks, complex image features can be extracted layer by layer. Each residual block retains the key information of the input through convolution operations and residual connections, while extracting higher-level features. The attention mechanism contained therein can adaptively focus on the important parts of the input feature map, improve the network's ability to focus on important information, and further enhance the effects of tasks such as dehazing and denoising. The CSP-ADD module can effectively improve the dehazing effect.

[0044] The decoder module processes features through the FastCNN module, and gradually restores the image details, and gradually decodes high-level features into low-level features, that is, recovers information that is gradually close to the input image from the high-level feature map of the network. This is particularly important for image reconstruction or image enhancement tasks (such as defogging, denoising, etc.), because it requires remapping abstract high-level features back to the pixel space of the original image. The FastCNN module can not only decode features, but also maintain and enhance detail information during the decoding process. Combining spatial mixing and multi-layer perceptrons, it can effectively process and optimize input features to ensure that no details are lost during feature decoding. Using filtering operations, this mechanism helps the network automatically select important features and remove irrelevant or redundant information, further improving the decoding effect and the visual quality of the image. The decoder module finally outputs the enhanced image.

[0045] In summary, the lightweight encoder module provided by the present invention can efficiently extract the spatial and channel information of the image and encode it into a higher-dimensional feature representation. Each residual block retains the key information of the input and extracts useful high-level features. The lightweight decoder module provided can gradually restore the detailed information of the image, and gradually decode the high-level features into low-level features, that is, recover information that is gradually close to the input image from the high-level feature map of the network. The lightweight encoder module and decoder module can effectively reduce the number of parameters and the amount of calculation of the model. In addition, the CSP-ADD module is used as a feature fusion module to enhance the ability of feature extraction by stacking multiple residual blocks, and effectively transfer gradients in the deep neural network while maintaining the detailed information of the input features. The attention mechanism contained therein can adaptively focus on the important parts of the input feature map, enhance the network's ability to focus on important information, and further enhance the effects of tasks such as defogging and denoising. Through this series of operations, the lightweight of the overall model and the maintenance of good image restoration quality are achieved.

[0046] In addition to being used for image defogging, the present invention is also applicable to image deraining, image desnowing, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a network structure diagram of a pedestrian image enhancement model in foggy weather according to an embodiment of the present invention;

[0048] Figure 2 : is a network structure diagram of the FastCNN module in an embodiment of the present invention;

[0049] Figure 3 is a network structure diagram of the Conv-S module in an embodiment of the present invention;

[0050] Figure 4 is a network structure diagram of a CSP-ADD module in an embodiment of the present invention;

[0051] Figure 5 : is a network structure diagram of the MSM attention mechanism module in an embodiment of the present invention;

[0052] Figure 6 is a network structure diagram of a DSA module in an embodiment of the present invention;

[0053] Figure 7 is a network structure diagram of a DRA module in an embodiment of the present invention;

[0054] Figure 8 is a network structure diagram of a CGA attention mechanism module in an embodiment of the present invention;

[0055] Fig. 9 This is a diagram showing the enhanced effect of pedestrian images in foggy weather according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention will be further described below in conjunction with the accompanying drawings.

[0057] The embodiment of the present invention provides a method for enhancing pedestrian images in foggy weather based on CNN multi-scale feature fusion, which specifically includes the following steps:

[0058] (1) Construct a pedestrian image enhancement model in foggy weather;

[0059] The pedestrian image enhancement model in foggy weather is a U-shaped network structure, including an encoder module, a feature fusion module and a decoder module. The encoder module is used to efficiently extract the spatial and channel information of the image; the feature fusion module is used to effectively fuse the features extracted by the encoder module; finally, the decoder module gradually restores the detailed information of the image and gradually decodes the high-level features into low-level features, that is, recovers information that is gradually close to the input image from the high-level feature map of the network.

[0060] like Figure 1As shown, the encoder module includes a first Conv module, a second Conv module, a third Conv module, a first FastCNN module, a second FastCNN module, a first Conv-S module and a second Conv-S module; the feature fusion module includes a CSP-ADD module; and the decoder module includes a fourth Conv module, a fifth Conv module, a sixth Conv module, a seventh Conv module, an eighth Conv module, a third FastCNN module and a fourth FastCNN module.

[0061] The original image is used as the input of the first Conv module, the first Conv-S module and the second Conv-S module respectively; the first Conv module performs feature extraction, and the output of the first Conv module is used as the input of the first FastCNN module. One output of the first FastCNN module is processed by wavelet downsampling (HWD) and spliced ​​with the output of the first Conv-S module (extracted feature map), and then used as the input of the second Conv module; the output of the second Conv module is used as the input of the second FastCNN module, the second FastCNN module performs encoding processing, and one output of the second FastCNN module is processed by wavelet downsampling (HWD) and spliced ​​with the output of the second Conv-S module (extracted feature map), and then used as the input of the third Conv module; the output of the third Conv module is used as the input of the CSP-ADD module.

[0062] One output of the CSP-ADD module is used as the input of the fourth Conv module, and the output of the fourth Conv module is the first image enhancement output; another output of the CSP-ADD module is spliced ​​with another output of the second FastCNN module after upsampling (Upsample) processing, and then used as the input of the fifth Conv module; the output of the fifth Conv module is used as the input of the third FastCNN module, the third FastCNN module performs decoding processing, and one output of the third FastCNN module is used as the input of the sixth Conv module, and the output of the sixth Conv module is the second image enhancement output; another output of the third FastCNN module is spliced ​​with another output of the first FastCNN module after upsampling (Upsample) processing, and then used as the input of the seventh Conv module; the output of the seventh Conv module is used as the input of the fourth FastCNN module, the fourth FastCNN module performs further decoding processing, and the tensor output by the fourth FastCNN module is used as the input of the eighth Conv module, and the output of the eighth Conv module is added to the original image to obtain the final enhanced image.

[0063] like Figure 2As shown in the figure, the FastCNN module includes a lightweight Pconv, four Convs and an MSM attention mechanism module. The input passes through Pconv, Conv1, and Conv2 in sequence. The convolution output result is added to the initial input. After this operation is repeated n times, the output is passed to the convolution block Conv3, and then passes through the MSM attention mechanism module to emphasize the information frequency component by re-weighing the weight of the high-pass filter; then the result is input into the convolution block Conv4, and finally the result is output after passing through a feedforward network.

[0064] like Figure 5 As shown, for the input tensor, the channel dimension is ignored for clarity, and the MSM attention mechanism module uses different average pooling (AP) to transform the input tensor into different feature spaces.

[0065] The input tensor is connected by three average pooling (AP) branches, and the downsampling rate is set to {8, 4, 2}. In each branch, the features generated after the MSA module are merged into the next branch through the addition operator. This operation can gradually eliminate degradation by imitating a multi-stage network. Then, the output tensors of the three branches are upsampled by 2, 4, and 8 times respectively, so that the size of the tensors is unified to the original input size and added. Finally, the output result is output after Conv convolution.

[0066] For the i-th branch, the output feature X formula is:

[0067]

[0068] Where i∈{1, 2, 3}, X0=0; Indicates downsampling to 2 4-i The average pooling layer of ;↑2 represents the interpolation with bilinear upsampling rate of 2;

[0069] The whole process of MSM can be formally expressed as:

[0070]

[0071] Among them, Conv 3x3 Represents a convolution with a 3×3 kernel.

[0072] like Figure 5 As shown in Figure 1, the MSA module consists of a Dilated Square Attention (DSA) module and a Dilated Rectangular Attention (DRA) module in parallel. The MSA module combines the advantages of the self-attention operator and the convolution operator, inheriting the content-aware properties of the former while maintaining the efficiency properties of the latter. In addition, MSA contains operators of different shapes and expansion mechanisms of different rates to enhance multi-shape and multi-scale representation learning and expand the receptive field.

[0073] like Figure 6 As shown in the figure, in the structure of the extended square attention (DSA) module, the input tensor first passes through a Conv, and then the output is input into the global average pooling layer (GAP) to generate attention weights that are adaptive to the input features, and then aggregated through the convolution Conv operation. After that, the hyperbolic tangent function (Tanh) is used to avoid the limitation of the low-pass filter and enhance the role of the high-pass filter in the attention weight. Subsequently, the result is decomposed into low / high-pass filters, and then the high-pass filter is re-evaluated using the rectangular FM. The output result is multiplied and added with the initial input tensor and the result is output. Therefore, the reorganized filter has adaptive ability and can highlight useful frequencies. Finally, the attention weight integrates information on the dilated pixels of the feature, improving the performance of multi-scale learning.

[0074] The generation process of attention weights can be expressed as:

[0075] A DSA =Tanh(Conv(GAP(Conv(X)))) (3)

[0076] Among them, A DSA represents the attention weight in the DSA module, Tanh represents the hyperbolic tangent function, and GAP represents the global average pooling;

[0077] Low-pass component in DSA module Extracted from the DSA module via:

[0078]

[0079] in, G represents the number of groups, K×K represents the absolute area size of the integration, Indicates a ring or a specific space;

[0080] Then, the high-pass component can be seen as the complementary part of the low-pass component:

[0081]

[0082] like Figure 7As shown in the figure, the structure of the extended rectangular attention (DRA) module is similar to that of the DSA module. The input tensor first passes through the global average pooling layer (GAP) to generate attention weights that are adaptive to the input features, and then aggregated through the convolution Conv operation. After that, the hyperbolic tangent function (Tanh) is used to avoid the limitations of the low-pass filter and enhance the role of the high-pass filter in the attention weight. Subsequently, the result is decomposed into low / high-pass filters, and then the high-pass filter is re-evaluated using the rectangular FM. The output result is multiplied and added with the starting input tensor and the result is output. The input tensor is operated twice, the tensor is operated horizontally and vertically respectively, and finally the processed result is output.

[0083] The output characteristic expression of each group of horizontal units is:

[0084]

[0085] in, represents the input feature, C represents the number of channels, H and W represent the height and width of the input feature respectively; g, h, w represent the group, height, and width; d is the expansion rate; K represents the width of the absolute area; G represents the number of groups, Indicates a ring or a specific space; A H represents the attention map of the horizontal unit in the DRA module; RFM refers to the attention weights of applying FM to the rectangle; GAP represents global average pooling, and Tanh represents the hyperbolic tangent function.

[0086] like Figure 3 As shown in the figure, the Conv-S module is composed of 4 stacked Conv convolutions, which can effectively extract key feature information.

[0087] The structure of the CSP-ADD module is as follows Figure 4 As shown in the figure, the input tensor first passes through two convolution blocks Conv1 and Conv2, and then passes through a feedforward network. After this operation is repeated n times, the output is passed to the convolution block Conv3, and then passes through the MSM attention mechanism module, and then the result is input into the convolution block Conv4, and then the result is input into the CGA attention mechanism module; the output of the CGA attention mechanism module is multiplied by the output obtained by the n-times cycle operation, and finally passes through a feedforward network to output the result of feature fusion.

[0088] The CGA attention mechanism module is formed by combining multiple attention mechanisms. It uses a coarse-to-fine approach to obtain an exclusive spatial importance map for each input feature channel, while fully mixing channel attention weights and spatial attention weights to ensure information interaction. CGA assigns a unique spatial importance map to each channel, guiding the model to focus on the important areas of each channel. Therefore, more useful information encoded in the features can be emphasized, effectively improving the defogging performance.

[0089] The CGA attention mechanism module structure is as follows Figure 8 As shown in the figure, it is mainly composed of spatial attention, channel attention and channel shuffle modules. For the input tensor, there are three branches. The first branch performs a global maximum pooling layer (GMP) operation on the tensor, and the second branch performs a global average pooling layer (GAP) operation on the tensor. Then the output results of the first and second branches are spliced ​​(C), and finally the spliced ​​result is convolved through Conv1; the third branch inputs the tensor into the global average pooling layer (GAP), and then outputs the result after convolution operations of Conv2 and Conv3. Then, the tensor results output by Conv1 and Conv3 are added, and the result after addition is spliced ​​with the tensor input at the beginning (C). The spliced ​​result passes through the channel confusion module (Channelshuffel) and Conv4, and then passes through the Sigmoid (S) activation function to output the result.

[0090] (2) Train a pedestrian image enhancement model for foggy days, and use the trained pedestrian image enhancement model to enhance pedestrian images in foggy days.

[0091] This embodiment uses the NH-HAZE dataset to train the foggy pedestrian image enhancement model. The training uses a GPU, sets the improved network model parameters, and sets the number of training rounds to 6000 rounds, the initial learning rate to 0.01, the minimum learning rate to 0.0001, the weight decay coefficient to 0.00005, and the batch-size to 4. During the model training process, the model parameters with the best accuracy will be saved and named best.pkl. Table 1 shows the results of testing the test set using the trained best.pkl, as well as the comparison with the ConvIR model.

[0092] Table 1 Experimental results

[0093]

[0094] Fig. 9 This is the defogging effect diagram of the algorithm of the present invention, and it can be seen that the defogging effect is good.

[0095] In summary, the model of the present invention is small in size, has small number of parameters and calculation, has strong real-time performance, and has good defogging effect. It can be deployed to enhance pedestrian images in adverse weather such as foggy days, thereby further meeting the accuracy requirements of pedestrian detection.

Claims

1. A method for enhancing pedestrian images in foggy weather based on CNN multi-scale feature fusion, characterized in that: include: (1) Construct a pedestrian image enhancement model in foggy weather; The foggy pedestrian image enhancement model is a U-shaped network structure, including an encoder module, a feature fusion module and a decoder module; The encoder module includes a first Conv module, a second Conv module, a third Conv module, a first FastCNN module, a second FastCNN module, a first Conv-S module and a second Conv-S module; the feature fusion module includes a CSP-ADD module; the decoder module includes a fourth Conv module, a fifth Conv module, a sixth Conv module, a seventh Conv module, an eighth Conv module, a third FastCNN module and a fourth FastCNN module; The original image is used as the input of the first Conv module, the first Conv-S module and the second Conv-S module respectively. The output of the first Conv module is used as the input of the first FastCNN module. One output of the first FastCNN module is processed by wavelet downsampling and concatenated with the output of the first Conv-S module, and then used as the input of the second Conv module. The output of the second Conv module is used as the input of the second FastCNN module. One output of the second FastCNN module is processed by wavelet downsampling and concatenated with the output of the second Conv-S module, and then used as the input of the third Conv module. The output of the third Conv module is used as the input of the CSP-ADD module. An output of the CSP-ADD module is used as the input of the fourth Conv module, and the output of the fourth Conv module is the first image enhancement output; another output of the CSP-ADD module is spliced ​​with another output of the second FastCNN module after upsampling, and then used as the input of the fifth Conv module; the output of the fifth Conv module is used as the input of the third FastCNN module, and an output of the third FastCNN module is used as the input of the sixth Conv module, and the output of the sixth Conv module is the second image enhancement output; another output of the third FastCNN module is spliced ​​with another output of the first FastCNN module after upsampling, and then used as the input of the seventh Conv module; The output of the seventh Conv module is used as the input of the fourth FastCNN module, the output of the fourth FastCNN module is used as the input of the eighth Conv module, and the output of the eighth Conv module is added to the original image to obtain the final enhanced image; (2) Train a pedestrian image enhancement model for foggy days, and use the trained pedestrian image enhancement model to enhance pedestrian images in foggy days.

2. The method for enhancing pedestrian images in foggy weather based on CNN multi-scale feature fusion according to claim 1, characterized in that: The FastCNN module includes a lightweight Pconv, four Convs and an MSM attention mechanism module. The input passes through Pconv, Conv1, and Conv2 in sequence. The convolution output is added to the initial input. After this operation is repeated n times, the output is passed to the convolution block Conv3, and then passes through the MSM attention mechanism module to emphasize the information frequency component by re-weighing the weight of the high-pass filter; then the result is input into the convolution block Conv4, and finally the result is output after passing through a feedforward network.

3. The method for enhancing pedestrian images in foggy weather based on CNN multi-scale feature fusion according to claim 2 is characterized in that: In the CSP-ADD module, the input tensor first passes through two convolution blocks Conv1 and Conv2, and then passes through a feedforward network. After this operation is repeated n times, the output is passed to the convolution block Conv3, and then passes through the MSM attention mechanism module. The result is then input into the convolution block Conv4, and then the result is input into the CGA attention mechanism module. The output of the CGA attention mechanism module is multiplied by the output obtained by performing n cycles, and finally the result of feature fusion is output after passing through a feedforward network.

4. The method for enhancing pedestrian images in foggy weather based on CNN multi-scale feature fusion according to claim 3 is characterized in that: The MSM attention mechanism module uses different average pooling to convert the input tensor into different feature spaces; the input tensor is connected using three average pooling layer branches, and the downsampling rate is set to {8, 4, 2}; in each branch, the features generated after the MSA module are merged into the next branch through the addition operator; then, the output tensors of the three branches are upsampled by 2, 4, and 8 times respectively, so that the size of the tensors is unified to the original input size and added; finally, the output result is output after Conv convolution; For the i-th branch, the output feature X formula is: Where i∈{1,2,3}, X0=0; Indicates downsampling to 2 4-i The average pooling layer of ;↑2 represents the interpolation with bilinear upsampling rate of 2; The whole process of MSM is formally expressed as: Among them, Conv 3x3 Represents a convolution with a 3×3 kernel.

5. The method for enhancing pedestrian images in foggy weather based on CNN multi-scale feature fusion according to claim 4 is characterized in that: The MSA module consists of a extended square attention module DSA and an extended rectangular attention module DRA in parallel.

6. The method for enhancing pedestrian images in foggy weather based on CNN multi-scale feature fusion according to claim 5, characterized in that: In the DSA module, the input tensor first passes through a Conv, and then the output is input into the global average pooling layer to generate attention weights that are adaptive to the input features, and then aggregated through the convolution Conv operation; after that, the hyperbolic tangent function is used to avoid the limitation of the low-pass filter and enhance the role of the high-pass filter in the attention weight; then, the result is decomposed into low / high-pass filters, and then the high-pass filter is re-evaluated using the rectangular FM; the output result is multiplied and added with the initial input tensor and the result is output; finally, the attention weight integrates information on the dilated pixels of the feature, improving the performance of multi-scale learning; Attention weight generation process is expressed as: A DSA =Tanh(Conv(GAP(Conv(X)))) (3) Among them, A DSA represents the attention weight in the DSA module, Tanh represents the hyperbolic tangent function, and GAP represents the global average pooling; Low-pass component in DSA module Extracted from the DSA module via: in, G represents the number of groups, K×K represents the absolute area size of the integration, It means a ring or a particular space; Then, the high-pass component can be seen as the complementary part of the low-pass component:

7. The method for enhancing pedestrian images in foggy weather based on CNN multi-scale feature fusion according to claim 5, characterized in that: In the DRA module, the input tensor first passes through the global average pooling layer to generate attention weights that are adaptive to the input features, and then is aggregated through the convolution Conv operation; after that, the hyperbolic tangent function is used to avoid the limitations of the low-pass filter and enhance the role of the high-pass filter in the attention weight; then, the result is decomposed into low / high-pass filters, and then the high-pass filter is re-evaluated using the rectangular FM; the output result is multiplied and added with the starting input tensor and the result is output; the input tensor is operated twice, respectively, in the horizontal and vertical directions of the tensor, and finally the processed result is output; The output characteristic expression of each group of horizontal units is: in, represents the input feature, C represents the number of channels, H and W represent the height and width of the input feature respectively; g, h, w represent the group, height, and width; d is the expansion rate; K represents the width of the absolute area; G represents the number of groups, Indicates a ring or a specific space; A H represents the attention map of the horizontal unit in the DRA module; RFM refers to the attention weights of applying FM to the rectangle; GAP represents global average pooling, and Tanh represents the hyperbolic tangent function.

8. The method for enhancing pedestrian images in foggy weather based on CNN multi-scale feature fusion according to claim 3, characterized in that: The CGA attention mechanism module includes spatial attention, channel attention and channel shuffle modules; for the input tensor, there are three branches; the first branch performs a global maximum pooling layer operation on the tensor, and the second branch performs a global average pooling layer operation on the tensor, and then the output results of the first and second branches are spliced, and finally the spliced ​​result is convolved through Conv1; the third branch inputs the tensor into the global average pooling layer, and then outputs the result after convolution operations through Conv2 and Conv3; then the tensor results output by Conv1 and Conv3 are added, and the result after addition is spliced ​​with the initial input tensor; the spliced ​​result passes through the channel confusion module and Conv4, and then passes through the Sigmoid activation function to output the result.

9. The method for enhancing pedestrian images in foggy weather based on CNN multi-scale feature fusion according to claim 1, characterized in that: The Conv-S module is composed of 4 stacked Conv convolutions.

10. The method for enhancing pedestrian images in foggy weather based on CNN multi-scale feature fusion according to claim 1, characterized in that: The model is trained using the NH-HAZE dataset.

Images