A method and device for detecting a severe weather target based on feature fusion

Abstract

The application discloses a kind of based on feature fusion's severe weather target detection method and device, comprising: the target detection model is as detection branch, and the shallow feature map of real severe weather image is extracted, and the detection loss of detection branch is constructed;Increase severe weather processing model as severe weather processing enhancement branch, and the real severe weather image is removed severe weather interference factor processing and obtains image after processing;Shallow feature map and image after processing are respectively through mixed random pooling and obtain the feature after pooling, the feature of two features after pooling is fused to obtain fusion feature;The similarity loss of the feature after pooling corresponding to fusion feature and shallow feature map is calculated, and the parameter optimization of target detection model is carried out according to similarity loss and detection loss;The target detection model after parameter optimization is used to detect the severe weather image to be detected, so as to improve the precision of target detection in severe weather scene with lower calculation consumption.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of target detection and computer vision, and particularly relates to a severe weather target detection method and device based on feature fusion. BACKGROUND

[0002] In the field of deep learning, target detection is a research hotspot in the field, and target detection in extreme severe weather such as fog, sand, dust, and low light has always been a difficulty. In the foggy scene, the commonly used method at present is mainly a cascaded dehazing network and a detection network, which restores a clean image through the dehazing network, and uses the clean image as the input of the detection network, for example, the following two pieces of published technical solutions:

[0003] The patent application with publication number CN118154886A discloses a method for infrared image denoising and small target detection in severe weather. First, a three-class infrared image small target data set is constructed, and the data set is randomly divided into three equal parts and added with three kinds of noise, namely Gaussian noise, Poisson noise and salt and pepper noise. Then, a network architecture for infrared image denoising and small target detection in severe weather conditions is constructed, and the overall framework includes a filter denoising module, a full-resolution denoising module and a YOLOv5 detector. A loss supervision function is used in the detector to improve the detection performance of small targets. Finally, based on the three-class infrared image small target data set with added noise, the proposed network architecture is trained to obtain a model for denoising and recognizing infrared images. In this way, noise can be removed and the region where the key features of the image are located can be extracted. The positioning information and category information of the target can be obtained by inputting the key region into the detector, which can better improve the performance of small target detection.

[0004] The patent application with publication number CN116704284A discloses an enhanced refinement network and method for target detection in severe weather, wherein the enhanced refinement network includes an adaptive image enhancement module, a feature refinement module and a target detection module connected in sequence. The adaptive image enhancement module includes a limited contrast adaptive histogram equalization filter, an adaptive illumination enhancement filter and a Laplacian pyramid sharpening structure connected in sequence. The feature refinement module includes a down-sampling layer, a feature refinement layer and an up-sampling layer connected in sequence through a residual block. Target recognition is performed after feature enhancement and feature refinement.

[0005] However, the cascaded network will bring a lot of additional computational consumption, which will greatly prolong the inference time required by the model, which is contrary to the demand for high real-time performance in actual engineering applications. Therefore, a method is needed that does not consume additional computational resources while improving inference accuracy. SUMMARY

[0006] In view of the above, the purpose of the present application is to provide a feature fusion-based severe weather target detection method and device to solve the technical problem of high false detection rate and missed detection rate of target detection in severe weather scenes, without additional computational consumption, while improving the detection accuracy of the model in severe weather scenes.

[0007] To achieve the above-mentioned purpose of the application, the embodiment provides a feature fusion-based severe weather target detection method, which comprises the following steps:

[0008] The target detection model is used as a detection branch, and the shallow feature map of the real severe weather image is extracted by using the target detection model network, and the detection loss of the detection branch is constructed;

[0009] A severe weather processing model with fixed parameters is added as a severe weather processing enhancement branch, and the real severe weather image is processed to remove the severe weather interference factors to obtain a processed image by using the severe weather processing model;

[0010] The shallow feature map and the processed image are respectively subjected to hybrid random pooling to obtain pooled features, and the two pooled features are fused to obtain a fused feature;

[0011] The structural similarity between the fused feature and the pooled feature corresponding to the shallow feature map is calculated as a similarity loss, and the target detection model is parameter-optimized according to the similarity loss and the detection loss;

[0012] The target detection model after parameter optimization is used for target detection of the severe weather image to be detected.

[0013] Preferably, the severe weather processing model is a model for removing one of the severe weather interference factors such as fog, rain, snow, low light, and strong light, for the fog interference factor, the severe weather processing model adopts a DehazeFormer defogging model, for the rain interference factor, a DeTformer rain removal model based on a Transformer framework is adopted, and for the low light interference factor, a LightenDiffusion low light recovery model based on diffusion is adopted.

[0014] Preferably, the shallow feature map and the processed image are respectively subjected to hybrid random pooling to obtain pooled features, which comprises:

[0015] The processed image I en is down-sampled to have the same size as the shallow feature map F low to obtain an image I′ en , the image I′ en and the shallow feature map F low are respectively subjected to random average pooling or maximum pooling along the channel to obtain pooled features I e .n and F′ low :

[0016] I e " n = λAvgPool(I′ en ) ⊕ (1 - λ)MaxPool(I′ en )

[0017] F′ low = λAvgPool(F low ) ⊕ (1 - λ)MaxPool(F low )

[0018] where λ is a random value of 0 or 1, AvgPool(·) represents the average value along the channel dimension, ⊕ represents adding the feature maps, and MaxPool(·) represents the maximum value along the channel dimension.

[0019] Preferably, the two pooled features are fused to obtain a fused feature, including:

[0020] pooling the shallow feature map corresponding to the feature F′ low and the image processing corresponding to the feature I e " n obtaining the fused feature F fuse by weighted addition:

[0021] F fuse = βI e " n ⊕ (1 - β)F′ low

[0022] where β is a fusion weight coefficient, which is a random value, and β ∈ [0, 1].

[0023] Preferably, the structural similarity between the fused feature and the shallow feature map corresponding to the pooled feature is calculated as a similarity loss, including:

[0024] respectively calculating the mean and standard deviation and and covariance

[0025] of the pixel points in the fused feature F fuse and the shallow feature map corresponding to the pooled feature F′ low and standard deviation and and covariance to calculate the structural similarity SSIM(F​fuse F low ,

[0026]

[0027] based on a structural similarity SSIM(F fuse ,F low ) to calculate a similarity loss L ssim :

[0028] L ssim =β*(1-SSIM(F fuse ,F ′ low ))

[0029] wherein β is a fusion weight coefficient of the fusion of the shallow feature map corresponding to the pooled feature F low and the processed image corresponding to the pooled feature I e ″ n , C1 and C2 are constants to prevent the denominator from being 0.

[0030] Preferably, the detection loss of the detection branch includes a regression loss L CIoU representing the distance between the target predicted frame and the target real frame, a target confidence loss L obj representing the reliability of the target predicted frame, and a classification loss L cls representing the reliability of the predicted correct category.

[0031] The total loss L of the parameter optimization of the target detection model is:

[0032] L=α box L CIoU +α obj L obj +α cls L cls +α ssim L ssim

[0033] wherein α box , α obj , α cls , α ssim are balance factors for balancing each loss, and L ssim is the similarity loss.

[0034] Preferably, the target detection model after parameter optimization is used for target detection of the to-be-detected severe weather image, comprising:

[0035] The detected severe weather image is only input into the target detection model after parameter optimization, and the confidence and coordinates of the target frame are obtained through forward reasoning to realize target detection.

[0036] Preferably, the target detection model adopts a YOLOv5 model, and shallow feature maps are extracted by using a backbone network of YOLOv5.

[0037] To achieve the above-mentioned purposes, the embodiment further provides a device for detecting targets in adverse weather based on feature fusion, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the above-mentioned method for detecting targets in adverse weather based on feature fusion.

[0038] To achieve the above-mentioned purposes, the embodiment further provides a computer-readable storage medium having a program stored thereon, wherein the program is executed by a processor to implement the above-mentioned method for detecting targets in adverse weather based on feature fusion.

[0039] Compared with the prior art, the present application has at least the following beneficial effects:

[0040] The present application adds an adverse weather processing enhancement branch, guides the detection branch to learn by introducing a new loss, i.e., a structural similarity SSIM loss, and improves the target detection accuracy in adverse weather scenes; the adverse weather processing enhancement branch of the present application only works in the training stage and does not participate in the inference stage, so it does not bring additional computational consumption; when the present application performs pooling on images, a hybrid random pooling scheme is designed, which replaces the conventional deterministic pooling, and can solve the overfitting problem to a certain extent and improve the accuracy of the model; the present application fuses images to different degrees and performs different degrees of loss penalty, which has the advantages of reducing the training difficulty and generating different feature maps to improve the generalization of the model. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0042] Figure 1 is a flowchart of the method for detecting targets in adverse weather based on feature fusion provided by the embodiment;

[0043] Figure 2 is a schematic diagram of the adverse weather processing enhancement branch provided by the embodiment. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0045] like Figure 1 As shown in the embodiment, a method for detecting targets in severe weather based on feature fusion includes the following steps:

[0046] S1 uses the object detection model as a detection branch, extracts shallow feature maps of real severe weather images using the object detection model network, and constructs the detection loss of the detection branch.

[0047] In this embodiment, the target detection model for the detection branch can use YOLOv5. The first layer is the backbone network of YOLOv5, specifically a Conv convolutional layer, which consists of three parts: a two-dimensional convolutional layer with a kernel size of 2 and a stride of 2, a BN layer, and a SiLU activation function layer. The input is a real severe weather image I∈R 3×H×W After passing through this layer, the height and width become The shallow feature map F is obtained. low The shallow feature map F low The input to the second layer for object detection is combined with subsequent calculations to obtain the detection loss, which includes a regression loss L representing the distance between the predicted object bounding box and the ground truth object bounding box. CIoU L represents the target confidence loss, indicating the reliability of the target prediction box. obj The classification loss L represents the reliability of predicting the correct category. cls The detection loss is used for backpropagation.

[0048] S2, add a severe weather processing model with fixed parameters as a severe weather processing enhancement branch, and use the severe weather processing model to remove severe weather interference factors from real severe weather images to obtain the processed image.

[0049] In this embodiment, the added severe weather processing enhancement branch guides the detection branch to learn the features of the image after removing severe weather interference factors. The severe weather processing model is designed to remove at least one of the following severe weather interference factors: fog, rain, snow, low light, and strong light. For example, for fog, the severe weather processing model uses the DehazeFormer dehazing model; for rain, it uses the DeTformer rain dehazing model based on the Transformer framework; and for low light, it uses the LightenDiffusion low light recovery model based on diffusion. These severe weather processing models have fixed parameters and operate like a black box during application, without updating the model parameters. This design facilitates the replacement of various processing models, such as other dehazing, deraining, and low light enhancement models, resulting in better versatility.

[0050] S3. The shallow feature map and the processed image are respectively subjected to mixed random pooling to obtain pooled features, and the two pooled features are fused to obtain fused features.

[0051] In the embodiments, such as Figure 2 As shown, the pooling process includes: dividing the processed image I... en Downsampling to shallow feature map F low Image I′ is obtained with consistent size en Image I′ en and shallow feature map F low Perform average pooling or max pooling randomly along each channel to obtain the pooled feature I. e " n and F′ low :

[0052] I e " n =λAvgPool(I′) en )⊕(1-λ)MaxPool(I′ en )

[0053] F′ low =λAvgPool(F low )⊕(1-λ)MaxPool(F low )

[0054] Here, λ is a random value of 0 or 1. During training, a random λ is generated for each batch of images. Assuming that each batch of images has n images, the first batch of images will generate λ1, the second batch of images will generate λ2, and the λ in each batch is the same. AvgPool(·) means to calculate the average value along the channel dimension, ⊕ means to add the feature maps, and MaxPool(·) means to calculate the maximum value along the channel dimension.

[0055] In the embodiment, as shown in Figure 2 , the feature fusion includes: fusing the pooled features F' low corresponding to the shallow feature maps and the pooled features I' e corresponding to the processed image. n The fused features F fuse are obtained by weighted addition.

[0056] F fuse = βI e ′ n ⊕(1-β)F′ low

[0057] wherein β is a weight coefficient of fusion, is a random value, β ∈ [0, 1], a random β is generated for each batch of images during training, assuming that each batch of images is n, a β1 is generated for the first batch of images, a β2 is generated for the second batch of images, and the β in each batch is the same.

[0058] S4, the structural similarity between the fused features and the pooled features corresponding to the shallow feature maps is calculated as a similarity loss, and the target detection model is parameter-optimized according to the similarity loss and a detection loss.

[0059] In the embodiment, the similarity loss is calculated including:

[0060] S4-1, all pixel points of the image F fuse and F′ low are traversed to obtain the values of all pixel points, after hybrid random pooling, the channel numbers of F fuse and F′ low are both 1, therefore, only the pixel values of the channel need to be traversed, for convenience of representation, F fuse and F′ low are represented by x and y respectively, and the mean values μ x and μ y of the images x and y are calculated.

[0061]

[0062] wherein N is the number of image pixel points, x i is the value of the i-th pixel point in the image F fuse , and y i is the value of the i-th pixel point in the image F′ low .

[0063] S4-2, the mean values μ x and μ y are used to calculate the standard deviations σ fuse and σ low of the images F x and F′ y .

[0064]

[0065] S4-3, using the mean μ x and μ y Calculate the covariance σ fuse of the image F low and F' xy :

[0066]

[0067] S4-4, using μ x , μ y , σ x , σ y and σ xy Calculate the structural similarity SSIM:

[0068]

[0069] Where C1 and C2 are constants to prevent the denominator from being 0;

[0070] S4-5, x and y are brought into F fuse and F' low respectively, and the obtained SSIM loss is defined as:

[0071] L ssim = β * (1-SSIM(F fuse ,F' low ))

[0072] Where β is the weight coefficient of the fusion of the image I" en and the image F low .

[0073] According to the similarity loss and the detection loss, the parameters of the target detection model are optimized, the total loss L is adopted and the gradient is calculated using the back propagation algorithm, then the parameters are updated using the SGD optimizer to minimize the loss function, and the gradient back propagation is as follows Figure 1 The dashed line represents. Where the total loss L is:

[0074] L = α box L CIoU + α obj L obj + α cls L cls + α ssim L ssim

[0075] Where α box , α obj , α cls , α ssimrespectively, are balance factors for balancing respective losses.

[0076] S5, performing target detection on the to-be-detected severe weather image by using the target detection model with the optimized parameters.

[0077] In the embodiment, when the application detection is performed, the to-be-recognized severe weather image is input into the target detection model with the optimized parameters, and during the inference, the shallow feature map does not need to be extracted, and the image does not need to be input into the severe weather processing enhancement branch, but only needs to be input into the target detection model for forward inference to obtain the confidence and coordinates of the target frame.

[0078] Based on the same inventive concept, the embodiment further provides a severe weather target detection device based on feature fusion, comprising a memory and one or more processors, the memory stores executable code, and the one or more processors execute the executable code to implement the above-mentioned severe weather target detection method based on feature fusion, and specifically comprises the following steps:

[0079] S1, taking the target detection model as a detection branch, extracting a shallow feature map of a real severe weather image by using a target detection model network, and constructing a detection loss of the detection branch;

[0080] S2, adding a severe weather processing model with fixed parameters as a severe weather processing enhancement branch, and performing severe weather interference factor removal processing on the real severe weather image by using the severe weather processing model to obtain a processed image;

[0081] S3, obtaining a pooled feature by mixing and random pooling the shallow feature map and the processed image respectively, and performing feature fusion on the two pooled features to obtain a fused feature;

[0082] S4, calculating a structural similarity between the fused feature and the pooled feature corresponding to the shallow feature map as a similarity loss, and optimizing the target detection model according to the similarity loss and the detection loss;

[0083] S5, performing target detection on the to-be-detected severe weather image by using the target detection model with the optimized parameters.

[0084] The computing device provided by the embodiment comprises, in addition to the processor and the memory, internal buses, network interfaces, memories and other hardware required by services. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to realize the feature fusion-based severe weather target detection method described in S1-S5. Of course, in addition to the software implementation, the present application does not exclude other implementation manners, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to the logic units, but can also be hardware or logic devices.

[0085] Based on the same inventive concept, the embodiment further provides a computer readable storage medium having a program stored thereon, which, when executed by a processor, realizes the feature fusion-based severe weather target detection method described above, and specifically comprises the following steps:

[0086] S1, taking a target detection model as a detection branch, extracting a shallow feature map of a real severe weather image by using the target detection model network, and constructing a detection loss of the detection branch;

[0087] S2, adding a severe weather processing model with fixed parameters as a severe weather processing enhancement branch, and processing the real severe weather image by removing severe weather interference factors by using the severe weather processing model to obtain a processed image;

[0088] S3, respectively passing the shallow feature map and the processed image through a hybrid random pooling to obtain a pooled feature, and performing feature fusion on the two pooled features to obtain a fused feature;

[0089] S4, calculating a structural similarity between the fused feature and the pooled feature corresponding to the shallow feature map as a similarity loss, and performing parameter optimization on the target detection model according to the similarity loss and the detection loss;

[0090] S5, performing target detection on a severe weather image to be detected by using the target detection model with the optimized parameters.

[0091] In the embodiment, the computer readable medium includes permanent and non-permanent, removable and non-removable media, which can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data.

[0092] The specific embodiments described above have explained the technical solutions and beneficial effects of the present application, and it should be understood that the above description is only the most preferred embodiment of the present application, and is not used to limit the present application. Any modification, supplement and equivalent replacement made within the principle range of the present application shall be included in the protection range of the present application.

Claims

1. A severe weather target detection method based on feature fusion, characterized in that, The method comprises the following steps: a target detection model is used as a detection branch to extract a shallow feature map of a real severe weather image by using a target detection model network, and a detection loss of the detection branch is constructed; a severe weather processing model with fixed parameters is added as a severe weather processing enhancement branch to obtain a processed image by removing severe weather interference factors from the real severe weather image by using the severe weather processing model; the shallow feature map and the processed image are respectively subjected to mixed random pooling to obtain pooled features, and the two pooled features are fused to obtain a fused feature; a structural similarity between the fused feature and a corresponding pooled feature of the shallow feature map is calculated as a similarity loss, and a target detection model is parameter-optimized based on the similarity loss and the detection loss; the parameter-optimized target detection model is used for target detection on a severe weather image to be detected. 2.The severe weather target detection method based on feature fusion according to claim 1, characterized in that, The severe weather processing model is a model for removing one of fog, rain, snow, low light, and strong light, and comprises a DehazeFormer defogging model for removing fog, a DeTformer rain removal model based on a Transformer framework for removing rain, and a LightenDiffusion low light recovery model based on diffusion for removing low light. 3.The severe weather target detection method based on feature fusion according to claim 1, characterized in that, The shallow feature map and the processed image are respectively subjected to mixed random pooling to obtain pooled features, comprising: The processed image I en is down-sampled to the shallow feature map F low with the same size to obtain the image I e ' n , and the shallow feature map F e ' n are respectively randomly averaged or maximally pooled along the channel to obtain the pooled feature I low and F e " n and F l ' ow : I e " n = λAvgPool(I e ' n )⊕(1-λ)MaxPool(I e ' n ) F l ′ ow = λAvgPool(F low ) ⊕ (1 - λ)MaxPool(F low ) wherein λ is a random value of 0 or 1, AvgPool(·) represents an average value along a channel dimension, ⊕ represents adding feature maps, and MaxPool(·) represents a maximum value along a channel dimension. 4.The method of claim 1, wherein, The two pooled features are fused to obtain a fused feature, comprising: pooling the feature F corresponding to the shallow feature map low pooling the feature I" corresponding to the processed image en obtaining the fusion feature F by weighted addition fuse : F fuse = β I" en ⊕ (1 - β) F' low wherein β is a fusion weight coefficient, and is a random value, and β∈[0,1]. 5.The severe weather target detection method based on feature fusion according to claim 1, characterized in that, The structural similarity between the fused feature and a corresponding pooled feature of the shallow feature map is calculated as a similarity loss, comprising: respectively fuse and the mean value of the pixel points in the fusion feature F low and the standard deviation and and the covariance ​ According to the mean and standard deviation and and covariance Calculate the structural similarity SSIM(F fuse ,F′ low ): Similarity loss L fuse is calculated based on structural similarity SSIM(F low , F′ ssim ): L ssin = β * (1 - SSIM(F fuse , F') ) + (1 - β) * DSSIM(F low , F') wherein β is a fusion weight coefficient of the shallow feature map corresponding to the pooled feature F′ low and the processed image corresponding to the pooled feature I″ en C1 and C2 are constants to prevent the denominator from being 0. 6.The method of claim 1, wherein, The detection loss of the detection branch includes a regression loss L representing a distance between a target predicted frame and a target real frame CIoU , a target confidence loss L representing a reliability degree of the target predicted frame obj , and a classification loss L representing a reliability degree of a predicted correct category cls ; a total loss L for parameter-optimizing the target detection model is: L = a box L CIoU + a obj L obj + a cls L cls + a ssim L ssim wherein, α box , α obj , α cls , α ssim are balance factors balancing respective losses, L ssim is a similarity loss. 7.The severe weather target detection method based on feature fusion according to claim 1, characterized in that, the parameter-optimized target detection model is used for target detection on a severe weather image to be detected, comprising: only inputting the detected severe weather image into the parameter-optimized target detection model to obtain a confidence and coordinates of a target frame through forward reasoning, thereby realizing target detection. 8.The severe weather target detection method based on feature fusion according to claim 1, characterized in that, The target detection model adopts a YOLOv5 model, and a backbone network of the YOLOv5 model is used to extract the shallow feature map.

9. A device for detecting a severe weather target based on feature fusion, comprising a memory and one or more processors, wherein the memory stores executable code, and the device is characterized in that, The one or more processors execute the executable code to implement the severe weather target detection method based on feature fusion according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The program is stored on the storage medium and is executed by the processor to implement the severe weather target detection method based on feature fusion according to any one of claims 1-8.

Images