Fog-haze scene target detection method based on similarity fusion attention mechanism

Abstract

The application discloses a haze scene target detection method based on a similarity fusion attention mechanism, and the method comprises the following steps: acquiring a pre-training data set and a fine-tuning data set under a real haze scene, using an artificial fogging method to haze the pre-training data set, and performing image enhancement on the two data sets; the similarity fusion attention mechanism is added to a target detection model, aiming to reduce the effect of haze degradation on the prediction confidence reduction through the method of similar feature weighted fusion, and then increase the prediction accuracy under the haze scene. The pre-trained target detection model is pre-trained using the processed pre-training data set, and the pre-trained target detection model is fine-tuned using the processed fine-tuning data set, so that the target detection model suitable for the real haze scene is obtained. Finally, the image to be detected is input into the trained target detection model, and the category and coordinate information of the target to be detected are output.

Description

Technical Field

[0001] This invention belongs to the field of computer vision target detection, and in particular relates to a target detection method for hazy scenes based on a similarity fusion attention mechanism. Background Technology

[0002] Object detection is a hot research area in computer vision, and an essential prerequisite for many advanced vision tasks. It combines principles from multiple disciplines, including image processing and statistical learning. Its main purpose is to find objects of interest in complex image backgrounds, accurately and efficiently pinpoint their location and size using rectangular bounding boxes, and determine their category. Most existing object detection research focuses on objects in simple, clean environments. While these models achieve good results on existing object detection datasets, they often perform poorly in real-world scenarios due to interference from complex weather conditions such as smoke, fog, and haze.

[0003] Datasets specifically designed for object detection in hazy scenarios are difficult to obtain; they are not only few in number but also contain few categories and are small in scale, making it difficult to train models to achieve good results. To augment the dataset, the traditional approach is to artificially add fog to regular object detection datasets based on atmospheric scattering models. However, because it is impossible to accurately obtain depth information under monocular vision, artificially added fog has a different effect than naturally occurring fog, resulting in trained models that cannot adapt well to real-world foggy scenes.

[0004] Traditional object detection methods in hazy scenes either employ a two-stage approach of dehazing and detection, first dehazing the original image and then inputting the dehazed image into the detection algorithm; or they use transfer learning to learn the neighborhood invariance between hazy and fog-free images, enabling the algorithm to directly detect objects in foggy images. Both approaches essentially attempt to overcome the degradation mechanism of haze on images. The former tries to restore the image to its fog-free state to reduce the impact of fog; the latter attempts to make the features extracted by the algorithm as similar as possible in foggy and fog-free conditions to ignore the influence of fog. However, both have limitations. The dehazing process can introduce artifacts, color distortion, and fog residue into the image, introducing new interference factors into the detection process. Therefore, not all images treated with dehazing algorithms will improve detection accuracy. Furthermore, transfer learning is unstable, leading to difficulties in training, convergence, and low accuracy. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the prior art and provide a target detection method for hazy scenes based on a similarity fusion attention mechanism. The main content is to propose a training method based on pre-training with artificial fog and fine-tuning in real foggy scenes, aiming to solve the problem that artificial fog methods cannot adapt well to real foggy scenes through pre-training and fine-tuning; and to propose a similarity fusion attention mechanism, aiming to reduce the impact of haze degradation on prediction confidence by using a weighted fusion method of similar features, thereby increasing prediction accuracy.

[0006] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0007] A target detection method for hazy scenes based on a similarity fusion attention mechanism includes:

[0008] S1. Obtain a pre-training dataset for object detection and a fine-tuning dataset for object detection in a real haze scene. Apply artificial fog to the pre-training dataset to blur it, and obtain a new pre-training dataset. Preprocess the new pre-training dataset and the fine-tuning dataset to obtain the processed pre-training dataset and the processed fine-tuning dataset.

[0009] S2. Construct an object detection model for object detection, wherein the first convolutional layer of each C2f module of the YOLOv8 backbone network is replaced with a convolutional module with a similarity fusion attention mechanism.

[0010] In the convolutional module with similarity fusion attention mechanism, the input data is convolved by the first standard convolutional layer to obtain the first feature map. The first feature map is divided equally along the channel dimension to obtain the first segment data. The first segment data is divided equally into multiple windows along the width and height dimensions to obtain the first window data. The input data is divided equally along the channel dimension to obtain the second segment data. The second segment data is divided equally into multiple windows along the width and height dimensions to obtain the second window data. The second standard convolutional layer is convolved with the second window data to obtain the second feature map. The width and height dimensions of the first window data are merged to obtain the third feature map. The Euclidean distance between each pair of column vectors of the third feature map along the channel dimension is calculated and the inverse of the calculation result is taken to obtain the distance result matrix. The column vector dimension of the distance result matrix is ​​subjected to a Softmax operation to obtain the weight matrix. The second feature map and the weight matrix are multiplied to obtain the fourth feature map. The first feature map and the fourth feature map are added to obtain the output data of the convolutional module.

[0011] S3. After randomly initializing the parameters in the object detection model, pre-train the object detection model on the processed pre-training dataset, and fine-tune the pre-trained object detection model on the processed fine-tuning dataset to obtain the trained object detection model.

[0012] S4. Input the image to be detected into the trained target detection model, and output the category and coordinate information of the target to be detected.

[0013] Based on the above scheme, each step can be implemented in the following preferred manner.

[0014] Preferably, in step S1, the pre-training dataset uses the MSCOCO dataset, and the fine-tuning dataset uses the RTTS dataset.

[0015] Preferably, the artificial fogging method involves adding fog blur to the original image using an atmospheric scattering model to obtain the fog-blurred image I(x):

[0016] I(x)=J(x)t(x)+A(1-t(x))

[0017] t(x)=e -βd(x)

[0018] Where J(x) is the original image, A is the atmospheric light intensity, t(x) represents the atmospheric transmittance, β is the atmospheric scattering coefficient, and d(x) represents the image depth matrix.

[0019] Preferably, in step S1, the preprocessing involves performing five image enhancement processes on the images in the new pre-training dataset and the fine-tuning dataset: random scaling and cropping, random color space adjustment, random inversion and rotation, random cropping, and mosaic enhancement.

[0020] Preferably, in step S2, the kernel size of the first standard convolutional layer is 1×1, and the stride is 1.

[0021] Preferably, in step S2, the kernel size of the second standard convolutional layer is 1×1, and the stride is 1.

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] This invention proposes a target detection method for hazy scenes based on a similarity fusion attention mechanism. This method is based on pre-training with artificial fog and fine-tuning training under real foggy scenes to enhance the adaptability of the target detection model to real foggy scenes. Furthermore, it proposes a similarity fusion attention mechanism to enhance the target detection effect in hazy scenes. Attached Figure Description

[0024] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0025] Figure 2 This is an architecture diagram of the convolutional module with a similarity fusion attention mechanism of the present invention;

[0026] Figure 3 This is a schematic diagram of the multi-head mechanism in the similarity fusion attention mechanism of the present invention;

[0027] Figure 4 This is a schematic diagram of the window mechanism in the similarity fusion attention mechanism of the present invention;

[0028] Figure 5 This is a flowchart of the target detection model training process of the present invention. Detailed Implementation

[0029] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0030] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0031] This invention addresses the problem of limited datasets for target detection in hazy scenes by providing a similarity-fusion attention mechanism-based target detection method. It employs a pre-training approach based on artificial fog and fine-tuning training in real-world foggy scenarios to enhance the target detection model's adaptability to such conditions. Furthermore, it designs an attention mechanism based on the mechanism of fog's effect. Fog reduces the overall latent space prediction confidence of the target detection model, causing it to fail to reach the confidence threshold for correct judgments. By incorporating the similarity-fusion attention mechanism, the target detection model uses similarity weights to boost low-confidence features, helping them overcome the confidence threshold and thus improving target detection performance.

[0032] like Figure 1 As shown, in a preferred embodiment of the present invention, the above-mentioned target detection method for hazy scenes based on a similarity fusion attention mechanism includes the following steps S1 to S4. The specific implementation process of each step will be described in detail below.

[0033] S1. Obtain a pre-training dataset for object detection and a fine-tuning dataset for object detection in a real haze scene. Apply artificial fog to the pre-training dataset to blur it, resulting in a new pre-training dataset. Perform preprocessing on the new pre-training dataset and the fine-tuning dataset to obtain the processed pre-training dataset and the processed fine-tuning dataset.

[0034] It should be noted that in step S1 of this invention, the pre-training dataset needs to have as many categories of targets to be detected, as many instances of targets to be detected, and as many images as possible, so that the target detection model can learn more general features. The fine-tuning dataset for real foggy scenes should include images of real foggy scenes so that the target detection model can learn features of real foggy scenes.

[0035] In step S1 of this embodiment, the pre-training dataset uses the MSCOCO dataset, and the fine-tuning dataset uses the RTTS dataset. The MSCOCO dataset contains 80 category labels and approximately 120,000 images, while the RTTS dataset contains 20 category labels and approximately 4,000 images.

[0036] It should be noted that in step S1 of this invention, for the pre-training dataset, an artificial fogging method is required to perform random fog blurring. Specifically, this means using an atmospheric scattering model to perform center fog enhancement on the image with random fog concentration, so that the target detection model can learn the features under different fog concentrations.

[0037] The above-mentioned artificial fogging method involves adding fog blur to the original image using an atmospheric scattering model, resulting in an image I(x) with fog added:

[0038] I(x)=J(x)t(x)+A(1-t(x))

[0039] t(x)=e -βd(x)

[0040] Where J(x) is the original image, A is the atmospheric light intensity, and t(x) represents the atmospheric transmittance, which is related to the distance between the target object and the camera in the real scene, and also to the fog concentration. β is the atmospheric scattering coefficient, and d(x) represents the image depth matrix, which is obtained by adding fog to the center, making the center darker and the edges lighter.

[0041] It should be noted that in step S1 of this invention, the pre-training dataset (i.e., the new pre-training dataset) and the fine-tuning dataset that have undergone fogging processing also need to be preprocessed. Specifically, this involves five image enhancement processes: random scaling and cropping, random color space adjustment, random inversion and rotation, random cropping, and mosaic enhancement. These five image enhancement methods are used to suppress overfitting of the object detection model during training.

[0042] In the preprocessing step S1 of this embodiment, random scaling and cropping involves randomly shrinking and enlarging the image and cropping a portion of it for training; random color space adjustment refers to modifying the color channels so that the object detection model is not affected by color changes; random reversal and rotation refers to randomly reversing the image vertically or horizontally and rotating the image at random angles; the aforementioned random cutting refers to randomly deleting parts of the image to make the object detection model robust to occlusion; and mosaic enhancement refers to combining four different images into a new image to create new content.

[0043] S2. Construct an object detection model for object detection. The object detection model replaces the first convolutional layer of each C2f module of the YOLOv8 backbone network with a convolutional module with a similarity fusion attention mechanism.

[0044] In the convolutional module with a similarity fusion attention mechanism, the input data is convolved by the first standard convolutional layer to obtain the first feature map X. out The first feature map is divided equally along the channel dimension to obtain the first segmentation data. The first segmentation data is then divided equally into multiple windows along the width and height dimensions to obtain the first window data. The input data X is then... in The first feature map y1 is obtained by dividing the data equally along the channel dimension. This second segment is then divided into multiple windows along both width and height dimensions. A second standard convolutional layer is used to convolve the second window data to obtain the second feature map y1. The width and height dimensions of the first window data are merged to obtain the third feature map y2. The Euclidean distance between each pair of column vectors along the channel dimension of the third feature map y2 is calculated, and the result is inversely represented to obtain the distance matrix. A softmax operation is performed on the column vector dimensions of the distance matrix to obtain the weight matrix y3. The second feature map y1 and the weight matrix y3 are multiplied to obtain the fourth feature map y1. out , the first feature map X out and the fourth feature map y out The results are summed to obtain the output data of the convolution module.

[0045] It should be noted that in step S2 of the present invention, the kernel size of the first standard convolutional layer is 1×1 and the stride is 1; the kernel size of the second standard convolutional layer is 1×1 and the stride is 1.

[0046] In this embodiment, the object detection model uses the YOLOv8 object detection framework, including a backbone network composed of CSPDarknet53, a feature fusion network composed of feature pyramids, and a detection head network composed of decoupled heads. Specifically, the CSPDarknet53 backbone network is composed of initial convolutional layers and C2F modules connected alternately. A similarity fusion attention mechanism is added to the first convolutional layer in each C2F module structure to form a convolutional module with a similarity fusion attention mechanism. This convolutional module replaces the original first convolutional layer in each C2F module. The initial convolutional layer uses a 3×3 convolutional kernel with a stride of 2 to reduce the dimensionality of the feature map in both width and height dimensions.

[0047] The aforementioned convolution module includes steps such as convolution operations, Euclidean distance calculation, Softmax, and matrix multiplication. It also incorporates a multi-head mechanism to improve feature extraction performance and a windowing mechanism to reduce computational cost. Specifically, the multi-head mechanism involves dividing the output of the intermediate layers of the neural network into n equal parts along the channel dimension, and then inputting the segmented results into the next layer of the neural network. This improves feature extraction performance by capturing the similarity information of different channel subspaces. The windowing mechanism involves dividing the output of the intermediate layers of the neural network into multiple windows along the width and height dimensions, extracting similarity information from each window separately, thus reducing computational cost.

[0048] like Figure 2 The diagram shows the architecture of a convolutional module with a similarity fusion attention mechanism. The dashed boxes marked with 1 represent the insertion points of the similarity fusion attention mechanism. For each C2f module structure of the YOLOv8 object detection framework CSPDarknet53 backbone network, the similarity fusion attention mechanism is added to the first convolutional layer, forming the aforementioned convolutional module. The input data X for the above convolutional module... in First, the input data is convolved by a first standard convolutional layer with a stride of 1 and a kernel of 1×1 to obtain the first feature map X. out The dashed box marked 2 represents the architecture diagram of the similarity fusion attention mechanism. In this mechanism, firstly, a multi-head similarity fusion attention mechanism is applied, such as... Figure 3 The diagram shows the specific operational process of the multi-head mechanism. Input data X... in and the first feature map X out Divide the data into n equal parts along the channel dimension, each forming a tensor of shape (n, C / n, W, H), resulting in two types of segmented data. The input data X... in After segmentation, the second segmentation data is obtained, and the first feature map X is obtained. outAfter segmentation, the first segmented data is obtained. Then, the two segmented data are input into subsequent calculations. This captures the similarity relationships between different feature subspaces, increasing the effectiveness of the attention mechanism without increasing computational cost. Then, a similarity-fusion attention windowing mechanism is applied, such as... Figure 4 The diagram shows the specific operation flow of the window mechanism. Specifically, the input data X is... in and the first feature map X out The feature map is divided into m equal parts along both the width (W) and height dimensions, forming a tensor of (m*m, C, W / m, H / m), which is then input into subsequent calculations. This effectively reduces the computational cost of the attention mechanism. Here, W, H, and C represent the width, height, and channel dimensions of the output feature map, respectively; n represents the number of segments along the channel dimensions; and m represents the number of segments along the width and height dimensions.

[0049] Input data X in and the first feature map X out After processing along the channel dimension and width and height dimensions respectively, the input data X in Corresponding to the second window data, the first feature map X out For the data in the first window, a second standard convolutional layer with a stride of 1 and a kernel of 1×1 is used to convolve the data in the second window to obtain the second feature map y1. The width and height dimensions of the first window data are merged to obtain the third feature map y2. The Euclidean distance between each pair of column vectors along the channel dimension of the third feature map y2 is calculated to obtain the distance result matrix. A softmax operation is performed on the column vector dimension of the distance result matrix to obtain the weight matrix y3. The second feature map y1 and the weight matrix y3 are multiplied together (matrix multiplication operation). Figure 2 , 3 From the matrix multiple in 4, we obtain the fourth feature map y. out , the first feature map X out and the fourth feature map y out The weights are summed to obtain the final output data of the convolutional module. The weight matrix described above has the following functional form:

[0050] y3 = Softmax(-cdist(y2)) T )

[0051] Where cdist is the Euclidean distance calculation function, used to calculate the Euclidean distance between any two column vectors of the distance result matrix; Softmax is the normalization exponent function, used to normalize the dimension of the matrix column vectors.

[0052] Additionally, it should be noted that the aforementioned C2f module structure also includes a Cross Stage Partial (CSP) structure for feature extraction, reducing gradient repetition during backpropagation and enhancing network performance. The spatial pyramid pooling layers at the ends of the aforementioned CSPDarknet53 backbone network capture features at multiple scales, enhancing detection performance. Furthermore, the aforementioned feature pyramid combines semantic information of different depths through feature concatenation and upsampling, and uses the C2f module to extract features, combining high-level spatial semantic features with low-level spatial information to improve the accuracy of object detection at different scales. The aforementioned Decoupled Head network structure extracts the location and category features of the image respectively, and outputs the location information and category respectively.

[0053] S3. After randomly initializing the parameters in the object detection model, pre-train the object detection model on the processed pre-training dataset, and fine-tune the pre-trained object detection model on the processed fine-tuning dataset to obtain the trained object detection model.

[0054] It should be noted that, in this embodiment, Figure 5 The training process of the object detection model is demonstrated. First, an artificial fogging method is applied to a publicly available object detection dataset for pre-training to blur and enhance the image, resulting in a processed pre-training dataset used for model pre-training. After pre-training, an image enhancement method other than artificial fogging is applied to a target detection dataset in a hazy scene for fine-tuning, resulting in a processed fine-tuning dataset used for model fine-tuning. The pre-trained target detection model is then fine-tuned on this dataset to obtain a well-trained target detection model for hazy scenes. During fine-tuning, a small learning rate is used to train the target detection model, optimizing the detection loss and performing gradient backpropagation. Finally, parameters are updated, and this process is iterated until convergence, resulting in the final trained target detection model.

[0055] S4. Input the image to be detected into the trained target detection model, and output the category and coordinate information of the target to be detected.

[0056] It should be noted that once the object detection model is trained, an image to be detected can be input, and the object detection model will perform inference to obtain information such as the center point, width, height, and category of the object to be detected in the image.

[0057] To better demonstrate the specific implementation and technical effects of the present invention, the target detection method for hazy scenes based on a similarity fusion attention mechanism shown in steps S1 to S4 of the above preferred implementation is applied to a specific example.

[0058] Example

[0059] To demonstrate the technical effectiveness of this embodiment, the invention was validated on the fogged MSCOCO dataset. The detection accuracy of Yolov8 models of different sizes with and without the similarity fusion attention mechanism is shown in Table 1:

[0060] Table 1. Localization accuracy of the present invention on two datasets.

[0061]

[0062]

[0063] In Table 1, Yolov8n+, Yolov8s+, and Yolov8m+ represent the target detection models constructed by adding a similarity fusion attention mechanism to the original Yolov8n, Yolov8s, and Yolov8m networks, respectively. As can be seen from Table 1, this invention helps overcome the effects of haze degradation and improves detection accuracy.

[0064] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A target detection method for hazy scenes based on a similarity fusion attention mechanism, characterized in that, Includes the following steps: S1. Obtain a pre-training dataset for object detection and a fine-tuning dataset for object detection in a real haze scene. Apply artificial fog to the pre-training dataset to blur it, and obtain a new pre-training dataset. Preprocess the new pre-training dataset and the fine-tuning dataset to obtain the processed pre-training dataset and the processed fine-tuning dataset. S2. Construct an object detection model for object detection, wherein the first convolutional layer of each C2f module of the YOLOv8 backbone network is replaced with a convolutional module with a similarity fusion attention mechanism. In the convolutional module with similarity fusion attention mechanism, the input data is convolved by the first standard convolutional layer to obtain the first feature map. The first feature map is divided equally along the channel dimension to obtain the first segment data. The first segment data is divided equally into multiple windows along the width and height dimensions to obtain the first window data. The input data is divided equally along the channel dimension to obtain the second segment data. The second segment data is divided equally into multiple windows along the width and height dimensions to obtain the second window data. The second standard convolutional layer is convolved with the second window data to obtain the second feature map. The width and height dimensions of the first window data are merged to obtain the third feature map. The Euclidean distance between each pair of column vectors of the third feature map along the channel dimension is calculated and the inverse of the calculation result is taken to obtain the distance result matrix. The column vector dimension of the distance result matrix is ​​subjected to a Softmax operation to obtain the weight matrix. The second feature map and the weight matrix are multiplied to obtain the fourth feature map. The first feature map and the fourth feature map are added to obtain the output data of the convolutional module. S3. After randomly initializing the parameters in the object detection model, pre-train the object detection model on the processed pre-training dataset, and fine-tune the pre-trained object detection model on the processed fine-tuning dataset to obtain the trained object detection model. S4. Input the image to be detected into the trained target detection model, and output the category and coordinate information of the target to be detected.

2. The target detection method for hazy scenes based on a similarity fusion attention mechanism as described in claim 1, characterized in that, In step S1, the pre-training dataset uses the MSCOCO dataset, and the fine-tuning dataset uses the RTTS dataset.

3. The target detection method for hazy scenes based on a similarity fusion attention mechanism as described in claim 1, characterized in that, The artificial fogging method involves adding fog blur to the original image using an atmospheric scattering model, resulting in an image I(x) with fog added: I(x)=J(x)t(x)+A(1-t(x)) t(x)=e -βd(x) Where J(x) is the original image, A is the atmospheric light intensity, t(x) represents the atmospheric transmittance, β is the atmospheric scattering coefficient, and d(x) represents the image depth matrix.

4. The target detection method for hazy scenes based on a similarity fusion attention mechanism as described in claim 1, characterized in that, In step S1, the preprocessing involves performing five image enhancement processes on the images in the new pre-training dataset and the fine-tuning dataset: random scaling and cropping, random color space adjustment, random inversion and rotation, random cropping, and mosaic enhancement.

5. The target detection method for hazy scenes based on a similarity fusion attention mechanism as described in claim 1, characterized in that, In step S2, the kernel size of the first standard convolutional layer is 1×1, and the stride is 1.

6. The target detection method for hazy scenes based on a similarity fusion attention mechanism as described in claim 1, characterized in that, In step S2, the kernel size of the second standard convolutional layer is 1×1, and the stride is 1.

Images