An image enhancement method based on multi-modal enhanced grid mask and fast fourier convolution

Abstract

The application discloses an image enhancement method based on multi-modal enhancement grid masking and fast Fourier convolution, which comprises the following steps: S1, collecting underwater typical target forward-looking sonar data to make an initial data set; S2, performing data enhancement on the original image of the forward-looking sonar in a multi-modal shielding mode; S3, performing data enhancement on the image after multi-modal shielding frame enhancement in a traditional image transformation mode, simulating changes in illumination, noise and viewing angle in different environments to generate diversified training data; S4, constructing an image enhancement network based on fast Fourier convolution, and improving the processing and enhancement capacity for the shielding area by using a global receptive field; and S5, enhancing the image by using the trained model to obtain a clear and detailed target shielding image. By combining the data enhancement method in the multi-modal shielding mode and the network based on fast Fourier convolution, the image enhancement quality and efficiency of the severely shielded target of the forward-looking sonar are significantly improved, and the method has good generalization capacity and practical value.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of sonar image processing, and particularly relates to a sonar image enhancement method based on multi-modal enhancement grid masking and fast Fourier convolution. BACKGROUND

[0002] Sonar technology has a wide range of applications in ocean exploration, target recognition and underwater exploration, especially in submarine monitoring, seabed resource management and underwater unmanned system navigation. Sonar image technology is an important tool for underwater target detection. However, due to the influence of the complexity of the underwater environment and the characteristics of sound wave propagation, sonar imaging often has problems such as large noise, low resolution and unstable imaging, which brings great challenges to underwater target detection and tracking.

[0003] In complex underwater environments, factors such as occlusion, lighting changes and noise interference further exacerbate the decline in image quality, significantly reducing the detection accuracy and recognition efficiency of the target. Traditional sonar image processing techniques often have limited effect in dealing with these problems, especially in the case of large-area occlusion and local target loss, existing methods cannot effectively restore the global consistency and local details of the image. Therefore, how to improve the clarity and detection accuracy of sonar images through more advanced image enhancement methods has become an important direction of current research. SUMMARY

[0004] The purpose of the present application is to propose an image enhancement method based on multi-modal enhancement grid masking and fast Fourier convolution, by introducing multi-modal data enhancement and frequency domain processing technology, simulating the occlusion and noise conditions in complex environments, effectively improving the global perception and detail processing capability of the image, thereby significantly improving the accuracy and robustness of underwater target detection, for target detection and recognition in complex underwater environments.

[0005] The technical solution to achieve the purpose of the present application is:

[0006] An image enhancement method based on multi-modal enhancement grid masking and fast Fourier convolution, comprising the following steps:

[0007] S1, collect underwater typical target forward-looking sonar data, extract and convert the sonar metadata to coordinate axes, obtain sonar images frame by frame, arrange and label the data set, and make recognition training and test data sets of sonar images;

[0008] S2, construct a multi-modal occlusion forward-looking sonar image data enhancement framework, simulate complex occlusion conditions, enhance the images of the recognition training and test data sets, and generate more challenging training data and corresponding occlusion masks;

[0009] S3, construct a forward-looking sonar image data enhancement framework of traditional image transformation, simulate the illumination, noise and view angle changes in different environments, enhance the data images obtained in step S2, and generate diversified data;

[0010] S4, construct an image enhancement network based on fast Fourier convolution, use a global receptive field combined with local convolution and a global Fourier transform branch, enhance the diversified data generated in step S3, and generate higher quality enhanced images.

[0011] Further, the step S1 specifically comprises:

[0012] S1.1, collecting underwater sonar data of typical targets by using a forward-looking sonar;

[0013] S1.2, extracting and converting the underwater sonar metadata to coordinate axes, obtaining sonar images frame by frame, scaling all sonar images to a resolution of 1280x720 pixels, and labeling them according to requirements to produce labeled sonar image data;

[0014] S1.3, processing the sonar image data obtained in step S1.2, extracting the target region in the image, generating the corresponding bounding box label, converting the bounding box label to YOLO format, and using it as a data set for target detection in the network training process.

[0015] Further, the typical targets include conical targets, mesh targets and irregular targets.

[0016] Further, the multi-modal occlusion forward-looking sonar image data enhancement framework in step S2 includes a grid occlusion module, a random erasure module and a rough discard module to simulate different types of occlusion in underwater environments.

[0017] Further, the grid occlusion module generates an alternating grid-shaped occlusion pattern on the image to mask some pixels, which is represented by the following formula:

[0018]

[0019] Where d is the size of the grid, representing the width and height of the grid; (x, y) is the pixel coordinate in the image, and the grid occlusion generates a binary mask M grid , which covers the alternating grid cells. After applying the grid mask, the occlusion result of the image is represented as:

[0020] I masked (x,y)=I(x,y)·M grid (x,y)

[0021] Where I(x, y) is the pixel value of the original image, and I masked(x, y) is the occluded image.

[0022] Further, the random erasing module simulates random noise or irregular occlusion by randomly selecting a rectangular region in the image and setting its pixel value to zero, which is expressed as follows:

[0023] R erase = (x1, y1, x2, y2)

[0024] where (x1, y1) and (x2, y2) represent the top-left and bottom-right coordinates of the randomly selected rectangular region, and the randomly erased image is represented as:

[0025]

[0026] That is, the pixel value in the randomly selected region is set to zero, and the original image pixel value is retained in other regions.

[0027] Further, the rough discarding module simulates object occlusion in the environment by generating multiple discontinuous rectangular holes in the image, which is described by the following formula:

[0028]

[0029] where n is the number of randomly generated discarded regions, and represent the top-left and bottom-right coordinates of the ith rectangular region, and the occlusion result is represented as:

[0030]

[0031] Further, the data image enhancement in step S3 of the forward-looking sonar image data enhancement framework specifically includes:

[0032] S3.1, horizontal rotation: flip the image left and right with a probability of 50%, which is expressed as:

[0033] I flipped (x, y) = I(W-x, y)

[0034] where I(x, y) is the original image pixel value, W is the width of the image, and I flipped (x, y) is the flipped image pixel value.

[0035] S3.2, rotation: randomly rotate the image in the range of -45° to +45°, which is:

[0036] I rotated (x', y') = I(xcosθ-ysinθ, xsinθ+ycosθ

[0037] where θ is the rotation angle, (x, y) is the pixel coordinate of the original image, and (x', y') is the pixel coordinate of the new position after rotation;

[0038] S3.3, brightness and contrast adjustment: randomly adjust the brightness and contrast of the image, the formula is as follows:

[0039] I adjusted (x, y) - a I(x, y) + β

[0040] where a is the contrast adjustment coefficient, β is the brightness offset, and I(x, y) is the pixel value of the original image;

[0041] S3.4, add Gaussian noise to the image, which is:

[0042] I noisy (x, y) - I(x, y) + N(0, σ 2 )

[0043] where N(0, σ 2 ) is Gaussian noise with mean 0 and variance σ 2 ;

[0044] S3.5, use a smoothing filter to blur the image, which is:

[0045]

[0046] where σ is the standard deviation of the Gaussian distribution, and I(x+i, y+j) is the pixel value in the image with (x, y) as the center and distance i, j;

[0047] S3.6, use Gaussian blur or motion blur to simulate underwater smoke or environmental blur effect;

[0048] S3.7, image fusion: including discrete cosine transform:

[0049]

[0050] where I(x, y) is the pixel value of the original image, F(u, v) is the coefficient after DCT transformation, and N is the size of the image block.

[0051] Further, the image enhancement network based on fast Fourier convolution in step S4 includes a local branch, a global branch and a branch fusion module. The local branch uses convolution operation to process the local detail information of the image. The global branch converts the image to the frequency domain through Fourier transform to capture global information. The branch fusion module is used to fuse the outputs of the two branches.

[0052] Further, the global branch converts the image to the frequency domain through Fourier transform specifically includes:

[0053] The input image dimension is HxWxC, and after Fourier transform, a complex frequency domain representation is obtained:

[0054]

[0055] The complex number is separated into real and imaginary parts, and then convolution operation is performed in the domain, combined with ReLU activation function, batch normalization layer and 1x1 convolution, to process the separated real components, enhance global information processing;

[0056] After frequency domain processing, the data is returned to the spatial domain through inverse Fourier transform.

[0057] Compared with the prior art, the present application has the following advantages:

[0058] 1. The multi-modal occlusion enhancement framework can effectively simulate the occlusion in the complex underwater environment, improve the adaptability of the model under complex occlusion conditions, and effectively improve the robustness of the model;

[0059] 2. Considering the influence of different underwater environmental light, noise and perspective changes, a traditional image transformation framework is added to enhance the model's ability to cope with variable environments, improve the diversity of images, and effectively enhance the generalization ability of the model;

[0060] 3. The traditional convolutional network has the disadvantage of limited local receptive field, and it is difficult to ensure global consistency when processing large-area occlusion and complex structure. The present application introduces fast Fourier convolution (FFC) technology, combines global Fourier transform with local convolution, and can preserve image details while ensuring global consistency, especially suitable for large-area occlusion repair in complex scenes, greatly improving the precision and robustness of image enhancement and repair;

[0061] 4. By combining fast Fourier convolution (FFC), multi-modal occlusion and traditional image transformation, the robustness and adaptability of the model in a diversified environment are improved, ensuring the diversity of generated training data in different scenarios, and thus improving the generalization performance and overall enhancement effect of the model. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 The algorithm flowchart of the present application;

[0063] Figure 2 The multi-modal occlusion structure flowchart in the present application;

[0064] Figure 3 The traditional image transformation structure flowchart in the present application;

[0065] Figure 4 The Fourier convolution network flowchart in the present application;

[0066] Figure 5 A sonar image for an embodiment of the present application unmanned underwater vehicle;

[0067] Figure 6 An image enhanced by a multi-modal enhancement structure for an embodiment of the present application;

[0068] Figure 7 An image enhanced by a fast Fourier convolution network for an embodiment of the present application. DETAILED DESCRIPTION

[0069] The present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0070] As shown in the drawings, Figure 1 The present application relates to an image enhancement method based on multi-modal enhancement grid masking and fast Fourier convolution, and the implementation process includes the following steps:

[0071] S1, collect underwater typical target forward-looking sonar data, extract and coordinate axis conversion of sonar metadata, obtain sonar image frame by frame, arrange and label the data set, and make recognition training and test data set of sonar image;

[0072] S1.1 uses the forward-looking sonar to collect underwater sonar data of typical targets, and the typical targets include conical targets, net-like targets, irregular targets, etc.

[0073] S1.2 extracts and coordinate axis conversion of sonar metadata, obtains sonar image frame by frame, scales all sonar images to 1280*720 pixel resolution, and labels according to requirements to make sonar image data special for the present application;

[0074] S1.3 processes the sonar image data, extracts the target area in the image, and generates the corresponding bounding box (bounding box) label. Convert these labels to YOLO format for use as a data set for target detection in the network training process.

[0075] S2, construct a multi-modal occlusion forward-looking sonar image data enhancement framework, simulate complex occlusion conditions, and generate more challenging training data and corresponding occlusion masks (mask);

[0076] The multi-modal occlusion forward-looking sonar image data enhancement framework is as shown in the drawings, Figure 2 The data enhancement process specifically includes:

[0077] S2.1 Three self-defined occlusion methods are mainly used for multi-modal occlusion data augmentation of sonar recognition dataset, which are GridMask, RandomErasing and CoarseDropout, respectively, to simulate different types of occlusion in underwater environment. The specific formula is as follows:

[0078] S2.2 GridMask generates an alternating grid-shaped occlusion pattern on the image to mask part of the pixels, and the formula is as follows:

[0079]

[0080] Where d is the size of the grid, and represents the width and height of the grid; (x, y) is the pixel coordinate in the image. GridMask generates a binary mask M grid , which covers the alternating grid cells. After applying the grid mask, the occlusion result of the image can be represented as:

[0081] I masked (x,y)=I(x,y)·M grid (x,y)

[0082] Where I(x, y) is the original image pixel value, and I masked (x,y) is the occluded image. GridMask forces the model to learn to extract effective information from the image under partial occlusion by using a regular occlusion pattern. It is suitable for situations where there are periodic or regular obstacles in the underwater environment, and enhances the robustness of the model under local information loss.

[0083] S2.3 RandomErasing randomly selects a rectangular region in the image and sets its pixel value to zero to simulate random noise or irregular occlusion, and the formula is as follows:

[0084] R erase =(x1,y1,x2,y2)

[0085] Where (x1, y1) and (x2, y2) represent the coordinates of the upper left corner and the lower right corner of the randomly selected rectangular region, respectively. The image after random erasing can be represented as:

[0086]

[0087] That is, the pixel values in the randomly selected area are set to zero, and the original image pixel values are retained in other areas. Random erasing simulates unpredictable interference in real-world scenarios, such as noise or irregular objects in underwater environments, through irregular occlusion. This effectively enhances the model's ability to adapt to sudden or random occlusions, enabling it to make accurate predictions even with incomplete information.

[0088] S2.4 CoarseDropout simulates object occlusion in the environment by generating multiple discontinuous rectangular holes in the image. The multiple dropout regions are described by the following formula:

[0089]

[0090] where n is the number of randomly generated dropout regions, and represent the top-left and bottom-right coordinates of the ith rectangular region. The occlusion result is represented as:

[0091]

[0092] CoarseDropout can simulate discontinuous occlusion in underwater environments, such as floating objects or debris, which can interfere with the image. Through this discontinuous occlusion method, the model can learn to extract key feature information even with a large amount of information loss, especially in complex and dynamic environments.

[0093] In the multi-modal occlusion data augmentation, the generated images retain important target information and are accompanied by corresponding occlusion masks (Mask). The mask is generated based on the occlusion region to ensure that the model can learn how to perform target detection under complex occlusion conditions during training. It can also be used as input for further data augmentation in the Fast Fourier Convolutional Network.

[0094] In this step, random combinations are performed through respective application probabilities, which are 0.4 for GridMask, 0.3 for RandomErasing, and 0.3 for CoarseDropout. During the occlusion process, the fill value of the occlusion mask region is set to 0. For GridMask, the number of grids is randomly selected between 3 and 6. The maximum occlusion ratio for RandomErasing is set to 0.1, and the minimum aspect ratio is set to 0.3. The number of holes for CoarseDropout is randomly selected between 1 and 30, and the height and width of the holes are set between 16 and 128. After multi-modal occlusion is completed, the occlusion area needs to be calculated, and the calculation formula is as follows:

[0095] A covered = overlap area (B target , Merase )

[0096] where B is the bounding box of the target, M is the boundary of the random erasing region, and overlap_area is a function that calculates the overlapping area of the bounding box and the erasing region. target erase

[0097] S3. Construct a traditional image transformation-based forward-looking sonar image data augmentation framework to simulate light, noise, and viewing angle changes in different environments and generate diverse training data.

[0098] The traditional image transformation-based forward-looking sonar image data augmentation framework is shown in FIG. 3. The image enhancement methods include: Figure 3

[0099] S3.1 Horizontal Flip: Flip the image left and right with a probability of 50%. The formula is as follows:

[0100] I flipped (x,y)=I(W-x,y)

[0101] where I(x,y) is the pixel value of the original image, W is the width of the image, and I flipped (x,y) is the pixel value of the flipped image. Horizontal flipping can effectively enhance the model's ability to handle viewing angle changes, especially in underwater environments where targets may be observed at different angles.

[0102] S3.2 Rotate: Randomly rotate the image within the range of -45° to +45°. The rotation transformation formula is as follows:

[0103] I rotated (x′,y′)=I(xcosθ-ysinθ,xsinθ+ycosθ

[0104] where θ is the rotation angle, (x,y) is the pixel coordinate of the original image, and (x′,y′) is the pixel coordinate of the new position after rotation. By increasing the viewing angle, the model can learn to detect targets at different angles, thereby improving its recognition ability for rotated objects.

[0105] S3.3 RandomBrightnessContrast: Randomly adjust the brightness and contrast of the image. The formula is as follows:

[0106] Iadjusted(x,y)-α·I(x,y)+β

[0107] ​​​where a is the contrast adjustment coefficient, b is the brightness offset, and I(x, y) is the original image pixel value. By adjusting the brightness and contrast, the model can adapt to different lighting conditions and improve performance in non-uniformly illuminated scenes.

[0108] S3.4 Gauss Noise (GaussNoise): Add Gaussian noise to the image, represented by the formula:

[0109] I noisy (x, y) - I(x, y) + N(0, σ 2 )

[0110] where N(0, σ 2 ) is Gaussian noise with mean 0 and variance σ 2 . By introducing noise enhancement, the model can learn to accurately detect targets under noise interference.

[0111] S3.5 Blur Filter (Blur, MotionBlur, GaussianBlur): Use a blur filter to blur the image, and the common Gaussian blur formula is:

[0112]

[0113] where σ is the standard deviation of the Gaussian distribution, and I(x+i, y+j) is the pixel value in the image centered at (x, y) with a distance of i, j. By blurring, the model can adapt to motion blur or image blur caused by environmental changes

[0114] S3.6 Smoke Augmentation (SmokeAugmentation): Use Gaussian blur or motion blur to simulate underwater smoke or environmental blur effects, similar to the Gaussian blur formula. Smoke augmentation can simulate low-visibility underwater scenes and enhance the model's detection ability in complex underwater environments.

[0115] S3.7 Image Fusion (1mageCompression): Image compression often uses JPEG compression algorithm, and its core operation includes Discrete Cosine Transform (DCT):

[0116]

[0117] where I(x, y) is the original image pixel value, F(u, v) is the DCT transformed coefficient, and N is the size of the image block. Image compression simulates low-quality image conditions, enabling the model to adapt to compressed or low-resolution image detection tasks.

[0118] In this step of the embodiment, the images are first randomly combined by respective application probabilities, which are: horizontal flip 0.5, rotation 0.5, brightness and contrast adjustment 0.2, Gaussian noise 0.5, smoothing filter 0.5, smoke augmentation 0.2, and image compression 0.3. During the execution of the image transformation: for horizontal flip, the image has a 50% probability of being flipped left and right; for rotation, the rotation angle is randomly selected between -45° and +45°; the brightness adjustment range of the brightness and contrast adjustment is set to -0.2 to +0.2, and the contrast adjustment coefficient range is 0.8 to 1.2; the variance range of the Gaussian noise is between 10 and 50; for smoothing filter, the blur degree is 7, and 50% probability is applied; the smoke augmentation uses Gaussian blur or motion blur combination, and the blur intensity is set to 5; the compression quality of the image compression is set to 60% to 90%; after completing these image transformations, the boundary of the target needs to be checked to ensure that the coordinates of the bounding box are always within the image range, and the boundary check formula is as follows:

[0119] x min =max(0,x min ),y min =max(0,y min )

[0120] x max =min(W,x max ),y max =min(H,y max )

[0121] where W and H are the width and height of the image, respectively, to ensure that the bounding box does not exceed the image range.

[0122] At the same time, a visibility check is performed to ensure that the visible area of the target is not less than 50%. The visibility calculation formula is as follows:

[0123]

[0124] where A visible is the visible area of the target, and A target is the original area of the target.

[0125] Finally, all images, occlusion masks, and YOLO format labels are uniformly adjusted to 640x640 pixels in size to ensure that they are suitable for the input format of the fast Fourier convolution network.

[0126] S4, construct an image enhancement network based on fast Fourier convolution (FFC), use a global receptive field combined with local convolution and global Fourier transform branch to improve the processing and image repair capability of the occlusion region, and generate higher quality enhanced images;

[0127] In this step, the image enhancement network of fast Fourier convolution is as shown in Figure 4 Specifically, it includes:

[0128] S4.1 The input image is converted to the frequency domain at an early stage of the network by introducing the Fourier transform, and the receptive field is expanded to cover the entire image. FFC combines local convolution and global Fourier transform two branches, effectively fusing local details and global context information in the entire image processing process. The two processing branches are as follows:

[0129] (1) Local branch: use traditional convolution (such as 3x3 convolution kernel) to process the local detail information of the image, ensure the integrity of the details.

[0130] (2) Global branch: convert the image to the frequency domain through Fourier transform, capture global information, and restore the overall structure of the image.

[0131] The outputs of the two branches are finally combined through a fusion operation to achieve effective combination of global perception and local details, and improve the processing and repair ability of the occluded area. The specific Fourier transform is as follows:

[0132] S4.2 First, the input image is subjected to two-dimensional Fourier transform, which is converted from spatial domain to frequency domain representation. The input image has a dimension of HxWxC, and after Fourier transform, a complex frequency domain representation is obtained:

[0133]

[0134] After Fourier transform, a complex frequency domain representation is obtained Separate the complex part into real and imaginary parts to facilitate subsequent convolution operations:

[0135]

[0136] Here, the channel number of the frequency domain data is expanded to 2C to include the components of the real and imaginary parts. Then, in the domain, the convolution operation is performed, combined with the ReLU activation function, BatchNormalization (BN) and 1x1 convolution, to process the separated real components, enhance global information processing, and further enhance the understanding of the occluded area:

[0137]

[0138] After frequency domain processing, the data is returned to the spatial domain through inverse Fourier transform to restore the spatial representation of the image:

[0139]

[0140] The processed global information is combined with the local detail information to form an enhanced image, which can effectively process the image inpainting problem of large-area occlusion, and especially exhibits good recovery effect under complex structures and patterns. The image enhancement task can be provided with higher quality and more detailed pictures, and the generalization ability of the network in various complex environments is further improved.

[0141] In this embodiment, an image of 640x640 pixels is input into the network together with a mask. In the global branch, the image is first converted from the spatial domain to the frequency domain by using a 2D fast Fourier transform, and the initial number of channels is 256. Subsequently, the number of channels is reduced to 64 by using a 1x1 convolution, and a ReLU activation function is used. After the convolution operation is completed, the features are converted back to the spatial domain using an inverse fast Fourier transform. At the same time, the local branch uses a 3x3 convolution kernel for operation, and the number of channels is set to 128. After each layer of convolution, a batch normalization process is performed, and a ReLU activation function is used. The local branch is mainly responsible for extracting the local features of the image to ensure the accuracy of the details of the non-mask area. After the feature extraction is completed, the features of the global branch and the local branch are spliced in the channel dimension. The fused features are further adjusted to 64 channels by using a 1x1 convolution layer.

[0142] The residual module of the network is composed of multiple stages. In stage 1, the input image is subjected to a convolution layer (kernel=7, step=2), a batch normalization layer (BN), a ReLU activation function, and a maximum pooling layer (kernel=3, step=2). The subsequent three stages respectively contain 3, 4, and 6 bottleneck structures, each of which is connected by convolution and adopts a residual strategy, and the number of input and output channels remains the same.

[0143] After the feature processing is completed, the network performs upsampling by using a bilinear interpolation to enlarge the feature map to a resolution of 640x640. Finally, a 3x3 convolution layer is used to output the enhanced image, and the number of channels is 3 to adapt to the output of the RGB image. The output layer is subjected to a sigmoid activation function to ensure that the pixel value is between [0, 1], and an enhanced image is generated.

[0144] During the training process, the loss function includes a high perceptual field perceptual loss (HRFPL) for capturing global context information to ensure that the image structure is consistent with the real image. In addition, an adversarial loss is used to improve the detail performance, and an L1 / L2 loss is combined to further reduce the pixel difference between the generated image and the real image, so that the repair result is more natural.

[0145] S5, using the constructed model and framework to enhance the image without occlusion to obtain a clear and detailed image with occlusion.

[0146] In this step, the execution flow is as shown in Figure 1

[0147] S5.1 The original image is enhanced using a multi-modal enhancement framework and a traditional image transformation framework, forming multiple versions of image data. Each version of the image simulates different scene changes and occlusion conditions, and further uses Fast Fourier Convolution (FFC) to refine the enhanced image, ensuring global consistency and preservation of local details.

[0148] S5.2 The generated enhanced dataset includes three types of images: (1) original images as the contrast base data; (2) enhanced images after multi-modal and traditional image transformation, simulating changes in lighting, viewing angle, occlusion, etc. in complex environments; (3) refined images after further processing by Fast Fourier Convolution (FFC), containing clear global information and more detailed information.

[0149] S5.3 In the entire enhanced dataset, the three types of images occupy different proportions, with original images accounting for 1 / 10, multi-modal and traditional image transformation enhanced images accounting for 5 / 10, and Fast Fourier Convolution processed images accounting for 4 / 10, ensuring the diversity and richness of the enhanced dataset, which can provide strong support for subsequent model training.

[0150] Taking the long-range detection of an underwater autonomous underwater vehicle in a self-made sonar dataset as Example 1, the original image detected by the M750d type forward-looking active sonar is as shown in Figure 5 , and the combined image is in turn the underwater autonomous underwater vehicle Figure 5 (left), the water bucket Figure 5 (middle bucket), and the fishing net Figure 5 (right) image, which can be observed in near distance to observe the target contour and details more completely. The present application enhances the multi-modal enhancement framework and the traditional image transformation framework of the near distance observation of the underwater autonomous underwater vehicle Figure 5 (left), water bucket Figure 5 (middle), and fishing net Figure 5 (right) as the example, and the enhancement result is as shown in Figure 6 It can be seen that the present application can simulate and generate scenes under complex occlusion and variable environments, while preserving the contours and details of the near distance targets. The present application further enhances the Fast Fourier Convolution network of the near distance observation of the underwater autonomous underwater vehicle Figure 6 (left), water bucket Figure 6 (middle), and fishing net Figure 6 (right) as the example, and the further enhancement result is as shown in Figure 7 ​As shown, it can be seen that the application can simulate higher quality and more detailed images for close-range targets, thereby further improving the generalization ability of the network in various complex environments.

[0151] The application significantly improves the image enhancement quality and efficiency of the forward-looking sonar severely occluded target by combining the data enhancement method of multi-modal occlusion mode and the network based on fast Fourier convolution, and has good generalization ability and practical value.

[0152] The above only describes the preferred embodiments of the present application, and it should be pointed out that for ordinary skilled persons in the art, several improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. An image enhancement method based on multimodal enhanced mesh masking and fast Fourier convolution, characterized in that: Includes the following steps: S1. Collect forward-looking sonar data of typical underwater targets, extract and transform the sonar metadata, acquire sonar images frame by frame, organize and label the dataset, and create a sonar image recognition training and testing dataset. S2. Construct a forward-looking sonar image data augmentation framework with multimodal occlusion, simulate complex occlusion situations, identify and augment images in the training and test datasets, and generate more challenging training data and corresponding occlusion masks. S3. Construct a forward-looking sonar image data enhancement framework based on traditional image transformation, simulate changes in illumination, noise and viewpoint under different environments, enhance the data images obtained in step S2, and generate diverse data. S4. Construct an image enhancement network based on fast Fourier convolution, using the global receptive field combined with local convolution and global Fourier transform branches to enhance the diverse data generated in step S3, and generate higher quality enhanced images. The forward-looking sonar image data enhancement framework for multimodal occlusion in step S2 includes a grid occlusion module, a random erasure module, and a coarse discard module to simulate different types of occlusion in the underwater environment. The grid occlusion module covers up some pixels by generating alternating grid-like occlusion patterns on the image, as expressed by the following formula: Where d is the size of the grid, representing the width and height of the grid; (x, y) are the pixel coordinates in the image, and grid occlusion generates a binary mask M. grid The occlusion result of an image after applying a grid mask, by covering alternating grid cells, is represented as follows: I masked (x,y)=I(x,y)·M grid (x,y) Where I(x,y) is the original image pixel value, I masked (x,y) represents the occluded image; The random erasure module simulates random noise or irregular occlusion by randomly selecting a rectangular region in the image and setting its pixel value to zero, as expressed by the following formula: R erase =(x1,y1,x2,y2) Where (x1, y1) and (x2, y2) represent the coordinates of the top-left and bottom-right corners of a randomly selected rectangular region, respectively. The image after random erasure is represented as follows: That is, the pixel values ​​located in randomly selected areas are set to zero, while the pixel values ​​of the original image are retained in other areas; The coarse discarding module simulates object occlusion in the environment by generating multiple discontinuous rectangular holes in the image. These multiple discontinuous rectangular holes are described by the following formula: Where n is the number of randomly generated discard regions. and The coordinates of the top-left and bottom-right corners of the i-th rectangular region are represented as follows:

2. The image enhancement method based on multimodal enhanced mesh masking and fast Fourier convolution according to claim 1, characterized in that: Step S1 specifically includes: S1.1, Use forward-looking sonar to collect underwater sonar data of typical targets; S1.2 Extract and transform the underwater sonar metadata, acquire sonar images frame by frame, scale all sonar images to a resolution of 1280×720 pixels, and annotate them as required to produce annotated sonar image data. S1.3, Process the sonar image data obtained in step S1.2, extract the target region in the image, generate the corresponding bounding box annotation, convert the bounding box annotation into YOLO format, and use it as the dataset for target detection during network training.

3. The image enhancement method based on multimodal enhanced mesh masking and fast Fourier convolution according to claim 2, characterized in that: The typical targets include cone-shaped targets, mesh-like targets, and irregular targets.

4. The image enhancement method based on multimodal enhanced mesh masking and fast Fourier convolution according to claim 1, characterized in that: The forward-looking sonar image data augmentation framework in step S3 specifically includes the following aspects for data image enhancement: S3.1, Horizontal Rotation: The image is flipped horizontally with a 50% probability, as shown in the following formula: I flipped (x,y)=I(W-x,y) Where I(x,y) is the pixel value of the original image, W is the width of the image, and I flipped (x,y) represents the pixel values ​​of the flipped image; S3.2, Rotation: Randomly rotate the image within the range of -45° to +45°, as follows: I rotated (x′,y′)=I(xcosθ-ysinθ,xsinθ+ycosθ) Where θ is the rotation angle, (x,y) are the pixel coordinates of the original image, and (x',y') are the pixel coordinates of the new position after rotation; S3.3, Brightness and Contrast Adjustment: Randomly adjusts the brightness and contrast of the image, using the following formula: I adjusted (x,y)=α·I(x,y)+β Where α is the contrast adjustment coefficient, β is the brightness offset value, and I(x,y) is the original image pixel value; S3.4, Add Gaussian noise to the image as follows: I noisy (x,y)=I(x,y)+N(0,σ 2 ) Where N(0,σ) 2 () is a variable with a mean of 0 and a variance of σ. 2 Gaussian noise; S3.5, using a smoothing filter to blur the image, is as follows: Where σ is the standard deviation of the Gaussian distribution, I(x+i) * ,y+j * ) is a region in the image centered at (x,y) and at a distance of i. * ,j * Pixel values; S3.6, use Gaussian blur or motion blur to simulate underwater smoke or environmental blur effects; S3.7, Image Fusion: Including Discrete Cosine Transform: Where I(x,y) is the original image pixel value, F(u,v) is the coefficient after DCT transformation, and N is the size of the image patch.

5. The image enhancement method based on multimodal enhanced mesh masking and fast Fourier convolution according to claim 1, characterized in that: The image enhancement network based on fast Fourier convolution in step S4 includes a local branch, a global branch, and a branch fusion module. The local branch uses convolution operations to process local details of the image; the global branch transforms the image to the frequency domain through Fourier transform to capture global information; and the branch fusion module is used to fuse the outputs of the two branches together.

6. The image enhancement method based on multimodal enhanced mesh masking and fast Fourier convolution according to claim 5, characterized in that: The global branch transforms the image to the frequency domain via Fourier transform, specifically including: The input image has dimensions H×W×C. After Fourier transform, a complex frequency domain representation is obtained: Complex numbers are separated into real and imaginary parts, and then convolution operations are performed in the domain. The separated real components are processed by combining ReLU activation function, batch normalization layer and 1×1 convolution to enhance global information processing. The data, after being processed in the frequency domain, is returned to the spatial domain through the inverse Fourier transform.

Images