Image enhancement method based on multi-modal enhanced grid masking and fast Fourier convolution

Abstract

The invention discloses an image enhancement method based on multi-modal enhanced grid masking and fast Fourier convolution, and the method comprises the steps: S1, collecting the foresight sonar data of an underwater typical target, and making an initial data set; s2, performing data enhancement in a multi-mode shielding mode on the original foresight sonar image; s3, performing data enhancement of a traditional image transformation mode on the multi-mode shielding frame enhanced image, simulating changes of illumination, noise, visual angles and the like in different environments, and generating diversified training data; s4, constructing an image enhancement network based on fast Fourier convolution, and improving the processing and enhancement capability of the occlusion region by using a global receptive field; and S5, enhancing the image by using the trained model to obtain a clear target occlusion image with rich details. According to the method, through the data enhancement method in combination with the multi-mode shielding mode and the network based on the fast Fourier convolution, the image enhancement quality and efficiency of the severely shielded target of the forward-looking sonar are remarkably improved, and the method has good generalization ability and practical value.

Description

Technical Field

[0001] The invention relates to the field of sonar image processing, and in particular to a sonar image enhancement method based on multi-modal enhanced grid masking and fast Fourier convolution. Background Art

[0002] Sonar technology is widely used in ocean detection, target identification and underwater exploration, especially in submarine monitoring, seabed resource management and navigation of underwater unmanned systems. Sonar imaging technology is an important tool for underwater target detection. However, because sonar imaging is affected by the complexity of the underwater environment and the characteristics of sound wave propagation, it often has problems such as high noise, low resolution and unstable imaging, which brings great challenges to the detection and tracking of underwater targets.

[0003] In complex underwater environments, factors such as occlusion, illumination changes, and noise interference further exacerbate the degradation of image quality, significantly reducing the detection accuracy and recognition efficiency of targets. Traditional sonar image processing technology often has limited effects in dealing with these problems, especially when dealing with large-area occlusion and partial loss of targets. Existing methods cannot effectively restore the global consistency and local details of the image. Therefore, how to improve the clarity and detection accuracy of sonar images through more advanced image enhancement methods has become an important direction of current research. Summary of the invention

[0004] The purpose of the present invention is to propose an image enhancement method based on multimodal enhanced grid masking and fast Fourier convolution. By introducing multimodal data enhancement and frequency domain processing technology, the occlusion and noise conditions in complex environments are simulated, and the global perception and detail processing capabilities of the image are effectively improved, thereby significantly improving the accuracy and robustness of underwater target detection, and being used for target detection and recognition in complex underwater environments.

[0005] The technical solution to achieve the purpose of the present invention is:

[0006] An image enhancement method based on multimodal enhanced grid masking and fast Fourier convolution comprises the following steps:

[0007] S1. Collect forward-looking sonar data of typical underwater targets, extract sonar metadata and convert coordinate axes, obtain sonar images frame by frame, organize and annotate data sets, and produce sonar image recognition training and test data sets;

[0008] S2. Build a multi-modal occluded forward-looking sonar image data enhancement framework to simulate complex occlusion situations, identify images in the training and test datasets for enhancement, and generate more challenging training data and corresponding occlusion masks;

[0009] S3, constructing a forward-looking sonar image data enhancement framework based on traditional image transformation, simulating illumination, noise and viewing angle changes in different environments, enhancing the data image obtained in step S2, and generating diversified data;

[0010] S4. Construct an image enhancement network based on fast Fourier convolution, use the global receptive field combined with local convolution and global Fourier transform branches to enhance the diverse data generated in step S3 and generate higher quality enhanced images.

[0011] Furthermore, the step S1 specifically includes:

[0012] S1.1, collect underwater sonar data of typical targets using forward-looking sonar;

[0013] S1.2, extract underwater sonar metadata and transform coordinate axes, obtain sonar images frame by frame, scale all sonar images to a resolution of 1280×720 pixels, and annotate them as required to produce annotated sonar image data;

[0014] S1.3, processing the sonar image data obtained in step S1.2, extracting the target area in the image, generating corresponding bounding box annotations, converting the bounding box annotations into YOLO format, and using them as a data set for target detection during network training.

[0015] Furthermore, the typical targets include cone-shaped targets, mesh-shaped targets and irregular targets.

[0016] Furthermore, the multi-modal occluded forward-looking sonar image data enhancement framework in step S2 includes a grid occlusion module, a random erasing module and a rough discarding module to simulate different types of occlusion situations in an underwater environment.

[0017] Furthermore, the grid occlusion module masks some pixels by generating an alternating grid-like occlusion pattern on the image, and the formula is as follows:

[0018]

[0019] Where d is the size of the grid, indicating the width and height of the grid; (x, y) is the pixel coordinate in the image, and the grid occlusion generates a binary mask M grid , masking alternating grid cells. After applying the grid mask, the occlusion result of the image is expressed as:

[0020] I masked (x,y)=I(x,y)·M grid (x,y)

[0021] Among them, I(x,y) is the original image pixel value, I masked(x,y) is the image after occlusion.

[0022] Furthermore, the random erasing module simulates random noise or irregular occlusion by randomly selecting a rectangular area in the image and setting its pixel value to zero. The formula is as follows:

[0023] R erase =(x 1 ,y 1 ,x 2 ,y 2 )

[0024] Among them, (x 1 ,y 1 ) and (x 2 ,y 2 ) represent the coordinates of the upper left corner and lower right corner of the randomly selected rectangular area, and the image after random erasure is expressed as:

[0025]

[0026] That is, the pixel values ​​within the randomly selected area are set to zero, and the original image pixel values ​​are retained in other areas.

[0027] Furthermore, the rough discarding module simulates object occlusion in the environment by generating a plurality of discontinuous rectangular holes in the image, and the plurality of discontinuous rectangular holes are described by the following formula:

[0028]

[0029] Where n is the number of randomly generated drop regions, and Represents the coordinates of the upper left corner and lower right corner of the i-th rectangular area, and the occlusion result is expressed as:

[0030]

[0031] Furthermore, in step S3, the forward-looking sonar image data enhancement framework specifically includes:

[0032] S3.1, horizontal rotation: The image is flipped left and right with a probability of 50%. The formula is as follows:

[0033] I flipped (x,y)=I(Wx,y)

[0034] Among them, I(x,y) is the pixel value of the original image, W is the width of the image, and I flipped (x, y) is the pixel value of the flipped image;

[0035] S3.2, Rotation: Randomly rotate the image in the range of -45° to +45°, as:

[0036] I rotated (x′,y′)=I(xcosθ-ysinθ,xsinθ+ycosθ

[0037] Where θ is the rotation angle, (x, y) is the pixel coordinate of the original image, and (x′, y′) is the image coordinate of the new position after rotation;

[0038] S3.3, brightness and contrast adjustment: Randomly adjust the brightness and contrast of the image, the formula is as follows:

[0039] I adjusted (x,y)-α·I(x,y)+β

[0040] Among them, α is the contrast adjustment coefficient, β is the brightness offset, and I(x,y) is the original image pixel value;

[0041] S3.4, add Gaussian noise to the image, as:

[0042] I noisy (x,y)-I(x,y)+N(0,σ 2 )

[0043] Among them, N(0,σ 2 ) has a mean of 0 and a variance of σ 2 Gaussian noise;

[0044] S3.5, blurring the image using a smoothing filter, is:

[0045]

[0046] Where σ is the standard deviation of the Gaussian distribution, I(x+i,y+j) is the image index value centered at (x,y) and at distance i,j in the image;

[0047] S3.6, use Gaussian blur or motion blur to simulate underwater smoke or environmental blur effects;

[0048] S3.7, Image Fusion: Including Discrete Cosine Transform:

[0049]

[0050] Among them, I(x, y) is the original image pixel value, F(u, v) is the coefficient after DCT transformation, and N is the size of the image block.

[0051] Furthermore, the image enhancement network based on fast Fourier convolution in step S4 includes a local branch, a global branch and a branch fusion module. The local branch uses a convolution operation to process local detail information of the image; the global branch converts the image to the frequency domain through Fourier transform to capture global information; and the branch fusion module is used to fuse the outputs of the two branches together.

[0052] Furthermore, the global branch converts the image into the frequency domain by Fourier transform, specifically comprising:

[0053] The input image dimension is H×W×C. After Fourier transform, the complex frequency domain representation is obtained:

[0054]

[0055] The complex number is separated into real and imaginary parts, and then a convolution operation is performed in the domain. The separated real components are processed by combining the ReLU activation function, batch normalization layer and 1×1 convolution to enhance global information processing;

[0056] The data processed in the frequency domain is returned to the spatial domain through inverse Fourier transform.

[0057] Compared with the prior art, the present invention has the following significant advantages:

[0058] 1. The multimodal occlusion enhancement framework can effectively simulate the occlusion in complex underwater environments, improve the adaptability of the model under complex occlusion conditions, and effectively improve the robustness of the model;

[0059] 2. Considering the impact of different underwater environmental lighting, noise and perspective changes, a traditional image transformation framework is added to enhance the model's ability to cope with changing environments, improve image diversity, and effectively enhance the model's generalization ability;

[0060] 3. Traditional convolutional networks have the disadvantage of limited local receptive field, and it is difficult to ensure global consistency when dealing with large-area occlusion and complex structures. This invention introduces fast Fourier convolution (FFC) technology, combines global Fourier transform with local convolution, and can preserve image details while ensuring global consistency. It is particularly suitable for large-area occlusion repair in complex scenes, and greatly improves the accuracy and robustness of image enhancement and repair;

[0061] 4. By combining the comprehensive processing of fast Fourier convolution (FFC), multimodal occlusion and traditional image transformation, the robustness and adaptability of the model in diverse environments are improved, ensuring the diversity of the generated training data in different scenarios, thereby improving the generalization performance of the model and the overall enhancement effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1It is the algorithm flow chart of the present invention;

[0063] Figure 2 This is a flow chart of the multi-modal occlusion structure in the present invention;

[0064] Figure 3 It is a flow chart of the traditional image transformation structure in the present invention;

[0065] Figure 4 It is a flow chart of the Fourier convolution network in the present invention;

[0066] Figure 5 This is a sonar image of an unmanned underwater vehicle according to an embodiment of the present invention;

[0067] Figure 6 This is a diagram after the multi-modal enhancement structure is enhanced according to an embodiment of the present invention;

[0068] Figure 7 This is a diagram after enhancement by the fast Fourier convolutional network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0070] like Figure 1 As shown, the present invention relates to an image enhancement method based on multi-modal enhanced grid masking and fast Fourier convolution, and the implementation process includes the following steps:

[0071] S1. Collect forward-looking sonar data of typical underwater targets, extract sonar metadata and convert coordinate axes, obtain sonar images frame by frame, organize and annotate data sets, and produce sonar image recognition training and test data sets;

[0072] S1.1 Use forward-looking sonar to collect underwater sonar data of typical targets, including cone-shaped targets, mesh targets, irregular targets, etc.;

[0073] S1.2 extracting sonar metadata and converting coordinate axes, acquiring sonar images frame by frame, scaling all sonar images to a resolution of 1280×720 pixels, and annotating them as required, to produce sonar image data dedicated to the present invention;

[0074] S1.3 processes the sonar image data, extracts the target area in the image, and generates the corresponding bounding box annotations. These annotations are converted into YOLO format and used as a dataset for target detection during network training.

[0075] S2. Build a multi-modal occluded forward-looking sonar image data enhancement framework to simulate complex occlusion situations and generate more challenging training data and corresponding occlusion masks.

[0076] Multimodal occluded forward-looking sonar image data enhancement framework Figure 2 As shown in Figure 2, the data enhancement process specifically includes:

[0077] S2.1 mainly uses three customized occlusion methods to perform multimodal occlusion data enhancement on the sonar recognition dataset, namely grid mask, random erasing, and coarse dropout, to simulate different types of occlusion in underwater environments. The specific formula is as follows:

[0078] S2.2 Grid Mask (GridMask) masks some pixels by generating an alternating grid-like masking pattern on the image. The formula is as follows:

[0079]

[0080] Where d is the size of the grid, indicating the width and height of the grid; (x, y) is the pixel coordinate in the image. Grid occlusion generates a binary mask M grid , masking alternating grid cells. After applying the grid mask, the occlusion result of the image can be expressed as:

[0081] I masked (x,y)=I(x,y)·M grid (x,y)

[0082] Where I(x,y) is the original image pixel value, I masked (x, y) is the image after occlusion. Grid occlusion uses a regular occlusion pattern to force the model to learn to extract effective information from the image under partial occlusion during training. It is suitable for situations where there are periodic or regular obstacles in underwater environments, and enhances the robustness of the model in the case of local information loss.

[0083] S2.3 Random Erasing randomly selects a rectangular area in the image and sets its pixel value to zero to simulate random noise or irregular occlusion. The formula is as follows:

[0084] R erase =(x 1 ,y 1 ,x 2 ,y 2 )

[0085] Among them, (x 1 ,y 1 ) and (x 2 ,y 2) represent the coordinates of the upper left corner and lower right corner of the randomly selected rectangular area. The image after random erasure can be expressed as:

[0086]

[0087] That is, the pixel values ​​in the randomly selected area are set to zero, and the original image pixel values ​​are retained in other areas. Random erasing simulates unpredictable interference in real scenes, such as noise or irregular objects in underwater environments, through irregular occlusion. It effectively enhances the model's adaptability when facing sudden or random occlusion, allowing the model to make accurate predictions even with incomplete information.

[0088] S2.4 Coarse Dropout simulates object occlusion in the environment by generating multiple discontinuous rectangular holes in the image. Multiple dropout regions are described by the following formula:

[0089]

[0090] where n is the number of randomly generated drop regions, and Represents the coordinates of the upper left corner and lower right corner of the i-th rectangular area. The occlusion result is expressed as:

[0091]

[0092] Rough discarding can simulate discontinuous occlusion in underwater environments, such as interference caused by floating objects or debris. Through this discontinuous occlusion method, the model can learn to extract key feature information even when a large amount of information is lost, which is especially advantageous in complex and dynamic environments.

[0093] S2.5 In multimodal occlusion data enhancement, the generated image retains important target information and is equipped with a corresponding occlusion mask. The mask is generated based on the occlusion area to ensure that during the model training process, it can learn how to detect targets in complex occlusion situations, and can also be used as the input of the fast Fourier convolutional network for further data enhancement.

[0094] In this step, they are first randomly combined according to their respective application probabilities, namely grid occlusion 0.4, random erasure 0.3, and coarse dropout 0.3. During the occlusion process, the fill value of the occlusion mask area is set to 0. For grid occlusion (GridMask), the number of grids is randomly selected between 3 and 6; the maximum occlusion ratio of random erasure (RandomErasing) is set to 0.1, and the minimum aspect ratio is set to 0.3; and the number of holes for coarse dropout (CoarseDropout) is randomly selected between 1 and 30, and the height and width of the holes are set between 16 and 128. After multimodal occlusion is completed, the occlusion area needs to be calculated. The calculation formula is as follows:

[0095] A covered =overlap area (B target ,M erase )

[0096] Among them: B target is the bounding box of the target, M erase is the boundary of the random erased area, and overlap_area is a function that calculates the overlapping area between the bounding box and the erased area.

[0097] S3, build a forward-looking sonar image data enhancement framework based on traditional image transformation, simulate illumination, noise and viewpoint changes in different environments, and generate diverse training data;

[0098] Forward-looking sonar image data enhancement framework based on traditional image transformation Figure 3 The image enhancement methods include:

[0099] S3.1 Horizontal Flip: Flip the image left and right with a probability of 50%. The formula is as follows:

[0100] I flipped (x,y)=I(Wx,y)

[0101] Among them, I(x,y) is the pixel value of the original image, W is the width of the image, and I flipped (x, y) is the pixel value of the flipped image. Horizontal flipping can effectively enhance the model's ability to cope with changes in viewing angles, especially in underwater environments where targets may be observed from different angles.

[0102] S3.2 Rotate: Randomly rotate the image in the range of -45° to +45°. The rotation transformation formula is:

[0103] I rotated (x′,y′)=I(xcosθ-ysinθ,xsinθ+ycosθ

[0104] Among them, θ is the rotation angle, (x, y) is the pixel coordinate of the original image, and (x′, y′) is the image coordinate of the new position after rotation. By increasing the perspective change, the model can learn to detect objects at different angles, thereby improving its recognition ability for rotated objects.

[0105] S3.3Brightness and contrast adjustment (RandomBrightnessContrast): Randomly adjust the brightness and contrast of the image. The formula is as follows:

[0106] Iadjusted(x,y)-α·I(x,y)+β

[0107] Among them, α is the contrast adjustment coefficient, β is the brightness offset, and I(x,y) is the original image index value. By adjusting the brightness and contrast, the model can adapt to different lighting conditions and improve the performance in scenes with uneven lighting.

[0108] S3.4 Gaussian noise (GaussNoise): Add Gaussian noise to the image. The formula is as follows:

[0109] I noisy (x,y)-I(x,y)+N(0,σ 2 )

[0110] Among them, N(0,σ 2 ) has a mean of 0 and a variance of σ 2 By introducing noise enhancement, the model can learn to perform accurate target detection under noise interference conditions.

[0111] S3.5 smoothing filter (Blur, MotionBlur, GaussianBlur): Use a smoothing filter to blur the image. The common Gaussian blur formula is:

[0112]

[0113] Where σ is the standard deviation of the Gaussian distribution, and I(x+i,y+j) is the image index value centered at (x,y) and at distance i,j in the image. Through blurring, the model can adapt to image blur caused by motion blur or environmental changes.

[0114] S3.6 Smoke Augmentation uses Gaussian blur or motion blur to simulate underwater smoke or environmental blur effects, similar to the formula of Gaussian blur. Fog augmentation can simulate scenes with low underwater visibility and enhance the detection ability of the model in complex underwater environments.

[0115] S3.7 Image Fusion (1mageCompression): Image compression often uses the JPEG compression algorithm, whose core operations include discrete cosine transform (DCT):

[0116]

[0117] Among them, I(x,y) is the original image pixel value, F(u,v) is the coefficient after DCT transformation, and N is the size of the image block. Image compression simulates low-quality image conditions, so that the model can adapt to the detection task of compressed or low-resolution images.

[0118] In this step of the embodiment, firstly, random combinations are made by their respective application probabilities, namely: horizontal flip 0.5, rotation 0.5, brightness and contrast adjustment 0.2, Gaussian noise 0.5, smoothing filter 0.5, smoke augmentation 0.2, image compression 0.3. In the process of performing image transformation: for horizontal flip, the image has a 50% probability of being flipped left and right; for rotation, the rotation angle is randomly selected between -45° and +45°; the brightness adjustment range of brightness and contrast adjustment is set to -0.2 to +0.2, and the contrast adjustment coefficient range is 0.8 to 1.2; the variance range of Gaussian noise is between 10 and 50; for smoothing filter, the blur level is 7, and the probability of application is 50%; smoke augmentation uses Gaussian blur or motion blur combined, and the blur intensity is set to 5; the compression quality of image compression is set to 60% to 90%; after completing these image transformations, the boundary of the target needs to be checked to ensure that the coordinates of the boundary box are always within the image range. The boundary check formula is as follows:

[0119] x min =max(0,x min ),y min =max(0,y min )

[0120] x max =min(W,x max ),y max =min(H,y max )

[0121] Where W and H are the width and height of the image, respectively, to ensure that the bounding box does not exceed the image range.

[0122] At the same time, a visibility check is performed to ensure that the visible area of ​​the target is not less than 50%. The visibility calculation formula is as follows:

[0123]

[0124] Among them, A visible is the visible area of ​​the target, A targetis the original area of ​​the target.

[0125] Finally, all images, occlusion masks, and YOLO format labels are uniformly resized to 640x640 pixels to ensure that they are suitable for the input format of the fast Fourier convolutional network.

[0126] S4. Construct an image enhancement network based on Fast Fourier Convolution (FFC), use the global receptive field combined with local convolution and global Fourier transform branches to improve the processing and image restoration capabilities of occluded areas, and generate higher quality enhanced images;

[0127] In this step, the image enhancement network of fast Fourier convolution is as follows Figure 4 Specifically including:

[0128] S4.1 introduces Fourier transform to convert the input image to the frequency domain in the early stage of the network and expand the receptive field to cover the entire image. FFC combines the two branches of local convolution and global Fourier transform to effectively fuse local details with global context information in the entire image processing process. The two processing branches are as follows:

[0129] (1) Local branch: Use traditional convolution (such as 3×3 convolution kernel) to process local detail information of the image to ensure the integrity of the details.

[0130] (2) Global branch: The image is converted to the frequency domain through Fourier transform to capture global information and restore the overall structure of the image.

[0131] The outputs of these two branches are finally combined through a fusion operation to achieve an effective combination of global perception and local details, improving the processing and repair capabilities of occluded areas. The specific Fourier transform is as follows:

[0132] S4.2 first performs a two-dimensional Fourier transform on the input image to convert it from the spatial domain to the frequency domain. The input image dimension is H×W×C. After the Fourier transform, the complex frequency domain representation is obtained:

[0133]

[0134] After Fourier transform, the complex frequency domain representation is obtained Separate the complex part into real and imaginary parts to facilitate subsequent convolution operations:

[0135]

[0136] Here, the number of channels of the frequency domain data is expanded to 2C to include both real and imaginary components. Then, a convolution operation is performed in the domain, combining the ReLU activation function, BatchNormalization (BN) and 1×1 convolution to process the separated real components, enhance global information processing, and further enhance the understanding of the occluded area:

[0137]

[0138] The data processed in the frequency domain is returned to the spatial domain through inverse Fourier transform to restore the spatial representation of the image:

[0139]

[0140] The processed global information is combined with the local detail information to form an enhanced image, which can effectively handle the image restoration problem of large-area occlusion, especially showing good restoration effects under complex structures and patterns. It can provide higher quality and more detailed pictures for image enhancement tasks, and further improve the generalization ability of the network in various complex environments.

[0141] In this embodiment, an image of 640×640 pixels is input into the network together with a mask. In the global branch, the image is first converted from the spatial domain to the frequency domain using a 2D fast Fourier transform, with an initial number of channels of 256. Subsequently, the number of channels is reduced to 64 by a 1×1 convolution, and the ReLU activation function is used. After the convolution operation is completed, the features are converted back to the spatial domain using an inverse fast Fourier transform. At the same time, the local branch operates with a 3×3 convolution kernel, the number of channels is set to 128, and each layer of convolution is processed by batch normalization, and the ReLU activation function is used. The local branch is mainly responsible for extracting local features of the image to ensure that the details of the non-masked area are accurate. After the feature extraction is completed, the features of the global branch and the local branch are spliced ​​in the channel dimension. The fused features are further adjusted to 64 channels through a 1×1 convolution layer.

[0142] The residual module of the network consists of multiple stages. In stage 1, the input image passes through a convolutional layer (kernel=7, step size=2), a batch normalization layer (BN), a ReLU activation function, and a maximum pooling layer (kernel=3, step size=2). The subsequent three stages contain 3, 4, and 6 bottleneck structures, respectively. Each bottleneck is connected by convolution, using a residual strategy, and the number of input and output channels remains consistent.

[0143] After feature processing, the network upsamples the feature map to a resolution of 640×640 using bilinear interpolation. Finally, a 3×3 convolutional layer is used to output the enhanced image with 3 channels to accommodate RGB image output. The output layer uses a sigmoid activation function to ensure that the pixel values ​​are between [0,1] to generate the enhanced image.

[0144] During the training process, the loss function includes the High Receptive Field Perceptual Loss (HRFPL) to capture global context information and ensure that the image structure is consistent with the real image. In addition, the adversarial loss is used to improve the detail performance, and the L1 / L2 loss is combined to further reduce the pixel difference between the generated image and the real image, making the restoration result more natural.

[0145] S5. Use the constructed model and framework to enhance the unoccluded image to obtain a clear and detailed image with occlusion.

[0146] In this step, the execution process is as follows Figure 1 Shown:

[0147] S5.1 The original image is enhanced using the multimodal enhancement framework and the traditional image transformation framework to form multiple versions of image data. Each version of the image simulates different scene changes and occlusion conditions, and the enhanced image is further refined using Fast Fourier Convolution (FFC) to ensure the global consistency of the image and the retention of local details.

[0148] The enhanced dataset generated by S5.2 includes three types of images: (1) original images, which serve as basic data for comparison; (2) enhanced images that have undergone multimodal and traditional image transformations to simulate changes in illumination, perspective, occlusion, etc. in complex environments; and (3) refined images that have been further processed by fast Fourier convolution (FFC), which contain clear global information and richer details.

[0149] S5.3 In the entire enhanced dataset, these three types of images occupy different proportions, among which the original images account for 1 / 10, the images enhanced by multimodal and traditional image transformation account for 5 / 10, and the images processed by fast Fourier convolution account for 4 / 10, ensuring the diversity and richness of the enhanced dataset, which can provide strong support for subsequent model training.

[0150] Taking the underwater unmanned submersible long-distance detection in the self-made sonar data set as Example 1, the original image obtained by the M750d model forward-looking active sonar detection is as follows Figure 5 As shown, the combined figure is the underwater unmanned submersible ( Figure 5 Left), Bucket( Figure 5Medium barrel), fishing net( Figure 5 Right) image, the target outline and details can be observed more completely at close range. Figure 5 Left), Bucket( Figure 5 Middle), Fishing Net( Figure 5 Right) The results of the multi-modal enhancement framework and the traditional image transformation framework for close-range observation are shown in Figure 2. Figure 6 As shown in the figure, it can be seen that the present invention can simulate and generate scenes under complex occlusion and changing environments, while retaining the outline and details of close-range targets. Figure 6 Left), Bucket( Figure 6 Middle), Fishing Net( Figure 6 Right) Close-up observation of the fast Fourier convolutional network for further enhancement results as shown in Figure 7 As shown, it can be seen that the present invention can simulate and generate higher quality and more detailed images for close-range targets, thereby further improving the generalization ability of the network in various complex environments.

[0151] The present invention significantly improves the image enhancement quality and efficiency of forward-looking sonar severely occluded targets by combining a multi-modal occlusion data enhancement method with a network based on fast Fourier convolution, and has good generalization ability and practical value.

[0152] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An image enhancement method based on multimodal enhanced grid masking and fast Fourier convolution, characterized in that: The following steps are involved: S1. Collect forward-looking sonar data of typical underwater targets, extract sonar metadata and convert coordinate axes, obtain sonar images frame by frame, organize and annotate data sets, and produce sonar image recognition training and test data sets; S2. Build a multi-modal occluded forward-looking sonar image data enhancement framework to simulate complex occlusion situations, identify images in the training and test datasets for enhancement, and generate more challenging training data and corresponding occlusion masks; S3, constructing a forward-looking sonar image data enhancement framework based on traditional image transformation, simulating illumination, noise and viewing angle changes in different environments, enhancing the data image obtained in step S2, and generating diversified data; S4. Construct an image enhancement network based on fast Fourier convolution, use the global receptive field combined with local convolution and global Fourier transform branches to enhance the diverse data generated in step S3 and generate higher quality enhanced images.

2. The image enhancement method based on multimodal enhanced grid masking and fast Fourier convolution according to claim 1, characterized in that: The step S1 specifically includes: S1.1, collect underwater sonar data of typical targets using forward-looking sonar; S1.2, extract underwater sonar metadata and transform coordinate axes, obtain sonar images frame by frame, scale all sonar images to a resolution of 1280×720 pixels, and annotate them as required to produce annotated sonar image data; S1.3, processing the sonar image data obtained in step S1.2, extracting the target area in the image, generating corresponding bounding box annotations, converting the bounding box annotations into YOLO format, and using them as a data set for target detection during network training.

3. The image enhancement method based on multimodal enhanced grid masking and fast Fourier convolution according to claim 2, characterized in that: The typical targets include cone-shaped targets, mesh-shaped targets and irregular targets.

4. The image enhancement method based on multimodal enhanced grid masking and fast Fourier convolution according to claim 1, characterized in that: The multi-modal occluded forward-looking sonar image data enhancement framework in step S2 includes a grid occlusion module, a random erasing module and a rough discarding module to simulate different types of occlusion situations in an underwater environment.

5. The image enhancement method based on multimodal enhanced grid masking and fast Fourier convolution according to claim 4, characterized in that: The grid occlusion module masks some pixels by generating an alternating grid-like occlusion pattern on the image. The formula is as follows: Where d is the size of the grid, indicating the width and height of the grid; (x, y) is the pixel coordinate in the image, and the grid occlusion generates a binary mask M grid , masking alternating grid cells. After applying the grid mask, the occlusion result of the image is expressed as: I masked (x,y)=I(x,y)·M grid (x,y) Among them, I(x,y) is the original image pixel value, I masked (x,y) is the image after occlusion.

6. The image enhancement method based on multimodal enhanced grid masking and fast Fourier convolution according to claim 4, characterized in that: The random erasing module simulates random noise or irregular occlusion by randomly selecting a rectangular area in the image and setting its pixel value to zero. The formula is as follows: R erase =(x1,y1,x2,y2) Among them, (x1, y1) and (x2, y2) represent the coordinates of the upper left corner and lower right corner of the randomly selected rectangular area, respectively. The image after random erasure is expressed as: That is, the pixel values ​​within the randomly selected area are set to zero, and the original image pixel values ​​are retained in other areas.

7. The image enhancement method based on multimodal enhanced grid masking and fast Fourier convolution according to claim 4, characterized in that: The rough discarding module simulates the occlusion of objects in the environment by generating multiple discontinuous rectangular holes in the image. The multiple discontinuous rectangular holes are described by the following formula: Where n is the number of randomly generated drop regions, and Represents the coordinates of the upper left corner and lower right corner of the i-th rectangular area, and the occlusion result is expressed as:

8. The image enhancement method based on multi-modal enhanced grid masking and fast Fourier convolution according to claim 1, characterized in that: The forward-looking sonar image data enhancement framework in step S3 specifically includes: S3.1, horizontal rotation: The image is flipped left and right with a probability of 50%. The formula is as follows: I flipped (x,y)=I(W-x,y) Among them, I(x,y) is the pixel value of the original image, W is the width of the image, and I flipped (x, y) is the pixel value of the flipped image; S3.2, Rotation: Randomly rotate the image in the range of -45° to +45°, as: I rotated (x′,y′)=I(xcosθ-ysinθ,xsinθ+ycosθ Where θ is the rotation angle, (x, y) is the pixel coordinate of the original image, and (x′, Y′) is the image coordinate of the new position after rotation; S3.3, brightness and contrast adjustment: Randomly adjust the brightness and contrast of the image, the formula is as follows: I adjusted (x,y)-α·I(x,y)+β Among them, α is the contrast adjustment coefficient, β is the brightness offset, and I(x,y) is the original image pixel value; S3.4, add Gaussian noise to the image, as: I noisy (x,y)-I(x,y)+N(0,σ 2 ) Among them, N(0,σ 2 ) has a mean of 0 and a variance of σ 2 Gaussian noise; S3.5, blurring the image using a smoothing filter, is: Where σ is the standard deviation of the Gaussian distribution, I(x+i,y+j) is the image index value centered at (x,y) and at distance i,j in the image; S3.6, use Gaussian blur or motion blur to simulate underwater smoke or environmental blur effects; S3.7, Image Fusion: Including Discrete Cosine Transform: Among them, I(x, y) is the original image pixel value, F(u, v) is the coefficient after DCT transformation, and N is the size of the image block.

9. The image enhancement method based on multimodal enhanced grid masking and fast Fourier convolution according to claim 1, characterized in that: The image enhancement network based on fast Fourier convolution in step S4 includes a local branch, a global branch and a branch fusion module. The local branch uses a convolution operation to process local detail information of the image; the global branch converts the image to the frequency domain through Fourier transform to capture global information; the branch fusion module is used to fuse the outputs of the two branches together.

10. The image enhancement method based on multi-modal enhanced grid masking and fast Fourier convolution according to claim 9, characterized in that: The global branch converts the image into the frequency domain by Fourier transform, specifically comprising: The input image dimension is H×W×C. After Fourier transform, the complex frequency domain representation is obtained: The complex number is separated into real and imaginary parts, and then a convolution operation is performed in the domain. The separated real components are processed by combining the ReLU activation function, batch normalization layer and 1×1 convolution to enhance global information processing; The data processed in the frequency domain is returned to the spatial domain through inverse Fourier transform.

Images