Underwater robot vision sharpening method based on multi-modal fusion network

Abstract

The invention discloses an underwater robot vision sharpening method based on a multi-modal fusion network, and belongs to the technical field of underwater image processing, and the method mainly comprises the steps: collecting a turbid underwater image and a corresponding clear underwater image, and constructing an underwater polarization image data set which comprises underwater polarization images, polarization degree images and polarization angle images of different angles; constructing an underwater robot vision clearness model based on the multi-modal fusion network, wherein the underwater robot vision clearness model comprises the multi-modal fusion network and an image enhancement network; training the multi-modal fusion network based on an underwater polarization image data set, updating RGB information and polarization information by using pixel multi-scale fusion during training, and generating fusion features; training the image enhancement network by using an underwater polarization image data set to obtain an image enhancement-based model; and obtaining a to-be-processed fusion feature with turbidity, and inputting the fusion feature into the network-based image enhancement model, thereby obtaining a clear underwater image.

Description

Technical Field

[0001] The present invention relates to the fields of marine environment perception, digital image processing and image enhancement technology, and in particular to a method for underwater robot vision clarity based on a multimodal fusion network. Background Art

[0002] With the development of human exploration of the ocean, underwater robots have become an important tool for obtaining seabed information. However, due to the strong absorption and scattering of light by water and the complex underwater environment, the underwater environment has energy attenuation due to the propagation of light underwater, and impurities and suspended particles in the water will cause light to scatter during propagation, making the images collected underwater more blurred than those collected on land. These problems will cause color deviation and low clarity in the obtained images, which will seriously affect the visual quality and the performance of downstream underwater visual tasks. Underwater optical imaging is currently the core way to complete underwater environment perception and detection tasks. It plays an indispensable role in scientific research fields such as underwater robots, marine scientific expeditions, and some downstream visual tasks (underwater target recognition or tracking). Due to the complexity and instability of the underwater environment, underwater images often face problems such as color deviation, low contrast and blur. Specifically, when light propagates in water, it is first affected by the depth of the water. As the depth increases, light of different wavelengths will continue to attenuate. Therefore, since red light disappears first, underwater images are often dominated by blue or green, resulting in image color deviation. At the same time, unlike the air medium on land, water contains a large number of suspended particles, which will cause light scattering in the water, further challenging underwater optical imaging. In addition, due to the uncertainty of the movement of imaging equipment underwater, the acquired images have problems such as blurred details and low contrast, which affects visual perception and poses more severe challenges to subsequent high-level visual tasks. At the same time, due to the lack of underwater scenes and high-quality images, UIE faces various challenges such as over-enhancement and blurred detail features. These problems limit the performance of the UIE method and lead to poor performance of downstream tasks. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention provides a method for underwater robot visual clarity based on a multimodal fusion network. Based on the inspiration obtained from multimodal learning, the present invention attempts to introduce polarization information as an additional modality to enhance the original underwater image. A novel detail-focused and polarization-guided multimodal fusion network is proposed, which integrates the RGB modality and the polarization modality to enhance the underwater image. The detail-focused differential convolution is used to capture more details and edge information, and the polarization degree information and polarization angle information are used to enhance the contrast and texture details of different areas in the image, thereby more accurately restoring the true color of the image, reducing its serious interference to subsequent computer vision tasks, and significantly improving the quality of underwater images. The technical solution is as follows:

[0004] A method for underwater robot visual clarity based on U-Net multimodal fusion network, characterized by comprising the following steps:

[0005] S1, collecting turbid underwater images and corresponding clear underwater images, and constructing an underwater polarization image dataset, wherein the underwater polarization image dataset includes underwater polarization images, polarization degree images, and polarization angle images at different angles;

[0006] S2. Construct an underwater robot visual clarity model based on a multimodal fusion network, including a multimodal fusion network and an image enhancement U-Net network;

[0007] S3, training the multimodal fusion network based on the underwater polarization image dataset, and using pixel multi-scale fusion to update the RGB information and polarization information and generate fusion features during training;

[0008] S4, using the underwater polarization image dataset to train the image enhancement U-Net network to obtain an image enhancement model based on the U-Net network;

[0009] S5. Obtain the fusion features with turbidity to be processed and input them into the image enhancement model based on the U-Net network to obtain a clear underwater image.

[0010] Furthermore, the method for constructing an underwater polarization image dataset in step S1 is as follows: by creating water bodies of different colors with different turbidity levels in a water scene, using a polarization camera to collect turbid underwater images corresponding to objects in water bodies of different colors with different turbidity levels, collecting clear underwater images corresponding to objects in pure water, and using the clear underwater images as label images; constructing a training set and a test set based on the turbid underwater images and the label images.

[0011] Furthermore, in step S2, the network architecture of the underwater robot vision clarity model based on the multimodal fusion network is an end-to-end structure, and feature maps of different sizes are generated at each level, so that the network can capture features of different scales; before the fusion network is performed, the turbid underwater polarization images in the training set are preprocessed to obtain RGB modal information and polarization modal information, and the polarization modal information includes polarization degree information DoLP and polarization angle information AoLP; the obtained RGB modal information and polarization modal information are used as input of the multimodal fusion network.

[0012] Furthermore, the multimodal fusion network includes two modules: a feature fusion module and a polarization-guided fusion module, wherein:

[0013] The feature fusion module robustly fuses the DoLP and AoLP features from the polarization modal input domain by leveraging global and local information. Two spatial attention maps are generated based on the two labeled embedding sequences provided by the two Conformers for the two input features DoLP and AoLP. The extracted convolutional features are then weighted according to the spatial attention maps and fused together to obtain the polarization modal features.

[0014] The polarization-guided fusion module is used to process modality deviations, enhance the input features of the RGB modality input feature X using attention operations, and update and generate the fusion feature X using the polarization modality feature M guidance. * , the polarization modal features and RGB modal features are concatenated and projected through a multi-layer perceptron to generate key (k x )、query(q x ) and value(v x ); Reduce the embedding height and width H and W of query and key through the spatial dimension, and learn the channel statistics S of query and key q , S k , thus obtaining the channel relationship M that guides the update * .

[0015] Furthermore, the method of concatenating and projecting the polarization modal features and the RGB modal features through the multi-layer perceptron is as follows: to generate key (k x )、query(q x ) and value(v x ), As learnable parameters:

[0016]

[0017] [X * ]=FC(softmax(FC([q x ;k x ]))⊙v x )

[0018] Among them, ⊙ represents element-wise multiplication and FC represents a fully connected layer with filtering.

[0019] Furthermore, we reduce the embedding height and width H and W of query and key by spatial dimension and learn the channel statistics S of query and key. q , S k , thus obtaining the channel relationship M that guides the update * The formula is as follows:

[0020] K m ,Q m ,V m=X,M,k x

[0021]

[0022] M * =M x +FC((s q Q m +s k K m )⊙V m ).

[0023] A further U-Net network-based image enhancement model consists of three parts: an encoder part, a feature variation part, and a decoder part; feature extraction blocks are deployed from the first layer to the third layer in the image enhancement U-Net network, that is, different blocks are used in different layers to extract corresponding features, and the third layer uses a detail enhancement attention block (DEAB) to capture more details and edge features;

[0024] The detail enhancement attention block (DEAB) includes a detail focusing convolution block and a content guided attention block. The detail focusing convolution block uses differential convolution to integrate prior information, supplements the convolution layer of the parallel processing operation, and enhances the representation capability. By using the reparameterization technology, the detail aggregation convolution is equivalently converted to a convolution operation, thereby reducing parameters and computational costs.

[0025] The content-guided attention block adopts a dynamic fusion method to obtain more useful information encoded in the features by assigning a unique spatial importance map to each channel: the low-dimensional features from the encoder part are fused with the high-dimensional features from the decoder part, and the features are modulated by the learned spatial weights, so as to adaptively fuse the low-dimensional features of the encoder part with the corresponding high-dimensional features from the decoder part; input features are added through jump connections to alleviate the gradient vanishing problem and simplify the learning process; the fused features are mapped through a 3×3 convolutional layer to obtain the final clear result.

[0026] Furthermore, two downsampling and two upsampling operations are used between different layers to ensure dimensional consistency. The downsampling operation halves the spatial dimension and doubles the number of channels. This is achieved through convolutional layers by setting the stride value to 2 and setting the number of output channels to twice the number of input channels. The upsampling operation is regarded as the inverse form of the downsampling operation, and the downsampling operation is achieved through deconvolutional layers, where the sizes of the first, second, and third layers are C×H×W, respectively.

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] The present invention proposes a method for underwater robot vision sharpening based on a multimodal fusion network. The method can effectively avoid the imaging defects caused by existing underwater image sharpening methods in the face of high turbidity conditions, improve the underwater image quality while removing turbidity, and has effectiveness and robustness, thus laying a theoretical and technical foundation for subsequent visual tasks such as seabed panoramic observation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 The present invention is a flowchart of a method for underwater robot vision clarity based on a multimodal fusion network in an embodiment of the present invention.

[0031] Figure 2 This is a network architecture diagram for underwater robot vision clarification based on a multimodal fusion network in an embodiment of the present invention.

[0032] Figure 3 These are the turbid image, polarization angle image and polarization degree image acquired by the underwater robot vision sharpening network based on the multimodal fusion network in the embodiment of the present invention.

[0033] Figure 4 It is a clear underwater image output by the underwater robot vision sharpening network based on the multimodal fusion network in an embodiment of the present invention.

[0034] Figure 5 The following is a comparison table of the underwater robot vision sharpening network based on the multimodal fusion network in the embodiment of the present invention and other existing networks in terms of five commonly used underwater image quality evaluation indicators. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] like Figure 1 As shown, the present invention provides a method for underwater robot vision clarity based on a multimodal fusion network, which mainly includes the following steps:

[0038] S1. Construct an underwater polarization image dataset, wherein the training dataset includes underwater polarization images, polarization degree images, and polarization angle images with four different angles. Specifically, by creating different colors of water bodies with different turbidity in an artificially constructed indoor water scene, collecting turbid underwater images corresponding to objects in different colors of water bodies with different turbidity, collecting clear underwater images corresponding to objects in pure water, and using the clear underwater images as label images.

[0039] For example, an indoor turbid underwater polarization image acquisition platform is built, which includes a glass water tank, a polarization camera, a computer, a lighting system and a camera tripod. The size of the glass water tank is 150cm×35cm×50cm, and the lighting system is three different colors of lumen lights, namely blue, green and white. The steps for collecting turbid underwater images are as follows:

[0040] First, the images of objects in water bodies with different turbidity levels in different color scenes were collected. Each group obtained object images in 5 different turbidity levels and 3 different color scenes, that is, turbid underwater images. Clear label images were also collected. During the shooting and collection process, the polarization camera and the objects in the tank were fixed by physical methods so that their relative positions would not change. Secondly, a variety of objects such as corals, starfish, conchs, and shells were prepared and divided into two groups during the data collection process. One group covered the bottom of the tank with bottom sand and gravel to simulate the seabed or riverbed, and fixed the objects on it as underwater images in complex scenes. The other group fixed the objects on a pure white background board as underwater images in simple scenes. A total of 1,600 turbid underwater images and their corresponding label images were obtained through the turbid underwater image acquisition platform, of which 800 turbid underwater images were obtained for simple scenes and 800 for complex scenes, respectively, with an image resolution of 1024×1224.

[0041] S2. Construct an underwater robot vision clarity model based on a multimodal fusion network, including a multimodal fusion network and an image enhancement U-Net network; the network architecture of the underwater robot vision clarity model based on a multimodal fusion network is an end-to-end structure, which generates feature maps of different sizes at each level, so that the network can capture features of different scales.

[0042] Before the fusion network is used, the turbid underwater polarization images in the training set are preprocessed to obtain RGB information and polarization information. The polarization camera measures the polarization angle φ through focal plane spectroscopy. pol The light intensity I passing through the linear polarizer pol , and its calculation formula is:

[0043] Ipol=Iun*(1+ρ*cos(2φ-2φpol))

[0044] S 0 =I 0° +I 90° =I 45° +I 135°

[0045] S 1 =I 0° -I 90°

[0046] S 2 =I 45° -I 135°

[0047]

[0048] Among them, I un is the total incident light entering the camera, generally unpolarized light; ρ is the linear polarization degree, φ is the linear polarization angle, S 0 , S 1 , S 2 represents the Stokes constant, where S 0 It is also used to represent RGB modal information. Polarization modal information includes DoLP and AoLP, where DoLP represents the degree of polarization information and AoLP represents the angle of polarization information. 0° I 90° I 45° I 135° They respectively represent the images captured by the polarization camera at angles of 0°, 45°, 90°, and 135° for the four linear polarization states of light; the obtained RGB modal information and polarization modal information are then used as the input of the multimodal fusion network.

[0049] S3. The multimodal fusion network in the present application includes: a feature fusion module and a polarization guided fusion module. The multimodal fusion network is trained based on an underwater polarization image dataset, and during training, the feature fusion module and polarization guided fusion are used to update the RGB modal information and the polarization modal information and generate fusion features.

[0050] S301, the feature fusion module is used to robustly fuse the DoLP and AoLP features from the polarization modal input domain by utilizing global and local information. Two spatial attention maps are generated based on the two labeled embedding sequences provided by the two Conformers for the two input features DoLP and AoLP. The extracted convolutional features are then weighted according to the attention maps and fused together:

[0051] M φ ,M ρ =softmax(Ω(T φ ),Ω(T ρ ))

[0052]

[0053] Among them, C and T are the convolutional features and tag embeddings generated by the conv and trans branches in Conformer, respectively. is element-wise multiplication. M is the attention map generated by φ (AoLP) and ρ (DoLP), and Ω is a function that first reduces the dimension of each token embedding to 1 through a fully connected layer, and then reshapes the generated embedding into a two-dimensional graph. The features extracted by Conformers in different layers are used to capture more details and edge information using DoLP and AoLP, and the contrast and texture details of different regions in the image are enhanced, thereby more accurately restoring the true color of the image.

[0054] S302. Since the polarization mode deviates greatly from the RGB mode, the importance of the clues collected from the RGB mode and the polarization mode is related to the scene. Simply combining these clues together may dilute the influence of strong clues with weak signals, or even amplify the adverse effects of mixed clues. In order to deal with this modality deviation, a polarization-guided fusion module is designed. The RGB modality input feature X is enhanced by the attention operation, and the polarization modality feature M is guided to update and generate the fusion feature X. * , the polarization modal features and RGB modal features are concatenated and projected through a multi-layer perceptron to generate key (k x )、query(q x ) and value(v x ), As learnable parameters:

[0055]

[0056] [X * ]=FC(softmax(FC([q x ;k x ]))⊙v x )

[0057] Among them, ⊙ represents element-wise multiplication and FC represents a fully connected layer with filtering.

[0058] Reduce the embedding height and width H and W of query and key by spatial dimension, and learn the channel statistics S of query and key q , S k , thus obtaining the channel relationship that guides the update.

[0059] K m ,Q m ,V m =X,M,k x

[0060]

[0061] M * =M x +FC((s q Q m +s k K m )⊙V m )

[0062] S4. The enhancement network is trained using an underwater polarization image dataset to obtain the output of the multimodal fusion network: the fusion features with turbidity to be processed, and input into an image enhancement network based on a U-Net network to obtain a clear underwater image; the image enhancement network based on a U-Net network consists of three parts: an encoder part, a feature change part, and a decoder part.

[0063] For the fusion features with turbidity to be processed, the goal of U-Net is to restore the corresponding clear image. However, for detail-sensitive tasks such as de-hazing, we cannot only consider converting features in low-resolution space, which will lead to information loss. Therefore, feature extraction blocks are deployed from the first layer to the third layer in the U-Net network, and different blocks are used in different layers to extract corresponding features. The first and second layers use conventional feature extraction blocks (DEB), and the third layer uses detail enhancement attention blocks (DEAB) to capture more details and edge features. At the same time, in U-Net, two downsampling and two upsampling operations are used. The downsampling operation halves the spatial dimension and doubles the number of channels; it is implemented through ordinary convolutional layers by setting the step value to 2 and setting the number of output channels to twice the number of input channels; the upsampling operation can be regarded as the inverse form of the downsampling operation, and the downsampling operation is implemented through deconvolution layers, where the sizes of the first, second, and third layers are C×H×W, respectively.

[0064] Furthermore, the proposed detail enhancement attention block consists of a detail-focused convolution block and a content-guided attention block, which is used to enhance feature learning and thus improve the dehazing performance.

[0065] The detail-focused convolution block uses differential convolution to integrate prior information, supplement the ordinary convolution information, and enhance the representation ability; then by using the re-parameterization technology, the detail-aggregated convolution can be equivalently converted to ordinary convolution, thereby reducing parameters and computational costs.

[0066] The content-guided attention block consists of channel attention and spatial attention, which are used to calculate the attention weights of the channel and spatial dimensions respectively. Channel attention calculates a channel vector, i.e. To recalibrate the features. Spatial attention computes the spatial importance map, i.e. To adaptively indicate the informative regions. The content-guided attention block treats different channels and pixels unequally, thus improving the denoising performance.

[0067]

[0068]

[0069] Among them, max(0,x) represents the ReLU activation function, denotes a convolutional layer with a kernel size of k×k and [·] denotes a channel-wise concatenation operation. and Represent the features processed by the global average pooling operation in the spatial dimension, the global average pooling operation in the channel dimension, and the global maximum pooling operation in the channel dimension. In order to reduce the number of parameters and limit the complexity of the model, the first 1×1 convolution reduces the channel dimension from C to The second 1×1 convolution expands the channel dimension to C.

[0070] W coa =W c +W s

[0071] Use content-guided attention blocks to obtain the exclusive spatial importance map of each single channel of the input feature in a coarse-to-fine manner, and fully mix the channel attention weights and spatial attention weights to ensure information interaction. According to the broadcast rule, W is added by a simple addition operation. c and W s Fusion together to obtain a coarse spatial importance map Because W c is channel-based, so W coa and X are consistent in the channel. In order to obtain the final perfect spatial importance map W, W coa Each channel of is adjusted according to the corresponding input features. Guided by the content of the input features, the final channel-specific spatial importance map W is generated. In particular, W coa Each channel of and X is rearranged in an alternating manner through a channel shuffle operation. Combined with the subsequent group convolutional layer, the number of parameters can be greatly reduced.

[0072]

[0073] Among them, σ represents the sigmoid operation, CS(·) represents the channel shuffle operation, represents a group convolution layer with a kernel size of k×k. In the implementation, the number of groups is set to C. The content-guided attention mechanism assigns a unique spatial importance map to each channel, guiding the model to focus on the important regions of each channel. Therefore, more useful information encoded in the features can be emphasized, effectively improving the dehazing performance.

[0074] Secondly, the dynamic fusion scheme based on content-guided attention blocks can effectively fuse features and help gradient flow, fusing the features after downsampling operation with the corresponding features before upsampling operation, using an encoder-decoder architecture. low and the feature F from the decoder part high F fuse, which is an effective technique in dehazing and other low-level vision tasks. Low-level features (such as edges and contours) play an important role in restoring clear images, but they gradually lose their effect after passing through many intermediate layers. Feature fusion can enhance the information flow from shallow to deep layers, which is beneficial to feature preservation and gradient backpropagation. Using a dynamic fusion scheme based on content-guided attention blocks, the features are modulated by learned spatial weights to adaptively fuse the low-level features of the encoder part with the corresponding high-level features. The core part chooses to use the content-guided attention mechanism to calculate spatial weights for feature modulation. The low-level features of the encoder part and the corresponding high-level features are input into the content-guided attention mechanism to calculate the weights, and then combined by the weighted summation method. The input features are also added through jump connections to alleviate the gradient vanishing problem and simplify the learning process. Finally, the fused features are mapped through a 3×3 convolutional layer to obtain the final sharpening result.

[0075] F fuse =C 1×1 (F low ·W+F high (1-W)+F low +F high )

[0076]

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for underwater robot visual clarity based on U-Net multimodal fusion network, characterized in that: The following steps are involved: S1, collecting turbid underwater images and corresponding clear underwater images, and constructing an underwater polarization image dataset, wherein the underwater polarization image dataset includes underwater polarization images, polarization degree images, and polarization angle images at different angles; S2. Construct an underwater robot visual clarity model based on a multimodal fusion network, including a multimodal fusion network and an image enhancement U-Net network; S3, training the multimodal fusion network based on the underwater polarization image dataset, and using pixel multi-scale fusion to update the RGB information and polarization information and generate fusion features during training; S4, using the underwater polarization image dataset to train the image enhancement U-Net network to obtain an image enhancement model based on the U-Net network; S5. Obtain the fusion features with turbidity to be processed and input them into the image enhancement model based on the U-Net network to obtain a clear underwater image.

2. According to claim 1, a method for underwater robot visual clarity based on multimodal fusion network is characterized in that: The method for constructing an underwater polarization image dataset in step S1 is as follows: by creating different colored water bodies with different turbidity levels in a water scene, using a polarization camera to collect turbid underwater images corresponding to objects in different colored water bodies with different turbidity levels, collecting clear underwater images corresponding to objects in pure water, and using the clear underwater images as label images; constructing a training set and a test set based on the turbid underwater images and the label images.

3. The method for underwater robot visual clarity based on multimodal fusion network according to claim 1 is characterized in that: In step S2, the network architecture of the underwater robot vision clarity model based on the multimodal fusion network is an end-to-end structure, and feature maps of different sizes are generated at each level, so that the network can capture features of different scales; before the fusion network is performed, the turbid underwater polarization images in the training set are preprocessed to obtain RGB modal information and polarization modal information, and the polarization modal information includes polarization degree information DoLP and polarization angle information AoLP; the obtained RGB modal information and polarization modal information are used as the input of the multimodal fusion network.

4. The underwater robot vision clarity method based on multimodal fusion network according to claim 1 is characterized in that: The multimodal fusion network includes two modules: a feature fusion module and a polarization-guided fusion module, wherein: The feature fusion module robustly fuses the DoLP and AoLP features from the polarization modal input domain by leveraging global and local information. Two spatial attention maps are generated based on the two labeled embedding sequences provided by the two Conformers for the two input features DoLP and AoLP. The extracted convolutional features are then weighted according to the spatial attention maps and fused together to obtain the polarization modal features. The polarization-guided fusion module is used to process modality deviations, enhance the input features of the RGB modality input feature X using attention operations, and update and generate the fusion feature X using the polarization modality feature M guidance. * , the polarization modal features and RGB modal features are concatenated and projected through a multi-layer perceptron to generate key (k x )、query(q x ) and value(v x ); Reduce the embedding height and width H and W of query and key through the spatial dimension, and learn the channel statistics S of query and key q , S k , thus obtaining the channel relationship M that guides the update * .

5. The method for underwater robot visual clarity based on multimodal fusion network according to claim 4 is characterized in that: The method of concatenating and projecting polarization modal features and RGB modal features through a multi-layer perceptron is as follows: x )、query(q x ) and value(v x ), As learnable parameters: [X * ]=FC(softmax(FC([q x ;to x ]))⊙v x ) Among them, ⊙ represents element-wise multiplication and FC represents a fully connected layer with filtering.

6. The underwater robot vision clarity method based on multimodal fusion network according to claim 5 is characterized in that: Reduce the embedding height and width H and W of query and key by spatial dimension, and learn the channel statistics S of query and key q , S k , thus obtaining the channel relationship M that guides the update * The formula is as follows: K m ,Q m ,V m =X,M,k x M * =M x +FC((s q Q m +with k K m )⊙V m )。 7. The underwater robot vision clarity method based on multimodal fusion network according to claim 1 is characterized in that: The image enhancement model based on the U-Net network consists of three parts: an encoder part, a feature change part and a decoder part. In the image enhancement U-Net network, feature extraction blocks are deployed from the first layer to the third layer, that is, different blocks are used in different layers to extract corresponding features. The third layer uses a detail enhancement attention block (DEAB) to capture more details and edge features. The detail enhancement attention block (DEAB) includes a detail focusing convolution block and a content guided attention block. The detail focusing convolution block uses differential convolution to integrate prior information, supplements the convolution layer of the parallel processing operation, and enhances the representation capability. By using the reparameterization technology, the detail aggregation convolution is equivalently converted to a convolution operation, thereby reducing parameters and computational costs. The content-guided attention block adopts a dynamic fusion method to obtain more useful information encoded in the features by assigning a unique spatial importance map to each channel: the low-dimensional features from the encoder part are fused with the high-dimensional features from the decoder part, and the features are modulated by the learned spatial weights, so as to adaptively fuse the low-dimensional features of the encoder part with the corresponding high-dimensional features from the decoder part; input features are added through jump connections to alleviate the gradient vanishing problem and simplify the learning process; the fused features are mapped through a 3×3 convolutional layer to obtain the final clear result.

8. The underwater robot vision clarity method based on multimodal fusion network according to claim 7 is characterized in that: Two downsampling and two upsampling operations are used between different layers to ensure dimensional consistency. The downsampling operation halves the spatial dimension and doubles the number of channels. This is achieved through convolutional layers by setting the stride value to 2 and setting the number of output channels to twice the number of input channels. The upsampling operation is considered to be the inverse form of the downsampling operation, while the downsampling operation is achieved through deconvolutional layers, where the first, second, and third layers have sizes of C×H×W, respectively.

Images