Underwater multimodal data acquisition device and image enhancement method

Abstract

The present invention discloses an underwater multimodal data acquisition device and an image enhancement method, which belong to the field of image enhancement and restoration technology. The data acquisition device includes a spectroscopic prism, an event camera, a polarization camera, an edge computer, and a control logic module; the two beams of light divided by the spectroscopic prism are incident on the event camera and the polarization camera at the same viewing angle respectively; the event camera is used to generate event data of an underwater target scene; the polarization camera is used to generate polarization data of an underwater target scene at different polarization angles; the edge computer uses an image enhancement method to store and analyze the event data and the polarization data at different polarization angles to obtain the final underwater target scene image; the control logic module is used to send control instructions to the event camera and the polarization camera. Through the above-mentioned data acquisition device and the image enhancement method based on the device, the present invention improves the adaptability and generalization ability of image processing in a variable underwater environment.

Description

Technical Field

[0001] The present invention relates to the technical field of image enhancement and restoration, and more particularly to an underwater multimodal data acquisition device and an image enhancement method. Background Art

[0002] Precision and high-quality underwater operations are inseparable from the collection and analysis of underwater images. However, due to the special physical properties of water and its solutes and suspended matter, underwater images directly collected using traditional cameras are often of poor quality.

[0003] On the one hand, traditional cameras have limited dynamic range. When capturing scenes with a large dynamic range, such as underwater environments with uneven lighting, some areas in the image are often too bright or too dark. At the same time, when there is relative motion between the camera and the scene, traditional camera imaging will produce dynamic blur, that is, the image of the scene will have obvious blur and drag traces. These two problems not only affect the usability of underwater images, but also limit the effectiveness of subsequent image processing and analysis.

[0004] On the other hand, existing underwater image enhancement or restoration technologies, such as physical model-based image restoration methods, can restore the authenticity of images to a certain extent, but their performance depends greatly on the accuracy of the model and the quality of the input data, which is often difficult to guarantee in complex underwater environments. Although deep learning has made some progress in image dehazing and deblurring, these methods usually require a large amount of annotated data, and their generalization ability in complex underwater environments remains a challenge.

[0005] Therefore, how to effectively solve the main problems faced by existing underwater imaging technologies and achieve higher quality underwater image acquisition and enhancement is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0006] In view of this, the present invention provides an underwater multimodal data acquisition device and an image enhancement method.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] On the one hand, the present invention discloses an underwater multimodal data acquisition device, including a spectroscopic prism, an event camera, a polarization camera, an edge computer, and a control logic module;

[0009] The two beams of light split by the beam splitting prism are incident on the event camera and the polarization camera respectively at the same viewing angle;

[0010] The event camera is used to generate event data of an underwater target scene;

[0011] The polarization camera is used to generate polarization data of underwater target scenes at different polarization angles;

[0012] The edge computer uses an image enhancement method to store and analyze the event data and polarization data at different polarization angles to obtain a final underwater target scene image;

[0013] The control logic module is used to send control instructions to the event camera and the polarization camera.

[0014] Preferably, the polarization camera is powered by a POE module.

[0015] Preferably, the control logic module is also used to synchronize and correct the time when the event camera and the polarization camera collect data.

[0016] Preferably, the event camera performs data transmission and power supply via a USB data cable.

[0017] Preferably, the polarization camera performs data transmission and power supply via Category 6 network cables.

[0018] On the other hand, the present invention discloses an underwater multimodal data image enhancement method, comprising the following steps:

[0019] Collect event data of underwater target scenes and polarization data at different polarization angles respectively;

[0020] Pre-encoding the event data using an event encoder to obtain a three-dimensional tensor of an underwater target scene;

[0021] The total light intensity and polarization degree of the underwater target scene are calculated based on the polarization data at different polarization angles;

[0022] The three-dimensional tensor, total light intensity and calculated polarization degree of the underwater target scene are respectively input into three corresponding branch encoders to obtain branch encoding results;

[0023] The three branch encoding results are concatenated and fused to obtain fused data;

[0024] The fused data is decoded by using a fusion decoder to obtain a final underwater target scene image.

[0025] Furthermore, the event encoder includes a pulse neural network, and the pulse neural network includes three event convolutional layers connected in sequence.

[0026] Furthermore, the three corresponding branch encoders have the same structure, and each branch encoder includes a zeroth convolution layer, a first residual block, a first convolution layer, a first multi-scale dehazing convolution module, a second residual block, a second convolution layer, a second multi-scale dehazing convolution module, a third residual block, a third convolution layer, and a third multi-scale dehazing convolution module at the bottom layer, which are connected in sequence according to the direction of the data flow.

[0027] Furthermore, the three branch encoding results are concatenated and fused to obtain fused data, which specifically includes the following steps:

[0028] Input the data output by the first residual block in each branch encoder into the corresponding first connection module for connection to obtain first fused data;

[0029] Input the data output by the second residual block in each branch encoder into the corresponding second connection module for connection to obtain second fused data;

[0030] Input the data output by the third residual block in each branch encoder into the corresponding third connection module for connection to obtain third fused data;

[0031] Input the data output by the third multi-scale dehazing convolution module at the bottom layer in each branch encoder into the corresponding fourth connection module for connection to obtain fourth fused data;

[0032] The first fused data, the second fused data, the third fused data and the fourth fused data constitute the final fused data.

[0033] Furthermore, the fusion data is decoded by using a fusion decoder, which specifically includes the following steps:

[0034] According to the direction of the data flow, the fusion decoder includes, in sequence, a fourth deconvolution layer, a third residual block, a third deconvolution layer, a third skip-layer fusion module, a second residual block, a second deconvolution layer, a second skip-layer fusion module, a first residual block, a first deconvolution layer, a first skip-layer fusion module, a zeroth residual block, and a zeroth deconvolution layer;

[0035] The fourth fused data output by the fourth connection module is used as the input of the fourth deconvolution layer, the third fused data output by the third connection module is used as the input of the third skip-layer fusion module, the second fused data output by the second connection module is used as the input of the second skip-layer fusion module, and the first fused data output by the first connection module is used as the input of the first skip-layer fusion module;

[0036] The input first fusion data, second fusion data, third fusion data and fourth fusion data are decoded by using a fusion decoder to obtain a final water-free image.

[0037] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses an underwater multimodal data acquisition device and an image enhancement method, which have the following beneficial effects:

[0038] The present invention integrates an event camera and a polarization camera together through a beam splitter prism, reduces the volume of the multimodal data acquisition device, and can capture richer scene information from different dimensions, providing a new means for underwater image enhancement and restoration.

[0039] The image enhancement algorithm proposed in the present invention simultaneously collects event data and polarization data of the target scene. Compared with traditional methods, it reduces the dependence of underwater data collection on the accuracy of complex physical models and a large amount of labeled data, and improves the adaptability and generalization ability of image processing in changeable underwater environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0041] Figure 1 This is a structural block diagram of an underwater multimodal data acquisition device provided in an embodiment of the present invention.

[0042] Figure 2 A flowchart of the overall steps of the underwater multimodal data image enhancement method provided in an embodiment of the present invention.

[0043] Figure 3 A block diagram of a neural network model in the underwater multimodal data image enhancement method provided in an embodiment of the present invention.

[0044] Figure 4 A schematic diagram of the internal structure of each branch encoder provided in an embodiment of the present invention.

[0045] Figure 5 A schematic diagram of the internal structure of a fusion decoder provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Example 1

[0048] Embodiment 1 of the present invention discloses an underwater multimodal data acquisition device, which mainly includes a spectroscopic prism, an event camera, a polarization camera, an edge computer, and a control logic module; wherein the two beams of light split by the spectroscopic prism are incident on the event camera and the polarization camera respectively at the same viewing angle; the event camera is used to generate event data of an underwater target scene; the polarization camera is used to generate polarization data of the underwater target scene at different polarization angles; the edge computer uses an image enhancement method to store and analyze the event data and the polarization data at different polarization angles to obtain a final underwater target scene image; the control logic module is used to send control instructions to the event camera and the polarization camera.

[0049] like Figure 1 As shown, based on the above main devices, a complete underwater multimodal data acquisition device includes a spectroscopic prism 1, an event camera 2, a polarization camera 3, a Xavier (edge ​​computer) 4, a POE module 5, a step-up transformer 6, a photosensitive switch 7, a control logic module 8, a time correction LED 9 and a status LED 10.

[0050] The beam splitter prism 1 is located at the front of the device. Its function is to split the natural light from the outside into two beams, so that two different types of cameras can capture light incident from the same viewing angle at the same time. If the two cameras are simply placed side by side, no matter how close they are, there will be parallax in the content they capture. However, the beam splitter prism allows the event camera and the polarization camera to synchronously collect the same light information, which is crucial to maintaining data consistency.

[0051] The underwater multimodal data acquisition device disclosed in the present invention uses a beam splitter prism as one of the core technical components, which provides a unified and high-quality data source for the polarization camera and the event camera, improving the accuracy, efficiency and reliability of image processing. First, the beam splitter prism ensures that the two cameras receive exactly the same scene information, eliminating the perspective differences that may occur in the traditional side-by-side configuration. This feature helps this patent achieve better image dewatering effects compared to existing patents. At the same time, this design is particularly important for subsequent applications that require precise synchronization of data, such as tracking fast-moving underwater targets in a dynamic environment or performing high-resolution three-dimensional reconstruction, ensuring seamless matching of polarization and event images.

[0052] Secondly, the beam splitter prism can simultaneously capture multi-dimensional information such as light intensity, direction and polarization state, while the existing patent only arranges the two cameras side by side. The original design cannot fully guarantee the consistency of the optical information of the two cameras, and there is a physical inaccuracy.

[0053] In addition, the design based on the beam splitter has the advantages of real-time and low latency. Since the polarization camera and the event camera share the same light source information and no additional calibration steps are required, the data processing process is more efficient and suitable for application scenarios that require instant feedback, such as high-speed machine vision systems.

[0054] The event camera 2 is a new type of visual sensor that, unlike traditional frame rate cameras, can capture light change events with extremely high temporal resolution. This means that when light changes suddenly (such as underwater creatures passing quickly), the event camera can record these changes without being limited by the frame rate like traditional cameras. This device uses the DVXplorer Mini event camera produced by iniVation, which has a spatial resolution of 640×480, a temporal resolution of 65-200μs, a dynamic range of 90-110dB, and a six-axis inertial measurement unit with an 8kHz sampling rate. It uses a USB3.0C port for power supply and data transmission. This device uses a USB3.0 data cable to connect the event camera to Xavier (edge ​​computer), which transmits both data and power supply.

[0055] Polarization camera 3 is used to capture the polarization state of light, which can provide underwater image degradation information such as surface characteristics of objects, underwater particle distribution, and light propagation process in water. The polarization camera model selected in this embodiment is PHX050S1-Q produced by LUCIDVision Labs Inc. This camera model uses PoE (Power Over Ethernet) power supply and data transmission through a network cable. It can acquire four-angle polarization images at a frame rate of up to 21.11FPS. The size of each polarization image is 1224×1024, supports automatic white balance, and the frame rate, aperture and exposure time are adjustable, with a maximum power of 3.1W. This embodiment uses a Cat6 network cable to transmit data and power the camera, and its power is provided by a POE (Power Over Ethernet) module. The POE module allows power and data to be transmitted through the same network cable, simplifying the wiring requirements.

[0056] Xavier (edge ​​computer) 4 is the core processing unit in the underwater multimodal data acquisition device. As a high-performance edge computer, the edge computer is responsible for processing the data from the camera. It can quickly pre-process the collected data, analyze and store it in real time, and if necessary, the subsequent supporting image processing algorithms can be deployed in it to achieve real-time acquisition and processing of underwater images.

[0057] In order to ensure the high efficiency, controllability and multifunctional adaptability of the present invention in underwater environment, the underwater multimodal data acquisition device is designed with a main control board as the core manual intervention mechanism. The main control board integrates control modules and multiple status LED indicators, giving divers operational flexibility and equipment management capabilities in complex and changeable underwater operations.

[0058] In this embodiment, the main control board of the underwater multimodal data acquisition device is composed of a control module and a variety of status LED indicators. The control module is directly related to the manual operation of the device. Considering the strict requirements of underwater operations on the sealing of the equipment and the needs of divers for operating experience, this embodiment uses a photosensitive switch as the core control element. This non-contact operation method simplifies the human-computer interaction process, ensures the integrity of the waterproof shell and the safe operation of the entire system, and gives the user the ability to respond quickly and accurately when facing complex underwater conditions. Specifically, the present invention uses a photosensitive switch as the switch of the control module. The essence of the photosensitive switch is an infrared photoelectric tube, which is divided into an infrared light emitting tube and an infrared light receiving tube. The light emitted by the former cannot be directly received by the latter, and the photoelectric tube is installed in the waterproof shell. When the diver wants to start collecting data, he puts his hand in the corresponding position outside the shell. At this time, the diver's hand will reflect the infrared light, so that the receiving tube receives the infrared light and starts the collection program.

[0059] The control module receives signals from the photosensitizer and controls the data acquisition process, and sends control instructions and information to the polarization camera and edge computer Xavier, such as starting acquisition and synchronization signals.

[0060] In addition, the control logic module also activates the time correction LED before and after the acquisition to assist in time correction, which is very important for ensuring accurate synchronization of data. The time correction LED is an essential component of the time correction engineering method. Generally speaking, for the same moment, there is an unknown offset between the timestamp of the event camera and the timestamp of the polarization camera. The software controls the time correction LED to flash at the beginning and end of the recording, and the timestamp of the time correction LED flashing is the same as the timestamp of the polarization camera. At the same time, because the time resolution of the event camera is extremely high and it is extremely sensitive to changes in illumination, the two LED flashing events captured by the event camera can be obtained, and the timestamps of these two events can be obtained. In this process, four timestamps are obtained, namely the timestamp recorded by the software at the two flashes (equal to the time of the polarization camera at the two flashes) and the timestamp of the two flashing events observed by the event camera (equal to the time of the event camera at the two flashes). The time difference of the two timestamps corresponding to the flash at the beginning of the recording is taken as the difference, and the time difference of the two timestamps corresponding to the flash at the end of the recording is taken as the difference, and the two differences are averaged to obtain the time deviation of the two cameras.

[0061] The reason why it only flashes at the beginning and end of recording (only twice in total) is that flashing during recording will reduce data quality (introducing flashes unrelated to the scene), and the time deviation between the two cameras in one recording remains basically unchanged (there is no need to flash multiple times).

[0062] In order to further enhance the effectiveness and convenience of manual intervention, the present invention is also equipped with a variety of status LED indicators, which can convey key information such as the system's current working mode, data collection progress, power level, etc. to divers in an intuitive way, thereby helping them make reasonable operational judgments and plan adjustments.

[0063] Specifically, in the present invention, the status LED is used to display the current status of the device, including battery power, device operating status, etc. There are multiple status LEDs, specifically, it can be set to 3, which are red, yellow and green. Different status combinations (on and off, flashing and always on, etc.) represent different system states (standby, recording, low power, etc.), which provides instant feedback to the operator to ensure the normal operation of the device or to terminate the collection work in time when an abnormality (such as low power) occurs.

[0064] The entire device is powered by a battery 11 with a nominal voltage of 14.8V (the voltage during operation is between 14.8V and 16.8V), and the capacity of the battery 11 is between 5.3AH and 10AH. The battery can ensure that the device can work normally without external cable power supply, which is very important for underwater data acquisition.

[0065] The image enhancement algorithm of the underwater multimodal data acquisition device can make comprehensive use of the information provided by the polarization camera and the event camera, and significantly improve the clarity and contrast of underwater images through specific data processing and analysis processes. The algorithm is a hybrid neural network that integrates multimodal data and is trained using a specific loss function. Its input is the four-channel polarization map and event stream data collected by the device, and its output is a dewatered image that removes color cast, scattering, and dynamic blur.

[0066] Example 2

[0067] Embodiment 2 of the present invention discloses a method for enhancing underwater multimodal data images, such as Figure 2 As shown, the following steps are included:

[0068] Collect event data of underwater target scenes and polarization data at different polarization angles respectively;

[0069] Pre-encoding the event data using an event encoder to obtain a three-dimensional tensor of an underwater target scene;

[0070] The total light intensity and polarization degree of the underwater target scene are calculated based on the polarization data at different polarization angles;

[0071] The three-dimensional tensor, total light intensity and calculated polarization degree of the underwater target scene are respectively input into three corresponding branch encoders to obtain branch encoding results;

[0072] The three branch encoding results are concatenated and fused to obtain fused data;

[0073] The fused data is decoded by using a fusion decoder to obtain a final underwater target scene image.

[0074] The underwater multimodal data image enhancement method proposed in the present invention is implemented by a hybrid neural network that fuses multimodal data and is trained using a specific loss function. Its input is the four-channel polarization map and event stream data collected by the device, and its output is a dewatered image that removes color cast, scattering and dynamic blur.

[0075] The underwater multimodal data image enhancement method thus inputs multimodal data as a whole. In the present invention, a Spike Neural Network (SNN) and a Deep Neural Network (DNN) are combined to design a hybrid neural network that can fuse multimodal data. The specific structure is as follows: Figure 3 As shown in Figure 2, the entire hybrid network structure mainly consists of an event encoder, three branch encoders, four connection modules and a fusion decoder, where the event encoder is a pulse neural network.

[0076] The hybrid neural network for fusing multimodal data of the present invention includes a spike part and a non-pulse part, wherein the spike part is composed of a spike neural network (SNN) as an event encoder; the non-pulse part is composed of a deep neural network (DNN) including a branch encoder and a fusion decoder.

[0077] The event encoder is used to pre-encode the event data collected by the event camera, align the event stream with the image completion time, and convert it into a three-dimensional tensor similar to the image, and then send the processed three-dimensional tensor data to one of the branch encoders. The event camera outputs data in the form of pulses, which has a natural similarity with the design concept of the spiking neural network SNN, allowing the latter to naturally process its data without too much format conversion while retaining the spatiotemporal information of the original data. The event camera data is suitable for pre-processing with the spiking neural network SNN. The event encoder used in the algorithm consists of three sequentially connected event convolutional layers, and in order to better preserve visual information, event data is also used as input to the intermediate layer and output layer.

[0078] Polarization camera can obtain The polarization subgraphs at four angles at the moment , first calculate the total light intensity and calculate the degree of polarization , these two are normalized and used as inputs of the other two branch encoders respectively. The normalization method adopted in the embodiment of the present invention is a simple linear scaling of the maximum and minimum values. For example, for the light intensity map, the default value range is 0~255, which is linearly mapped to the range of -1~1. The polarization degree map (default value range 0~1) is also normalized to the preset range by mapping.

[0079] Event data (3D tensor) and total light intensity after being processed by the event encoder , calculate the degree of polarization are sent to three branch encoders with the same structure. The structure of each branch encoder is as follows Figure 4 As shown, it is composed of a multi-layer neural network. According to the flow direction of the data flow, each branch encoder includes the (zeroth) convolution layer (Conv), (first) residual block (ResBlock), (first) convolution layer (Conv), (first) multi-scale defogging convolution module (MDCBlock), (second) residual block (ResBlock), (second) convolution layer (Conv), (second) multi-scale defogging convolution module (MDCBlock), (third) residual block (ResBlock), (third) convolution layer (Conv), (third) multi-scale defogging convolution module (MDCBlock), and its overall structure is similar to an inverted trapezoid. The closer to the lower layer of the branch encoder, the smaller the size (length and width) of the corresponding output feature map. The data output from the three residual blocks of each branch encoder and the bottom multi-scale defogging convolution module are connected to four connection modules respectively. Convolutional layers and residual blocks are classic DNN structures. The former extracts image feature information through convolution operations, while the residual connection adds short circuits in the network so that the output of the shallow layer of the residual block can be directly added to the output of the deep layer, thereby avoiding the problems of gradient vanishing and negative optimization. The multi-scale dehazing convolution module has multiple convolutional layers built in. It extracts features from the information (feature map) fed into the multi-scale dehazing convolution module using convolution kernels of different scales to reduce information loss, and then uses deconvolution, pooling and other structures to fuse feature information.

[0080] Figure 4 In the example, the input of the branch encoder (encoded event data, polarization map or total light intensity map) is first fed into the top-level convolutional neural network, and then calculated in order from top to bottom. The branch encoder includes four special network layers, namely three residual layers and the bottom multi-scale dehazing convolution module, whose output results will be used as the output of the branch encoder.

[0081] Each connection module only connects the output of the first residual block, the output of the second residual block, the output of the third residual block, or the output of the bottom-level multi-scale dehazing convolution module in the three branch encoders. Therefore, there are 4 connection modules in the hybrid neural network that integrates multimodal data. Assuming that the output of the first branch encoder is [R11, R12, R13, M1], the output of the second branch encoder is [R21, R22, R23, M2], and the output of the third branch encoder is [R31, R32, R33, M3], then the input of the first connection module is [R11, R21, R31], and so on. Figure 4 It can be seen that data flows from top to bottom. The closer the information is to the bottom, the more advanced and macro information it implicitly contains, such as the overall color tone of the picture and the light absorption rate of water; the closer the information is to the top, the more micro information it contains, such as the edge information and brightness information of a small area in the picture. Advanced information comes from low-level information, and they complement each other. Both are indispensable.

[0082] The above fusion steps are summarized as follows: the data output by the first residual block in each branch encoder is input into the corresponding first connection module for connection to obtain the first fused data; the data output by the second residual block in each branch encoder is input into the corresponding second connection module for connection to obtain the second fused data; the data output by the third residual block in each branch encoder is input into the corresponding third connection module for connection to obtain the third fused data; the data output by the third multi-scale dehazing convolution module at the bottom layer in each branch encoder is input into the corresponding fourth connection module for connection to obtain the fourth fused data.

[0083] In the above process, the feature maps output by the three branch encoders (corresponding to the three modes of event data, polarization degree and total light intensity) are stacked in depth, and then connected using connection modules (convolutional layers and residuals) to achieve preliminary fusion of features of different modalities in order to improve the performance and generalization ability of the network.

[0084] The fused data is then sent to the corresponding position of the feature map of the same size as the decoder to complete the connection between the encoder and the decoder. The fusion decoder receives the fused data from the connection module and uses multiple deconvolution, residual blocks, skip layer fusion and other modules in sequence to process the data to obtain the final water-free image.

[0085] Specifically, the structure of the fusion decoder is as follows Figure 5As shown, according to the direction of data flow (from bottom to top), it includes the (fourth) deconvolution layer (DeConv), the (third) residual block (ResBlock), the (third) deconvolution layer (DeConv), the (third) jump layer fusion module (JumpFuse), the (second) residual block (ResBlock), the (second) deconvolution layer (DeConv), the (second) jump layer fusion module (JumpFuse), the (first) residual block (ResBlock), the (first) deconvolution layer (DeConv), the (first) jump layer fusion module (JumpFuse), the (zeroth) residual block (ResBlock), and the (zeroth) deconvolution layer (DeConv).

[0086] Correspondingly, in this embodiment, the fused data is sent to the corresponding position of the feature map of the same size as the decoder to complete the connection between the encoder and the decoder. The specific process is as follows: the fourth fused data output by the fourth connection module is used as the input of the fourth deconvolution layer, the third fused data output by the third connection module is used as the input of the third skip-layer fusion module, the second fused data output by the second connection module is used as the input of the second skip-layer fusion module, and the first fused data output by the first connection module is used as the input of the first skip-layer fusion module. The data is processed by the fusion decoder to obtain the final water-free image.

[0087] In the training process of the hybrid neural network fusing multimodal data, L1 loss (mean absolute error loss), LPIPS loss (learning-perceptual image patch similarity loss) and DISTS loss (deep image structure and texture similarity loss) are used to measure the difference between the predicted output image and the real water-free image.

[0088] The L1 loss measures the pixel-level error between the predicted value and the true water-free image by calculating the Manhattan distance, which is calculated as follows:

[0089] ;

[0090] in Represents the pixel value of the network's output prediction value at coordinates [x, y, c], Represents the pixel value of the true water-free image at the coordinate [x, y, c]; , , Respectively refer to the length, width, and number of channels of the image, , , Together they form the dimensions of the output image features. L1 loss can protect the low-level visual features of the image to a certain extent, and is helpful for color correction and edge texture restoration.

[0091] Different from L1 loss, LPIPS (Learnable Perceptual Image Patch Similarity, LPIPS) aims to capture the macroscopic perception of image quality and fidelity by humans, which is closer to the subjective feeling of human eyes. The LPIPS loss calculation consists of a fixed pre-trained network and a learnable linear layer. The calculation formula is as follows:

[0092] ;

[0093] In the formula That is the feature layer, That is, the corresponding learnable linear layers. The algorithm uses the pre-trained VGG network as its fixed part and selects five feature layers to calculate the image feature loss, that is, . The calculation is from and The Euclidean distance of the multi-scale feature information extracted from the image is averaged and used as the network loss. The LPIPS loss can guarantee the high-level visual features of the image to a certain extent.

[0094] Similar to LPIPS loss, Deep Image Structure and Texture Similarity (DISTS) loss also uses neural networks to extract features and uses distance functions for comparison. The difference is that compared to focusing on human macroscopic subjective feelings, DISTS loss aims to capture the structure and texture information of the image and is robust to common distortions. Its formula is as follows:

[0095] ;

[0096] ;

[0097] ;

[0098] in For the neural network The feature layer channels, the input image is The extracted features can be obtained and , , is the mean of the corresponding feature, , is the standard deviation of the corresponding feature, is the covariance of the two features, and The algorithm uses a pre-trained VGG network with different network weights from LPIPS to complete feature extraction. .

[0099] In summary, the total loss function The definition is as follows:

[0100] ;

[0101] in , , It is a hyperparameter and is set to 1 during training.

[0102] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0103] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for underwater multimodal data image enhancement, characterized in that: The following steps are involved: Collect event data of underwater target scenes and polarization data at different polarization angles respectively; Pre-encoding the event data using an event encoder to obtain a three-dimensional tensor of an underwater target scene; the event encoder includes a pulse neural network, and the pulse neural network includes three sequentially connected event convolution layers; The total light intensity and polarization degree of the underwater target scene are calculated based on the polarization data at different polarization angles; The three-dimensional tensor, total light intensity and calculated polarization degree of the underwater target scene are respectively input into three corresponding branch encoders to obtain branch encoding results; The three branch encoding results are spliced ​​and fused to obtain fused data; the three corresponding branch encoders have the same structure, and each branch encoder includes a zeroth convolution layer, a first residual block, a first convolution layer, a first multi-scale defogging convolution module, a second residual block, a second convolution layer, a second multi-scale defogging convolution module, a third residual block, a third convolution layer, and a third multi-scale defogging convolution module at the bottom layer, which are connected in sequence according to the flow direction of the data flow; The fused data is decoded by using a fusion decoder to obtain a final underwater target scene image.

2. The underwater multimodal data image enhancement method according to claim 1, characterized in that: The three branch encoding results are concatenated and fused to obtain fused data, which specifically includes the following steps: Input the data output by the first residual block in each branch encoder into the corresponding first connection module for connection to obtain first fused data; Input the data output by the second residual block in each branch encoder into the corresponding second connection module for connection to obtain second fused data; Input the data output by the third residual block in each branch encoder into the corresponding third connection module for connection to obtain third fused data; Input the data output by the third multi-scale dehazing convolution module at the bottom layer in each branch encoder into the corresponding fourth connection module for connection to obtain fourth fused data; The first fused data, the second fused data, the third fused data and the fourth fused data constitute the final fused data.

3. The underwater multimodal data image enhancement method according to claim 2, characterized in that: Decoding the fused data using a fusion decoder specifically includes the following steps: According to the direction of the data flow, the fusion decoder includes, in sequence, a fourth deconvolution layer, a third residual block, a third deconvolution layer, a third skip-layer fusion module, a second residual block, a second deconvolution layer, a second skip-layer fusion module, a first residual block, a first deconvolution layer, a first skip-layer fusion module, a zeroth residual block, and a zeroth deconvolution layer; The fourth fused data output by the fourth connection module is used as the input of the fourth deconvolution layer, the third fused data output by the third connection module is used as the input of the third skip-layer fusion module, the second fused data output by the second connection module is used as the input of the second skip-layer fusion module, and the first fused data output by the first connection module is used as the input of the first skip-layer fusion module; The input first fusion data, second fusion data, third fusion data and fourth fusion data are decoded by using a fusion decoder to obtain a final water-free image.

Images