Underwater image enhancement method, system and device
Through data augmentation generative adversarial network and repair generative adversarial network, the color distortion and light scattering problems in underwater imaging are solved, and high-quality underwater image recovery is achieved.
Patent Information
- Application Number
- CN202111311399.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-08
- Publication Date
- 2025-08-26
- Estimated Expiration
- 2041-11-08
AI Technical Summary
Underwater imaging has problems such as color distortion, light scattering and insufficient exposure, which affects the effectiveness of the robot's vision system.
Data augmentation generative adversarial network and repair generative adversarial network are used to generate an underwater image repair model through preprocessing, feature extraction and loss function training to restore underwater image details.
It improves the quality of underwater image recovery, enhances the color accuracy and detail recovery ability of the image, and adapts to different underwater environments.
Smart Images
Figure CN114004766B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of digital image processing, and in particular to an underwater image enhancement method, system and equipment. Background Art
[0002] Many underwater environments, such as the ocean, rely on underwater robots for mining and research. Due to the rapid development of marine aquaculture, traditional seafood harvesting relies on manual salvaging with extensive experience in underwater operations. The complex underwater environment poses a life-threatening risk to salvage personnel. Furthermore, seafood's high temperature and environmental requirements necessitate rapid recovery in extreme weather conditions. Consequently, traditional salvaging methods are also subject to constraints such as ocean weather.
[0003] To address these issues, there is a huge demand for optical robots. Underwater robots can eliminate the need for humans to perform dangerous tasks underwater. However, compared to normal air environments, underwater imaging is relatively complex and presents the following main issues:
[0004] 1) The colors of underwater images are usually distorted due to depth, lighting conditions, water type, and different light wavelengths.
[0005] 2) A large number of suspended particles cause light scattering and propagation direction deviation. Both suspended particles and water affect the scene contrast by reflecting light to the camera lens;
[0006] 3) The absorption of light energy leads to underexposure, and the overall color is green or blue, which creates interference factors for vision-based robots. Summary of the Invention
[0007] In order to solve the above technical problems, the purpose of the present invention is to provide an underwater image enhancement method, system and device that can accurately restore underwater image degradation and other problems.
[0008] To achieve the above object, the technical solution provided by the present invention is as follows:
[0009] According to one aspect of the present invention, a method for underwater image enhancement is characterized by comprising the following steps:
[0010] Acquiring underwater video or image data;
[0011] Using a data-enhanced generative adversarial network, the video or image data is preprocessed using a preset algorithm to obtain an attenuation map and a fog effect map, features are extracted from the attenuation map and the fog effect map to obtain underwater simulation data, and a paired dataset is generated by combining the underwater video or image data with the corresponding underwater simulation data;
[0012] The discriminator loss of the obtained paired dataset is calculated using the repair generative adversarial network and trained to obtain the underwater image repair model.
[0013] Furthermore, the algorithm formula for preprocessing the video or image data using a preset algorithm to obtain the attenuation map and the fog effect map is as follows:
[0014] I (x) =J (x) t (x) +B(1-t (x) )
[0015] t (x) =e -β(λ)d(x)
[0016] Where I(x) represents the generated simulated underwater image, J(x) is the image in the air environment, t(x) is the transmission map of the scene, B is the atmospheric ambient light of the scene, β is the attenuation coefficient of light of different wavelengths, and d(x) is the range between the scene and the camera.
[0017] Furthermore, features of the attenuation map and the atomization effect map are extracted to obtain underwater simulation data, including: using a neural network to train the generated attenuation map and the atomization effect map, and using several convolution modules to extract attenuation and atomization features to obtain an underwater simulation data set.
[0018] Furthermore, the repaired generative adversarial network is used to calculate the discriminator loss of the obtained paired dataset, including the WGAN loss function, L1 loss function, and edge detection loss function.
[0019] Furthermore, the expression of the underwater image restoration model is:
[0020]
[0021] in,
[0022]
[0023] L1=E x,y [||yG(x)||1]
[0024] L edge =E x,y [||F edge (y)-F edge (G(x))||2]
[0025] In the formula, x represents underwater video or image data, y represents underwater simulation dataset, G(x) represents the image generated based on the generative network, are samples along the straight line between point pairs from the true data distribution and the generator distribution, λ, λ1, λ2 represent weight coefficients, and their values are determined empirically.
[0026] Furthermore, the edge detection loss function obtains edge detection information by dividing the image into three RGB channels for grayscale processing, and then fusing the three RGB channels for convolution, pooling and activation operations to extract the edge detection information of the generated network.
[0027] According to another aspect of the present invention, an underwater image enhancement system includes:
[0028] a data acquisition module configured to acquire underwater video or image data;
[0029] a data enhancement generative adversarial network module configured to preprocess the video or image data using a preset algorithm to obtain an attenuation map and a fog effect map, extract features from the attenuation map and the fog effect map to obtain underwater simulation data, and generate a paired data set by combining the underwater video or image data with the corresponding underwater simulation data;
[0030] The repair generative adversarial network module is configured to calculate the loss of the discriminator on the obtained paired dataset and train it to obtain an underwater image repair model.
[0031] Furthermore, the data enhancement generative adversarial network module includes:
[0032] A preprocessing unit configured to preprocess the video or image data using a preset algorithm to obtain an attenuation map and a fogging effect map;
[0033] a simulation data generation unit configured to train the generated attenuation map and atomization map using a neural network, and extract attenuation and atomization features using a plurality of convolution modules to obtain underwater simulation data;
[0034] The data pairing unit is configured to generate a pairing data set by combining the underwater video or image data with the corresponding underwater simulation data.
[0035] Furthermore, the repair generative adversarial network includes: a generative network, a discriminative network, and a loss function, where:
[0036] The generation network adopts the restoration network of encoder-decoder architecture;
[0037] The discriminant network adopts the PatchGAN network structure, takes the paired dataset as the input of the discriminator, and outputs a 32×32×1 feature matrix;
[0038] The loss functions include GAN loss function, L1 loss function and edge detection loss function.
[0039] According to another aspect of the present invention, there is provided a device comprising:
[0040] one or more processors;
[0041] a memory for storing one or more programs,
[0042] When the one or more programs are executed by the one or more processors, the one or more processors are caused to perform any of the above methods.
[0043] According to another aspect of the present invention, a computer-readable storage medium storing a computer program is provided, and when the program is executed by a processor, any of the above methods is implemented.
[0044] Compared with the prior art, the present invention has the following beneficial effects:
[0045] 1. The underwater image enhancement method exemplified in this invention can generate a realistic underwater simulation dataset from an atmospheric dataset, while also exhibiting good generalization. The proposed generative network-based underwater image restoration algorithm can restore realistic underwater images, and its edge detection method can restore detailed underwater image information, significantly improving the quality of underwater image restoration.
[0046] It can generate corresponding simulation paired data sets according to the actual underwater degradation quality without the need for parameter adjustment, and restore the underwater image color based on the underwater enhancement algorithm of multi-scale feature fusion.
[0047] 2. The underwater image enhancement system of the present invention can generate similar simulation data sets based on actual underwater scenes. At the same time, the data generated by the present invention based on the neural network has better generalization than the data generated based on the physical model.
[0048] Edge detection as a loss function enables the network model to pay more attention to the details of the image restoration, making the restored underwater images have higher quality.
[0049] 3. The device of the present invention implements the method described in any one of the above items to implement an underwater image enhancement method, which can accurately restore problems such as underwater image degradation.
[0050] 4. The computer-readable storage medium of the device of the present invention implements the method described in any one of the above items when the program is executed by the processor, and can accurately restore problems such as underwater image degradation. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 This is a flow chart of the image enhancement algorithm for complex environments according to the present invention;
[0052] Figure 2This is a schematic diagram of data augmentation based on a generative adversarial network in the present invention;
[0053] Figure 3 2 is a schematic diagram of underwater image simulation according to the present invention;
[0054] Figure 4 This is a schematic diagram of the underwater repair network architecture based on the generative adversarial network of the present invention;
[0055] Figure 5 Schematic diagram of the edge feature extraction network of the present invention;
[0056] Figure 6 This is the underwater image restoration effect diagram of the present invention;
[0057] Figure 7 It is a structural schematic diagram of the device of the present invention. DETAILED DESCRIPTION
[0058] In order to better understand the technical solution of the present invention, the present invention is further described below with reference to the accompanying drawings and specific embodiments.
[0059] Example 1
[0060] An underwater image enhancement system, comprising
[0061] The data acquisition module is configured to acquire underwater video or image data, where the underwater environment may be a marine environment; the data acquisition module may be an underwater camera device, preferably an underwater camera device having good camera function, and the acquired underwater images are high-resolution, high-definition, high-quality images captured by the device. Preferably, the acquired underwater video or image data is an RGB-D public dataset with depth in an atmospheric environment.
[0062] a data enhancement generative adversarial network module configured to preprocess the video or image data using a preset algorithm to obtain an attenuation map and a fog effect map, extract features from the attenuation map and the fog effect map to obtain underwater simulation data, and generate a paired data set by combining the underwater video or image data with the corresponding underwater simulation data;
[0063] The repair generative adversarial network module is configured to calculate the loss of the discriminator on the obtained paired dataset and train it to obtain an underwater image repair model.
[0064] Specifically, the data enhancement generative adversarial network module includes:
[0065] A preprocessing unit configured to preprocess the video or image data using a preset algorithm to obtain an attenuation map and a fogging effect map;
[0066] a simulation data generation unit configured to train the generated attenuation map and atomization map using a neural network, and extract attenuation and atomization features using a plurality of convolution modules to obtain underwater simulation data;
[0067] The data pairing unit is configured to generate a pairing data set by combining the underwater video or image data with the corresponding underwater simulation data.
[0068] Repairing the generative adversarial network includes: generating network, discriminant network and loss function, among which,
[0069] The generation network adopts the restoration network of encoder-decoder architecture;
[0070] The discriminant network adopts the PatchGAN network structure, takes the paired dataset as the input of the discriminator, and outputs a 32×32×1 feature matrix;
[0071] The loss functions include GAN loss function, L1 loss function and edge detection loss function.
[0072] This embodiment provides a method for an underwater image enhancement system, including the following steps:
[0073] Step 1: Obtain underwater video or image data.
[0074] Step 2: Generate an adversarial network through data enhancement, preprocess the video or image data using a preset algorithm to obtain an attenuation map and a fog effect map, extract features from the attenuation map and the fog effect map to obtain underwater simulation data, and generate a paired data set by combining the underwater video or image data with the corresponding underwater simulation data.
[0075] The algorithm formula for preprocessing the video or image data using a preset algorithm to obtain the attenuation map and the fog effect map is as follows:
[0076] I (x) =J (x) t (x) +B(1-t (x) )
[0077] t (x) =e -β(λ)d(x)
[0078] Where I(x) represents the generated simulated underwater image, J(x) is the image in the air environment, t(x) is the transmission map of the scene, B is the atmospheric ambient light of the scene, β is the attenuation coefficient of light of different wavelengths, and d(x) is the range between the scene and the camera.
[0079] Among them, the attenuation map and the atomization effect map are extracted to obtain underwater simulation data, including: using a neural network to train the generated attenuation map and the atomization effect map, and using several convolution modules to extract the features of attenuation and atomization to obtain an underwater simulation data set. Specifically, after obtaining the attenuation map and the atomization effect map, the generated attenuation map and the atomization map are trained using a neural network. The present invention uses four convolution modules to extract the features of attenuation and atomization. Each convolution module operation includes a convolution layer, an activation layer, a BN layer and a downsampling layer. After each convolution layer is extracted, the attenuation feature map and the atomization feature map are fused, and downsampling is performed using a convolution kernel of 1*1 and a step size of 2, wherein the convolution kernel of each layer is 3*3 in size, and the activation function uses the LeakyReLu function; after extraction to the fourth layer, 2 times upsampling is performed, and the results of the previous layer are merged using a jump connection, and the RGB underwater simulation image is finally output through continuous upsampling. Preferably, the generated simulated image and the real underwater image are used as the input of the discriminator. The discriminator adopts a full convolutional network. Using a full convolutional network as the discriminant network can process images of any resolution. The discriminant network structure has four convolutional layers. Each convolutional layer uses a 5*5 convolution kernel to extract features, then performs BN, and finally uses an activation function; after the last convolution layer is completed, a sigmoid function is used to output the result. In the present invention, after inputting the simulated image and the real underwater image, the real and real images are discriminated to obtain loss data. After the real image and the simulated image are discriminated to obtain loss data, the final loss result of the discriminator is the sum of the two parts.
[0080] In step 2 of the specific embodiment: Figure 2 First, the attenuation and fog images are obtained by the underwater image degradation algorithm, and then the underwater simulation image is generated by the neural network. The neural network architecture used here draws on the encoder-decoder idea. First, the attenuation map and the fog map are respectively extracted. Each layer of the encoder includes downsampling with a 1*1 convolution kernel with a step size of 2, and feature extraction with a 3*3 convolution kernel with a step size of 1 (the number of channels is doubled after each layer of convolution processing). Then, the LeakyRelu activation function and BN processing are used. After processing each layer, the fog map feature map and the attenuation feature map are added as the feature map required for the jump link; then the encoder is used for upsampling to output the underwater image simulation result, wherein each layer of the encoder includes upsampling with a 1*1 convolution kernel with a step size of 2, splicing with the encoder mirror layer feature map using jump links, and then convolution operation with a 3*3 convolution kernel followed by activation function processing; finally, the loss function is calculated by the discriminator, and then the network parameters are updated.
[0081] Step 3: Utilize a generative adversarial network (GAN) to perform discriminator loss calculations on the paired dataset and train the model to obtain an underwater image restoration model. A Unet network with an encoder-decoder structure can be employed for the GAN. The GAN is used to perform discriminator loss calculations on the paired dataset using a WGAN loss function, an L1 loss function, and an edge detection loss function. The edge detection loss function obtains edge detection information by grayscaling the image into three RGB channels, then fusing the three RGB channels for convolution, pooling, and activation operations to extract edge detection information from the generative network.
[0082] The expression of the underwater image restoration model is: in,
[0083]
[0084] L1=E x,y [||yG(x)||1]
[0085] L edge =E x,y [||F edge (y)-F edge (G(x))||2]
[0086] In the formula, x represents underwater video or image data, y represents underwater simulation dataset, G(x) represents the image generated based on the generative network, are samples along the straight line between point pairs from the true data distribution and the generator distribution, λ, λ1, λ2 represent weight coefficients, and their values are determined empirically.
[0087] To facilitate further understanding, the present invention is further explained below with reference to specific examples: 1. An enhanced generative adversarial network is used to generate a paired dataset.
[0088] First, the present invention adopts a generative adversarial network architecture to synthesize underwater simulation images. In the generator part, a general underwater image degradation algorithm is used to generate attenuation maps and scattering maps. The algorithm formula is shown in the following formula (1):
[0089] I (x) =J (x) t (x) +B(1-t (x) )
[0090] t (x) =e -β(λ)d(x)
[0091] In formula (1), I(x) represents the generated simulated underwater image, J(x) represents the image in the air environment, and t(x) represents the scene's transmission map. B represents the scene's atmospheric ambient light. β represents the attenuation coefficient for light of different wavelengths, and d(x) represents the range between the scene and the camera. The above algorithm generates an attenuation map and a corresponding fogging effect map.
[0092] Then, a neural network is used to train the generated attenuation map and fog map. The present invention adopts four convolution modules to extract the features of attenuation and fog. Each convolution module operation includes a convolution layer, an activation layer, a BN layer and a downsampling layer. After each convolution layer is extracted, the attenuation feature map and the fog feature map are fused, and downsampling is performed using a convolution kernel 1*1 with a step size of 2, wherein the convolution kernel of each layer is 3*3 in size, and the activation function adopts the LeakyReLu function; after extraction to the fourth layer, 2 times upsampling is performed, and the results of the previous layer are merged using a jump connection. By continuously upsampling, an RGB underwater simulation image is finally output.
[0093] Finally, the generated simulated image and the real underwater image are used as the input of the discriminator. The discriminator adopts a full-convolution network. Using a full-convolution network as the discriminant network can process images of any resolution. The discriminant network structure has four convolutional layers. Each convolutional layer uses a 5*5 convolution kernel to extract features, and then performs BN.
[0094] Finally, an activation function is used; after the last convolution layer is completed, a sigmoid function is used to output the result. In this invention, after inputting the simulated image and the real underwater image, the real image and the real image are discriminated to obtain loss data. The loss data is obtained by discriminating the real image and the simulated image again. The final loss result of the discriminator is the sum of the two parts.
[0095] 2. Restoring underwater images based on generative adversarial networks
[0096] 2.1 Generative Network: A Unet network with an encoder-decoder structure is used. A data domain X (underwater dataset) and a target domain (enhanced dataset) are set. The ultimate goal is to learn the mapping relationship G:X->Y from the data domain to the target domain. In order to automatically learn this mapping relationship, the present invention adopts the game concept of the fully convolutional generative adversarial network to learn the mapping relationship from the data domain to the target domain. In the generator part, the present invention proposes an optimized Unet underwater image restoration network, which uses an encoder network. The upsampling layer uses skip links to merge the mirror layers. The skip links are very effective for image-to-image style conversion. Specifically, a 7-layer convolutional structure is used for feature extraction to obtain more advanced feature information, and the residual is used to extract image features in each convolutional layer. The network input is a 256*256*3 color image, and the encoding layers d1-d7 learn feature maps of different sizes. The decoder uses these feature maps and skip links to generate a 256*256*3 RGB underwater restoration image. The network proposed in this invention is a full-convolution network that can process images of any resolution. The encoding layer of each layer includes a 4*4 convolution kernel, an activation function, and BN operations.
[0097] 2.2 Discriminant Network
[0098] The generated simulated image and the real underwater image are used as input to the discriminator. The discriminator adopts the PatchGAN network structure, which can process images of any size in a fully convolutional manner. The discriminator is modeled as a PatchGAN and performs discrimination at the image patch level. Unlike conventional discriminators, the PatchGAN discriminator outputs a 32×32×1 feature matrix, rather than a set of 0s or 1s, which allows for more detailed information.
[0099] 2.3 Loss Function
[0100] Since the original generative adversarial network has the problem of gradient vanishing and mode collapse in generating unique images during training, the present invention uses the optimized Wasserstein GAN algorithm, which proposes to use the Earth-Mover or Wasserstein-1 distance W by constructing a value function by utilizing the Kantorovich-Rubinstein duality. In this formula, given a set of k-Lipschitz functions f modeled as neural networks to approximate W, in order to ensure that f is a k-Lipschitz function, the weights of the discriminator are clipped to a certain range [-c, c]. This algorithm can effectively solve the above problems. The WGAN loss function of this algorithm is expressed as the following formula (2):
[0101]
[0102] In formula (2), x, y represent the underwater image and the restored image, G(x) represents the image generated based on the generative network, It is defined as the samples along the straight line between the point pairs from the real data distribution and the generator distribution. λ represents the weight coefficient. The parameter is mainly determined by experience. The best performance parameter in the experiment is: λ = 6.
[0103] This paper uses the L1 loss function to assist in generating adversarial networks. The L1 loss function formula is as follows:
[0104] L1=E x,y [||yG(x)||1]
[0105] In order to better solve the image blur problem caused by the L1 loss function, this paper proposes a novel algorithm for enhancing image details using edge detection.
[0106] First, the edge detection model is trained. Second, multi-scale edge detection is performed on the surface image. Multi-scale detection essentially involves continuously downsampling the generated and real images (typically five times at five scales or resolutions) to obtain images with multiple resolutions. The multi-scale edge detection images are then fused. Next, multi-scale edge detection is performed on the underwater simulated image restored by the generator, and the multi-scale edge detection results are fused. Finally, the loss value of the two images is calculated. Fusion of multi-scale information enables better restoration of image details.
[0107] The edge detection loss function is defined as follows:
[0108] L edge =E x,y [||F edge (y)-F edge (G(x))||2]
[0109] Finally, the present invention combines the above loss functions according to certain weight coefficients to form the final objective function. The final objective function formula is as follows:
[0110] Among them, λ1 and λ2 represent weight coefficients. The parameters are mainly determined by experience. The best performance parameters in the experiment are λ1=6 and λ2=12.
[0111] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0112] like Figure 1 As shown, the present invention proposes an image enhancement method in a complex ocean environment based on a dual-path generative adversarial network, comprising:
[0113] Step 1: Use underwater camera equipment to collect underwater videos and images to create a real underwater environment dataset, and obtain a public RGB-D dataset with depth under normal atmospheric environment.
[0114] Step 2: The obtained RGB-D dataset is preprocessed using the ocean image degradation algorithm to obtain the attenuation map and the fog effect map. Then a series of convolution operations are performed to extract features. Finally, the underwater simulation dataset is upsampled and the discriminator loss is calculated between the underwater simulation dataset and the underwater real dataset to update the network parameters.
[0115] Step 3: Generate an underwater simulation dataset and an original RGB-D public dataset with depth under normal atmospheric conditions using the model trained in step 2.
[0116] Step 4: Use the data restoration generative adversarial network to train the paired dataset obtained in step 3 to obtain the underwater image restoration model.
[0117] Step 5: Use the underwater restoration model obtained in step 4 to restore the real underwater image or video, and apply the restoration results to actual scenes.
[0118] The underwater image restoration method proposed in the present invention requires the construction of corresponding acquisition equipment and transmission systems, including a set of underwater camera acquisition equipment and image and video data transmission systems. In step 1 of the specific embodiment: the underwater camera equipment should have good camera functions, and the collected ocean underwater images are high-resolution, high-definition, high-quality images captured by it.
[0119] In step 2 of the specific embodiment: Figure 2 First, the attenuation and fog images are obtained by the ocean underwater image degradation algorithm, and then the underwater simulation image is generated by the neural network. The neural network architecture used here draws on the encoder-decoder idea. First, the attenuation map and the fog map are respectively extracted. Each layer of the encoder includes downsampling with a 1*1 convolution kernel with a step size of 2, and feature extraction with a 3*3 convolution kernel with a step size of 1 (the number of channels is doubled after each layer of convolution processing). Then, the LeakyRelu activation function and BN processing are used. After processing each layer, the fog map feature map and the attenuation feature map are added as the feature map required for the jump link; then the encoder is used to upsample and output the underwater image simulation result. Each layer of the encoder includes upsampling with a 1*1 convolution kernel with a step size of 2, splicing with the jump link and the encoder mirror layer feature map, and then convolution operation with a 3*3 convolution kernel followed by activation function processing; finally, the discriminator calculates the loss function and then updates the network parameters.
[0120] In step 3 of the specific embodiment: Figure 3, the underwater image simulation model trained in step 2 is used to process the dataset under normal atmospheric conditions. The processed results are shown in the reference Figure 3 As shown in the figure, (a) represents the initialized normal dataset source, (b) represents the attenuation map after being processed by the ocean underwater image degradation algorithm, (c) represents the fog map after being processed by the ocean underwater image degradation algorithm, (d) represents the simulation map processed by the neural network, and (e) is the real underwater image dataset.
[0121] By comparison, it can be seen that the underwater image simulation network proposed in the present invention can generate similar simulation data sets according to actual underwater scenes. At the same time, the data generated by the present invention based on the neural network has better generalization than the data generated based on the physical model.
[0122] In step 4 of the specific embodiment: Figure 4 First, the present invention adopts the Unet network with encoder-decoder structure. In order to improve the underwater image restoration capability, the present invention optimizes the network model structure on the one hand, and designs a new loss function on the other hand.
[0123] In terms of network structure, the network is deepened and feature maps of 7 scales are extracted. The multi-scale feature map extraction can effectively improve the learning ability of the network model. The Res residual structure is used to extract features at each scale. The Res residual structure includes:
[0124] (1) Convolution operation: The convolution kernel is 1*1 with a step size of 2 to perform downsampling operation and the output feature map is recorded as fm1. The convolution kernel is 3*3 with a step size of 1 to extract the feature map with the number of channels reduced by one half and the output feature map is recorded as fm2. The convolution kernel is 3*3 with a step size of 1 to double the number of channels and record it as fm3. The feature maps of fm1 and fm3 are added together to form the final output feature map of this layer scale.
[0125] (2) leakyRelu activation function and BN layer operation.
[0126] In terms of loss function, this paper uses edge detection to enhance the edge detail recovery ability of the underwater repair network. Figure 5 ,The present invention adopts an edge detection network model to extract ,edge information at each scale, and finally fuses the multi-scale edge ,information as the edge detection output result.
[0127] In step 5 of the specific embodiment: Figure 6 , use the underwater image restoration model trained in step 4 to restore the underwater image. The processed results are shown in the reference Figure 6As shown in the figure, A represents the degraded real underwater image, B represents the edge information extracted from the degraded underwater image, C represents the restored underwater image, and D represents the edge information extracted from the restored underwater image. This comparison shows that the underwater image restoration network proposed in this paper is capable of restoring actual underwater scenes. Furthermore, using edge detection as a loss function enables the network model to focus more on restoring image details, resulting in higher quality restored underwater images.
[0128] A device according to this embodiment includes: one or more processors;
[0129] a memory for storing one or more programs,
[0130] When the one or more programs are executed by the one or more processors, the one or more processors execute any of the above methods, which can repair the actual underwater scene. At the same time, using edge detection as a loss function can enable the network model to pay more attention to the details of the image restoration, so that the repaired underwater image has higher quality.
[0131] This embodiment includes a computer-readable storage medium storing a computer program. When executed by a processor, the program implements any of the aforementioned methods. The program also stores a method for detecting anomalies in ocean time-series observation data. The method can repair actual underwater scenes. Furthermore, using edge detection as a loss function enables the network model to focus more on image detail repair, resulting in higher-quality repaired underwater images. Further description follows:
[0132] The computer system includes a central processing unit (CPU) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage unit into a random access memory (RAM) 103. Various programs and data required for system operation are also stored in the RAM 103. The CPU 101, ROM 102, and RAM 103 are connected to each other via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.
[0133] The following components are connected to the I / O interface 105: an input section 106 including a keyboard, a mouse, and the like; an output section including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 108 including a hard disk and the like; and a communication section 109 including a network interface card such as a LAN card or a modem. The communication section 109 performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface 105 as needed. Removable media 511, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 510 as needed, so that computer programs read therefrom can be installed into the storage section 108 as needed.
[0134] In particular, according to an embodiment of the present invention, the above reference process Figure 1 The described process can be implemented as a computer software program. For example, embodiment 1 of the present invention includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication portion and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU) 101, the above-mentioned functions defined in the system of the present application are performed.
[0135] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.
[0136] Frames in the accompanying drawings Figure 7 , illustrates the possible implementation architecture, functions and operations of the system, method and computer program product according to various embodiments 1 of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0137] The units involved in the embodiments of the present invention may be implemented in software or in hardware, and the modules described may also be provided in a processor. The names of these module elements do not, in some cases, limit the units themselves. The units or modules described may also be provided in a processor. For example, they may be described as: an underwater image enhancement system, comprising: a data acquisition module, a data enhancement generative adversarial network module, and a repair generative adversarial network module. The names of these units may, in some cases, not limit the units themselves. For example, the data acquisition module may also be described as a "data acquisition module for acquiring underwater video or image data."
[0138] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments, or may exist independently and not be incorporated into the electronic device. The computer-readable medium carries one or more programs, and when executed by the electronic device, the electronic device implements the method for detecting anomalies in ocean time-series observation data as described in the above embodiments.
[0139] For example, the electronic device can implement: step S1: obtaining underwater video or image data; step S2: using a data enhancement generative adversarial network, preprocessing the video or image data using a preset algorithm to obtain an attenuation map and a fog effect map, extracting features from the attenuation map and the fog effect map to obtain underwater simulation data, and generating a paired data set for the underwater video or image data and the corresponding underwater simulation data; step S3: using a repair generative adversarial network to perform a discriminator calculation on the obtained paired data set and train it to obtain an underwater image repair model.
[0140] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.
[0141] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0142] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention herein is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the inventive concept. For example, the above-mentioned features may have similar functions to (but not limited to) those disclosed in this application.
[0143] The above data show that the underwater image enhancement method of the present invention can accurately restore underwater image degradation and other problems. Since the present invention has been described through the above embodiments, any equivalent replacement is obvious to the present invention and is included in the present invention.
[0144] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
[0145] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.
Claims
1. A method for underwater image enhancement, characterized in that: include: Acquiring underwater video or image data; Using a data-enhanced generative adversarial network, the video or image data is preprocessed using a preset algorithm to obtain an attenuation map and a fog effect map, features are extracted from the attenuation map and the fog effect map to obtain underwater simulation data, and a paired dataset is generated by combining the underwater video or image data with the corresponding underwater simulation data; The discriminator loss is calculated on the obtained paired dataset using the repair generative adversarial network and trained to obtain an underwater image repair model. Extracting features from the attenuation map and the atomization effect map to obtain underwater simulation data, including: using a neural network to train the generated attenuation map and the atomization effect map, and using a plurality of convolution modules to extract attenuation and atomization features to obtain an underwater simulation data set; Among them, four convolution modules are used to extract attenuation and fog features. Each convolution module operation includes a convolution layer, an activation layer, a batch normalization layer, and a downsampling layer. After each convolution layer is extracted, the attenuation feature map and the fog feature map are fused, and downsampling is performed using a convolution kernel of 1*1 with a step size of 2. The convolution kernel of each layer is 3*3 in size, and the activation function uses the LeakyReLu function. After extracting to the fourth layer, 2 times upsampling is performed, and the results of the previous layer are merged using a jump connection. Through continuous upsampling, an RGB underwater simulation image is finally output. Use the repaired generative adversarial network to calculate the discriminator loss of the obtained paired dataset, including the WGAN loss function, L1 loss function, and edge detection loss function; The edge detection loss function obtains edge detection information by dividing the image into three RGB channels for grayscale processing, then fusing the three RGB channels for convolution, pooling and activation operations, and extracting the edge detection information of the generated network.
2. The underwater image enhancement method according to claim 1, characterized in that: The algorithm formula for preprocessing the video or image data using a preset algorithm to obtain the attenuation map and the fog effect map is as follows: Where I(x) represents the generated simulated underwater image, J(x) is the image in the air environment, t(x) is the transmission map of the scene, B is the atmospheric ambient light of the scene, β is the attenuation coefficient of light of different wavelengths, and d(x) is the range between the scene and the camera.
3. The underwater image enhancement method according to claim 1, characterized in that: The expression of the underwater image restoration model is: ,in, In the formula, x represents underwater video or image data, y represents underwater simulation dataset, G(x) represents the image generated based on the generative network, are samples along the straight line between point pairs from the true data distribution and the generator distribution, and λ, λ1, λ2 represent weight coefficients.
4. A device comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to perform the method according to any one of claims 1 to 3.