Image Enhancement Method, Device, Terminal Device, and Storage Medium

Abstract

The present invention discloses an image enhancement method, device, terminal device, and storage medium. The image enhancement method includes: obtaining image data to be enhanced; inputting the image data to be enhanced into a pre-created image enhancement model for enhancement to obtain enhanced image data, where the image enhancement model is trained based on a generative adversarial network and combined with a preset enhanced image algorithm. The present invention solves the problem of low real-time performance of image processing capabilities and improves the image enhancement efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to an image enhancement method, apparatus, terminal device, and storage medium. Background Art

[0002] In fields such as underwater archaeology and seabed exploration that require high-quality underwater images, there are high requirements for the image processing capabilities of underwater image processing equipment. The lightweight net underwater image real-time enhancement technology is a key technology for improving the camera sensing distance, feature extraction, and visual positioning capabilities of underwater inspection robots.

[0003] However, in the process of processing underwater images, if traditional unsupervised image enhancement methods are used alone, there are problems such as large computational complexity, complex operations, and inability to be deployed on the GPU when processing a large amount of image data; if convolutional neural network-based methods are used alone, it is required that the underwater image dataset has both underwater distorted images and clear images, and it is also required to collect images in water and in air at the same position and with the same parameters.

[0004] Therefore, the existing technology has low real-time performance for image processing capabilities and is not convenient for real-time image processing on underwater robots; for collecting underwater image datasets, clear images cannot be obtained in many cases. Summary of the Invention

[0005] The main purpose of the present invention is to provide an image enhancement method, apparatus, terminal device, and storage medium, aiming to improve the real-time processing of underwater images and the preprocessing capabilities of large data streams, and enhance the image enhancement efficiency.

[0006] To achieve the above object, the present invention provides an image enhancement method, which includes the following steps:

[0007] Obtain the image data to be enhanced;

[0008] Input the image data to be enhanced into a pre-created image enhancement model for enhancement to obtain enhanced image data, where the image enhancement model is trained based on a generative adversarial network and combined with a preset enhanced image algorithm.

[0009] Optionally, before the step of inputting the data to be enhanced into a pre-created image enhancement model for enhancement to obtain enhanced data, it further includes:

[0010] Create the image enhancement model, specifically including:

[0011] Obtain the original dataset;

[0012] Establish a generator;

[0013] Build a discriminator;

[0014] Based on the generator and the discriminator, construct a generative adversarial network;

[0015] Based on the multi-scale homomorphic filtering labels and the original dataset, train the generative adversarial network to obtain the image enhancement model.

[0016] Optionally, after the step of building the generator, the following steps are further included:

[0017] Input a first parameter into the generator to initialize the generator;

[0018] Input the original dataset into the generator to obtain a corresponding pseudo-enhanced dataset;

[0019] The step of training the generative adversarial network based on the multi-scale homomorphic filtering labels and the original dataset to obtain the image enhancement model includes:

[0020] Based on the multi-scale homomorphic filtering labels, the original dataset and the pseudo-enhanced dataset, train the generative adversarial network to obtain the image enhancement model.

[0021] Optionally, the step of training the generative adversarial network based on the multi-scale homomorphic filtering labels, the original dataset and the pseudo-enhanced dataset to obtain the image enhancement model includes:

[0022] Based on the multi-scale homomorphic filtering labels, enhance the data in the original dataset to obtain a corresponding enhanced dataset;

[0023] Based on the original dataset, the enhanced dataset and the pseudo-enhanced dataset, train the generative adversarial network to obtain a trained generative adversarial network.

[0024] Optionally, the step of enhancing the data in the original dataset based on the multi-scale homomorphic filtering labels to obtain a corresponding enhanced dataset includes:

[0025] Calculate the data in the original dataset relative to the corresponding noise reduction term for denoising the data in the original dataset;

[0026] Calculate the spatial distance weight value from the pixel point to the center point and the adjacent pixel value weight of the data in the original dataset;

[0027] Based on the adjacent pixel value weight and the spatial distance weight value, calculate the pixel weight sum;

[0028] Based on the pixel weights sum and the noise reduction term, a reflected image is calculated;

[0029] Based on the reflected image and a preset Gaussian function, a reflection response image is calculated for multi-scale spatial construction of the data in the original dataset;

[0030] Based on the reflection response image and the noise reduction term, the data in the original dataset relative to the corresponding enhanced data is calculated, and so on, to obtain the enhanced dataset.

[0031] Optionally, the step of training the generative adversarial network based on the original dataset, the enhanced dataset, and the pseudo-enhanced dataset to obtain the trained generative adversarial network includes:

[0032] The data in the original dataset and the corresponding data in the enhanced dataset are combined to obtain an image pair dataset;

[0033] The enhanced dataset and the pseudo-enhanced dataset are input into the discriminator for the discriminator to discriminate between the enhanced dataset and the pseudo-enhanced dataset to obtain a discrimination result;

[0034] The discriminator is trained and updated in combination with the discrimination result;

[0035] The image pair dataset is input into the generator for the generator to calculate the image pair dataset to obtain a first calculation result;

[0036] The generator is trained and updated in combination with the first calculation result;

[0037] The original dataset is input into the generator for the generator to calculate the original dataset to obtain a second calculation result;

[0038] The second result is used as the pseudo-enhanced dataset; and return to execute the step: input the enhanced dataset and the pseudo-enhanced dataset into the discriminator for the discriminator to discriminate between the enhanced dataset and the pseudo-enhanced dataset to obtain a discrimination result;

[0039] Loop according to a preset number of loops until the loop is completed, terminate the training, and obtain the trained image enhancement model.

[0040] Optionally, the step of inputting the image pair dataset into the generator for the generator to calculate the image pair dataset to obtain a first calculation result includes:

[0041] Input the image pair dataset into the generator, and calculate the adversarial loss function based on the image pair dataset and the expected value of the distribution function.

[0042] Extract the output features of the corresponding levels from the feature maps of several levels through the multi-level image block network, and calculate the features of the corresponding levels.

[0043] Calculate the maximum mutual information and the features of the corresponding levels of the feature maps based on the noise contrast estimation framework to obtain the noise contrast estimation repair function.

[0044] Calculate the total loss function based on the adversarial loss function and the noise contrast estimation repair function.

[0045] Calculate the second parameter based on the gradient descent algorithm and the total loss function.

[0046] Take the second parameter as the first parameter; and return to execute the step: calculate the second parameter based on the gradient descent algorithm and the total loss function.

[0047] Repeat this cycle, perform parameter iteration according to the preset number of iterations until the iteration is completed, and take the first parameter as the first calculation result.

[0048] An embodiment of the present invention also proposes an image enhancement device, and the image data enhancement device includes:

[0049] An acquisition module for acquiring the image data to be enhanced.

[0050] An enhancement module that inputs the image data to be enhanced into a pre-created image enhancement model for enhancement to obtain the enhanced image data, where the image enhancement model is trained based on a generative adversarial network and combined with a preset enhanced image algorithm.

[0051] An embodiment of the present invention also proposes a terminal device, and the terminal device includes: a memory, a processor, and an image enhancement program stored on the memory and executable on the processor, and the image enhancement program is configured to implement the steps of the above-mentioned image enhancement method.

[0052] An embodiment of the present invention also proposes a storage medium, and an image enhancement program is stored on the storage medium, and when the image enhancement program is executed by a processor, it implements the steps of the above-mentioned image enhancement method.

[0053] The image enhancement method, device, terminal device, and storage medium proposed in the embodiments of the present application obtain the image data to be enhanced; input the image data to be enhanced into a pre-created image enhancement model for enhancement to obtain the enhanced image data, where the image enhancement model is trained based on a generative adversarial network in combination with a preset enhanced image algorithm. Through the trained image enhancement model, the real-time processing of underwater images and the preprocessing ability of large data streams can be improved, and the image enhancement efficiency can be enhanced. Starting from the problem of low real-time performance of image processing capabilities, this embodiment takes the generative adversarial network image enhancement based on multi-scale homomorphic filtering labels as the research object, deeply analyzes the advantages and disadvantages of traditional image enhancement technologies, designs a generative adversarial network image enhancement model based on multi-scale homomorphic filtering labels, and verifies the effectiveness of the image enhancement method proposed in the present invention on this image enhancement model. Finally, the real-time performance of the image enhancement of the solution of the present invention has been significantly improved. Description of the Drawings

[0054] Figure 1 It is a schematic diagram of the functional modules of the terminal device to which the image enhancement device of the present invention belongs;

[0055] Figure 2 It is a schematic flowchart of the first exemplary embodiment of the image enhancement method of the present invention;

[0056] Figure 3 It is a schematic diagram of the overall data flow involved in the first exemplary embodiment of the image enhancement method of the present invention;

[0057] Figure 4 It is a schematic flowchart of another exemplary embodiment of the image enhancement method of the present invention;

[0058] Figure 5 It is a schematic flowchart of a training process of an image enhancement model involved in the embodiments of the image enhancement method of the present invention;

[0059] Figure 6 It is Figure 5 a schematic flowchart of the refinement process of step S51 in

[0060] Figure 7 It is Figure 6 a schematic flowchart of the refinement process of step S52 in

[0061] Figure 8 It is Figure 6 a schematic flowchart of the refinement process of step S603 in

[0062] The realization, functional features, and advantages of the object of the present invention will be further described in conjunction with the embodiments with reference to the drawings. Detailed Embodiments

[0063] It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0064] Technical terms related to the embodiments of the present invention:

[0065] Generative adversarial network, GAN, generative adversarial networks;

[0066] Graphics Processing Unit, GPU;

[0067] Generator;

[0068] Discriminator;

[0069] Encoder;

[0070] Decoder;

[0071] Multi-scale space;

[0072] Homomorphic filtering;

[0073] Multi-layer Perceptron, MLP.

[0074] Among them, the generative adversarial network (also called generative adversarial network, GAN, Generative Adversarial Networks) is a deep learning model and one of the most promising methods for unsupervised learning on complex distributions in recent years. The generative adversarial network model generates quite good outputs through the mutual game learning of (at least) two modules in the framework: the generative model (Generative Model, or Generator) and the discriminative model (Discriminative Model, or Discriminator) (in the embodiments of this application, there are three modules: encoder, generator, and discriminator). In the original GAN theory, it is not required that both the generative model and the discriminative model are neural networks, as long as they are functions that can fit the corresponding generation and discrimination. However, in practice, deep neural networks are generally used as the generative model and the discriminative model. An excellent GAN application requires a good training method, otherwise, the output may be unsatisfactory due to the freedom of the neural network model.

[0075] Ian J. Goodfellow et al. proposed a new framework for estimating generative models through an adversarial process in Generative Adversarial Networks in October 2014. In the framework, two models are trained simultaneously: a generative model that captures the data distribution, and a discriminative model that estimates the probability that a sample comes from the training data. The training procedure of the generative model is to maximize the probability of the discriminative model making mistakes.

[0076] Generally, machine learning models can be roughly divided into two categories: generative models and discriminative models. Discriminative models require input variables and predict through a certain model. Generative models generate observed data randomly given some hidden information. For example:

[0077] Discriminative model: Given a picture, determine whether the animal in the picture is a cat or a dog;

[0078] Generative model: Given a series of pictures of cats, generate a new cat (not in the dataset).

[0079] For discriminative models, the loss function is easy to define because the output target is relatively simple. But for generative models, it is not so easy to define the loss function. So, it is better to let the discriminative model handle the feedback part of the generative model. This tightly combines the two major types of models, Generative and Discriminative, in machine learning.

[0080] The basic principle of GAN is illustrated as follows using generating pictures as an example:

[0081] Suppose there are two networks, G (Generator) and D (Discriminator). Their functions are as follows:

[0082] G is a network that generates pictures. It receives a random noise z and generates pictures through this noise, denoted as G(z).

[0083] D is a discriminative network that determines whether a picture is "real". Its input parameter is x, where x represents a picture, and the output D(x) represents the probability that x is a real picture. If it is 1, it means the picture is 100% real, and if the output is 0, it means the picture cannot be real.

[0084] During the training process, the goal of the generative network G is to generate as realistic images as possible to deceive the discriminative network D. The goal of D, on the other hand, is to distinguish between the images generated by G and the real images. In this way, G and D constitute a dynamic "game process". In the most ideal state, the final result of the game is that G can generate images G(z) that are "indistinguishable from the real ones". For D, it is difficult to determine whether the images generated by G are real or not.

[0085] In this way, the goal is achieved: a generative model G is obtained, which can be used to generate images. Goodfellow theoretically proved the convergence of the algorithm, and when the model converges, the generated data has the same distribution as the real data (ensuring the model effect).

[0086] Multi-scale space: Embed the original signal into a series of signals obtained by a single-parameter transformation. Each signal obtained by the transformation corresponds to a parameter in the single-parameter family. An important requirement is that the coarser scales in the multi-scale representation should be simplifications of the finer scales, and the coarser scales are obtained by smoothing the finer-scale images in a certain fixed way. To satisfy this property, there are various implementation methods. However, one thing remains the same, that is, the Gaussian function is the only available smoothing function.

[0087] There are various ways to implement multi-scale representation. For example, in the early stage, quadtrees or octrees, as well as image pyramids, were used. The pyramid is an image representation method that combines downsampling operations and smoothing operations. A great advantage of it is that the number of pixels in each layer from bottom to top continuously decreases, which greatly reduces the computational amount; the disadvantage is that the quantization of the pyramid becomes coarser and coarser from bottom to top, and the speed is very fast. (It should be emphasized that the pyramid construction method here is similar to the construction method of the wavelet pyramid. After smoothing a layer of the image, downsampling is performed. The purpose of smoothing is to make the pixel points after downsampling better represent the pixel points of the original image, which is completely different from the purpose of smoothing in multi-scale representation).

[0088] The quadtree or octree and pyramid representation methods mentioned above are rather crude in the steps taken to obtain multiple scales, and the "interval" between scales is too large. The "Scale-Space" representation method to be mentioned here is another effective method for multi-scale representation. Its scale parameter is continuous, and the number of spatial sampling points at all scales is the same (in fact, what is obtained at one scale is an image, and the scale-space sampling points are the pixel points of the image at that scale. That is to say, the resolution of the images at all scales in the scale-space representation method is the same). The main idea of the scale-space representation is to generate a series of signals from the original signal (such as an image) and use these signals to represent the original signal. In this process, the information at fine scales is gradually smoothed out (it can be considered that the detailed information is discarded).

[0089] The multi-scale space is obtained by convolving the original image with a two-dimensional Gaussian function. By continuously changing the parameter t, continuously changing images are obtained. The information of these images gradually decreases compared with the original image, and the detailed information is gradually smoothed out, but the number of pixels remains unchanged, that is, the resolution remains unchanged. The image pyramid reduces the resolution by reducing the number of pixels in several rows each time, resulting in a reduction of image information. The two are different.

[0090] Homomorphic filtering: A technique widely used in signal and image processing. The original signal is transformed through a non-linear mapping to a different domain where a linear filter can be used, and then mapped back to the original domain after the operation. The property of homomorphism is to keep the relevant attributes unchanged, and the advantage of homomorphic filtering is to transform the originally complex operation into an operation with the same efficiency but relatively simpler. This concept was proposed by Thomas Stockham, Alan V. Oppenheim, and Ronald W. Schafer at the Massachusetts Institute of Technology in the 1960s.

[0091] Homomorphic filtering can be used to remove multiplicative noise, and can simultaneously increase the contrast and standardize the brightness, thereby achieving the purpose of image enhancement.

[0092] An image can be expressed as the product of its illumination component and reflectance component. Although these two are inseparable in the time domain, they can be linearly separated in the frequency domain through Fourier transform. Since the illumination can be regarded as the lighting in the environment and has relatively little change, it can be regarded as the low-frequency component of the image; while the reflectance has relatively large changes and can be regarded as the high-frequency component. By separately processing the effects of illumination and reflectance on the pixel gray value, usually by means of a high-pass filter, the illumination of the image is made more uniform, achieving the purpose of enhancing the detailed features in the shadow area.

[0093] MLP Multi - layer Perceptron: It is a forward - structured artificial neural network ANN that maps a set of input vectors to a set of output vectors. An MLP can be regarded as a directed graph composed of multiple node layers, with each layer fully connected to the next layer. Except for the input nodes, each node is a neuron with a non - linear activation function. The MLP is trained using a supervised learning method with the BP backpropagation algorithm. The MLP is a generalization of the perceptron, overcoming the weakness of the perceptron that it cannot recognize linearly inseparable data.

[0094] Compared with the single - layer perceptron, in addition to the input and output layers, the MLP multi - layer perceptron can have multiple hidden layers in the middle. The simplest MLP contains only one hidden layer, that is, a three - layer structure. The layers of the multi - layer perceptron are fully connected between layers. The bottom layer of the multi - layer perceptron is the input layer, the middle is the hidden layer, and the last is the output layer. In the embodiments of the present invention, it is considered that in the process of underwater image processing, if only traditional unsupervised image enhancement methods are used, when dealing with a large amount of image data, there are problems such as large computational complexity, complex operations, and inability to be deployed on the GPU; if only convolutional neural network - based methods are used, it is required that the underwater image dataset contains both underwater distorted images and clear images, and it is also required to collect images with and without water at the same position and the same parameters.

[0095] Therefore, in the embodiments of the present invention, starting from the problem of low real - time performance of image processing capabilities, a generative adversarial network is combined with traditional unsupervised image enhancement methods to design an image enhancement model based on a generative adversarial network with multi - scale homomorphic filtering labels, improving the real - time processing of underwater images and the pre - processing ability of large data streams, and enhancing the image enhancement efficiency.

[0096] Specifically, referring to Figure 1 , Figure 1 is a schematic diagram of the functional modules of the terminal device to which the image enhancement device of this application belongs. The image enhancement device can be a device independent of the terminal device that can perform image processing and network model training, and it can be carried on the terminal device in the form of hardware or software. The terminal device can be an intelligent mobile terminal with data - processing functions such as a mobile phone or a tablet computer, or a fixed terminal device or a server with data - processing functions, etc.

[0097] In this embodiment, the terminal device to which the image enhancement device belongs at least includes an output module 110, a processor 120, a memory 130, and a communication module 140.

[0098] The operating system and the image enhancement program are stored in the memory 130. The image enhancement device can store the to-be-enhanced image data obtained, the enhanced image data obtained by enhancing the to-be-enhanced image data through the image enhancement network model, and the original data set obtained; input the original data set into the generator to obtain the corresponding pseudo-enhanced data set; enhance the data in the original data set based on the multi-scale homomorphic filtering label to obtain the corresponding enhanced data set; combine the data in the original data set and the corresponding image data in the enhanced data set to obtain the image pair data set; input the image pair data set into the generator for the generator to calculate the image pair data set to obtain the first calculation result; input the original data set into the generator for the generator to calculate the original data set to obtain the second calculation result, etc. information is stored in the memory 130; the output module 110 can be a display screen, etc. The communication module 140 can include a WIFI module, a mobile communication module, a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.

[0099] Among them, when the image enhancement program in the memory 130 is executed by the processor, the following steps are implemented:

[0100] Obtain the to-be-enhanced image data;

[0101] Input the to-be-enhanced image data into a pre-created image enhancement model for enhancement to obtain the enhanced image data, where the image enhancement model is trained based on a generative adversarial network and combined with a preset enhanced image algorithm.

[0102] Further, when the image enhancement program in the memory 130 is executed by the processor, the following steps are also implemented:

[0103] Create the image enhancement model, specifically including:

[0104] Obtain the original data set;

[0105] Establish a generator;

[0106] Establish a discriminator;

[0107] Based on the generator and the discriminator, construct a generative adversarial network;

[0108] Based on the multi-scale homomorphic filtering label and the original data set, train the generative adversarial network to obtain the image enhancement model.

[0109] Further, when the image enhancement program in the memory 130 is executed by the processor, the following steps are also implemented:

[0110] Input a first parameter into the generator to initialize the generator;

[0111] Input the original dataset into the generator to obtain a corresponding pseudo-augmented dataset;

[0112] The steps of training the generative adversarial network based on the multi-scale homomorphic filtering labels and the original dataset to obtain the image enhancement model include:

[0113] Train the generative adversarial network based on the multi-scale homomorphic filtering labels, the original dataset, and the pseudo-augmented dataset to obtain the image enhancement model.

[0114] Further, when the image enhancement program in the memory 130 is executed by the processor, the following steps are also implemented:

[0115] Enhance the data in the original dataset based on the multi-scale homomorphic filtering labels to obtain a corresponding enhanced dataset;

[0116] Train the generative adversarial network based on the original dataset, the enhanced dataset, and the pseudo-augmented dataset to obtain a trained generative adversarial network.

[0117] Further, when the image enhancement program in the memory 130 is executed by the processor, the following steps are also implemented:

[0118] Calculate the data in the original dataset relative to the corresponding noise reduction term for denoising the data in the original dataset;

[0119] Calculate the spatial distance weight value from the pixel point to the center point and the adjacent pixel value weight of the data in the original dataset;

[0120] Calculate the pixel weight sum based on the adjacent pixel value weight and the spatial distance weight value;

[0121] Calculate a reflected image based on the pixel weight sum and the noise reduction term;

[0122] Calculate a reflected response image based on the reflected image and a preset Gaussian function for multi-scale spatial construction of the data in the original dataset;

[0123] Calculate the data in the original dataset relative to the corresponding enhanced data based on the reflected response image and the noise reduction term, and so on, to obtain the enhanced dataset.

[0124] Further, when the image enhancement program in the memory 130 is executed by the processor, the following steps are also implemented:

[0125] Combine the data in the original dataset and the corresponding data in the augmented dataset to obtain an image pair dataset;

[0126] Input the augmented dataset and the pseudo-augmented dataset into the discriminator for the discriminator to discriminate the augmented dataset and the pseudo-augmented dataset to obtain a discrimination result;

[0127] Train and update the discriminator in combination with the discrimination result;

[0128] Input the image pair dataset into the generator for the generator to calculate the image pair dataset to obtain a first calculation result;

[0129] Train and update the generator in combination with the first calculation result;

[0130] Input the original dataset into the generator for the generator to calculate the original dataset to obtain a second calculation result;

[0131] Use the second result as the pseudo-augmented dataset; and return to execute the step: input the augmented dataset and the pseudo-augmented dataset into the discriminator for the discriminator to discriminate the augmented dataset and the pseudo-augmented dataset to obtain a discrimination result;

[0132] Loop according to a preset number of loop times until the loop is completed and terminate the training to obtain a trained image enhancement model.

[0133] Further, when the image enhancement program in the memory 130 is executed by the processor, the following steps are also implemented:

[0134] Input the image pair dataset into the generator and calculate an adversarial loss function based on the image pair dataset and the expected value of the distribution function;

[0135] Extract the output features of the corresponding levels from the feature maps of several levels through a multi-level image block network and calculate the features of the corresponding levels;

[0136] Calculate the maximum mutual information and the features of the corresponding levels of the feature map based on the noise contrast estimation framework to obtain a noise contrast estimation repair function;

[0137] Calculate a total loss function based on the adversarial loss function and the noise contrast estimation repair function;

[0138] Calculate a second parameter based on the gradient descent algorithm and the total loss function;

[0139] Use the second parameter as the first parameter; and return to the execution step: Calculate the second parameter based on the gradient descent algorithm and the total loss function;

[0140] Repeat this loop, perform parameter iteration according to the preset number of iterations until the iteration is completed, and use the first parameter as the first calculation result.

[0141] An embodiment of the present invention provides an image enhancement method. Refer to Figure 2 , Figure 2 which is a schematic flowchart of the first embodiment of an image enhancement method of the present invention.

[0142] In this embodiment, the image enhancement method includes:

[0143] Step S1001: Obtain the image data to be enhanced;

[0144] Step S1002: Input the image data to be enhanced into a pre-created image enhancement model for enhancement to obtain the enhanced image data, where the image enhancement model is trained based on a generative adversarial network and combined with a preset enhanced image algorithm.

[0145] The execution subject of the method in this embodiment can be an image enhancement device, or an image enhancement terminal device or server. In this embodiment, an image enhancement device is taken as an example. This image enhancement device can be integrated on terminal devices such as a computer, a smart phone, and a tablet computer with data processing functions, and is applicable to the research and development of the front end of computer vision image processing.

[0146] This embodiment mainly realizes image enhancement, especially the real-time performance of image enhancement, improves the real-time processing of underwater images and the preprocessing ability of large data streams, and improves the image enhancement efficiency.

[0147] Specifically, in underwater archaeology, seabed exploration, etc., underwater images with high imaging quality are required. The lightweight net underwater image real-time enhancement technology is the key technology to improve the camera perception distance, feature extraction and visual positioning ability of underwater inspection robots. Therefore, it is necessary to obtain the image data to be enhanced, input the data to be enhanced into a pre-created image enhancement model for enhancement to obtain the enhanced image data, so as to obtain underwater images with high quality. Among them, the pre-created image enhancement model is trained based on a generative adversarial network and combined with a preset enhanced image algorithm.

[0148] Among them, the preset enhanced image algorithm can enhance the image data and greatly reduce the difficulty of collecting the data set. Among them, the preset enhanced image algorithm can be the improved Multi-Scale Retinex algorithm; or non-physical model methods such as histogram equalization, gray world assumption, contrast limited histogram equalization, multi-scale retina enhancement with color restoration, automatic white balance, color constancy, wavelet transform, etc.; or restoring underwater images by assuming conditional inversion, restoring images through scene statistical priors; some are based on the optical properties of underwater imaging, such as improving the restoration of images through the atmospheric turbulence model, restoring images by designing a new underwater imaging model, restoring images by considering the characteristics of the underwater imaging model, and physical model-based methods such as image dehazing.

[0149] In this embodiment, specifically, the underwater image restored based on the improved Multi-Scale Retinex algorithm is used as the training label, and the neural network Contrastive Unpaired Translation Model is used to realize real-time enhancement of the lightweight netting underwater image.

[0150] More specifically, starting from the problem of low real-time performance of image processing capabilities, this embodiment takes the generative adversarial network image enhancement based on multi-scale homomorphic filtering labels as the research object, deeply analyzes the advantages and disadvantages of traditional image enhancement technologies, and designs a generative adversarial network image enhancement model based on multi-scale homomorphic filtering labels. The image enhancement model is trained based on the generative adversarial network with multi-scale homomorphic filtering labels.

[0151] In this embodiment, the image data to be enhanced is obtained; the image data to be enhanced is input into a pre-created image enhancement model for enhancement to obtain the enhanced image data, where the image enhancement model is trained based on the generative adversarial network and combined with the preset enhanced image algorithm. By using the generative adversarial network to enhance low-light images, the dependence on the prior knowledge of the image in the traditional method can be eliminated; through the generative adversarial network, the mutual conversion between images is realized in an unpaired manner, and paired images corresponding one by one are not required; the generative adversarial network can also arrange the image enhancement model on the GPU and use the GPU for calculation to realize the lightweight processing of the underwater original image; by combining the generative adversarial network with the traditional unsupervised enhanced image algorithm, the difficulty of collecting the data set is effectively reduced, the lightweight processing of the underwater original image is realized, and the real-time processing of underwater images and the preprocessing ability of large data streams are significantly improved. The present invention is applicable to the research and development of the front end of computer vision image processing, and has important engineering value and theoretical guiding significance for improving the efficiency of image enhancement, image recognition, image restoration, etc. under large data sets.

[0152] In this embodiment, a picture restoration network model is used to enhance an image. The framework of the image enhancement network model includes: a generative adversarial network, where the generative adversarial network includes a generator and an encoder. The image enhancement model is trained based on a generative adversarial network with multi-scale homomorphic filtering labels. The overall data flow of the network is as Figure 3 shown:

[0153] The original data set is denoised, multi-scale space constructed, and image enhanced through traditional image processing methods (multi-scale homomorphic filtering label algorithm) to obtain a corresponding enhanced data set;

[0154] The generator is used to reconstruct the input original data set to obtain a pseudo-enhanced data set and provide it to the discriminator;

[0155] The generator is also used to calculate the input original data set and the corresponding enhanced data set to obtain a calculation result, train the generator in combination with the calculation result, and provide the calculation result to the discriminator for the discriminator to discriminate the calculation result;

[0156] The discriminator is used to discriminate the input enhanced data set and pseudo-enhanced data set to obtain a discrimination result, and train the discriminator in combination with the discrimination result;

[0157] Based on the trained generator and the trained discriminator, an image data processing end (image enhancement model) is constructed;

[0158] The image data processing end is used to process the input underwater original image data to obtain an underwater image enhanced data set.

[0159] Refer to Figure 4 , Figure 4 which is a schematic flowchart of another exemplary embodiment of the image enhancement method of the present invention. Based on the above Figure 2 shown embodiment, in this embodiment, before the step S1002 of inputting the data to be enhanced into a pre-created image enhancement model for enhancement to obtain enhanced image data, the image enhancement method further includes:

[0160] Creating the image enhancement model specifically includes:

[0161] Step S10: Obtain the original data set;

[0162] Step S20: Establish a generator;

[0163] Step S30: Establish a discriminator;

[0164] Step S40: Based on the generator and the discriminator, construct a generative adversarial network;

[0165] Step S50: Based on the multi-scale homomorphic filtering labels and the original data set, train the generative adversarial network to obtain the image enhancement model.

[0166] In this embodiment, steps S10 to S50 are implemented before step S1001. In other embodiments, steps S10 to S50 can also be implemented between step S1001 and step S1002.

[0167] Compared with the above Figure 2 illustrated embodiment, this embodiment further includes a solution for training an image enhancement model.

[0168] Specifically, in this embodiment, a number of random image data are collected in advance to form an original data set. For example, the original data set is used to train the image enhancement model.

[0169] Then, an image generator G(r) is established as follows. G(r) is composed of an encoder G dec (r) and a decoder G enc (r).

[0170]

[0171] In the formula, is the pseudo-enhanced image generated by the original image data f under the generator G(f). The generator is used to calculate the input image data, obtain the calculation result, train and update the generator in combination with the calculation result, and provide the calculation result to the discriminator for the discriminator to discriminate the calculation result.

[0172] Then, a discriminator is established. The discriminator is used to discriminate the input image data. When the input image data is enhanced image data, the discriminator outputs a high score (close to 1). When the input data is pseudo-enhanced image data, the discriminator outputs a low score (close to 0) to obtain the discrimination result, and train and update the discriminator in combination with the discrimination result.

[0173] Then, based on the generator and the discriminator, a generative adversarial network is constructed.

[0174] Finally, based on the multi-scale homomorphic filtering labels, the original data set is enhanced to obtain an enhanced data set, and the generative adversarial network is trained to obtain the image enhancement model.

[0175] After that, the original data set can be enhanced by the trained image enhancement model.

[0176] In this embodiment, through the above solution, specifically, the original data set is obtained; a generator is established; a discriminator is established; based on the generator and the discriminator, a generative adversarial network is constructed; based on the multi-scale homomorphic filtering labels and the original data set, the generative adversarial network is trained to obtain the image enhancement model. By using the generative adversarial network to enhance low-light images, the dependence on prior knowledge of images in traditional methods can be eliminated; by introducing an enhanced image algorithm of multi-scale homomorphic filtering, the difficulty of collecting data sets can be effectively reduced; by combining the generative adversarial network with traditional unsupervised multi-scale homomorphic filtering labels, the image enhancement model can be deployed on the GPU and calculated using the GPU to achieve lightweight processing of underwater original images. Through the trained image enhancement model, the real-time processing of underwater images and the preprocessing ability of large data streams can be improved, and the image enhancement efficiency can be enhanced.

[0177] Referring to Figure 5 , Figure 5 is a schematic diagram of a training process of an image enhancement model related to an embodiment of the image enhancement method of the present invention. Based on the above Figure 4 shown embodiment, in this embodiment, after step S20 of establishing the generator, the following steps are further included:

[0178] Step S21: Input a first parameter into the generator to initialize the generator;

[0179] Step S22: Input the original data set into the generator to obtain a corresponding pseudo-enhanced data set;

[0180] The step S50 of training the generative adversarial network based on the multi-scale homomorphic filtering labels and the original data set to obtain the image enhancement model includes:

[0181] Step S51: Train the generative adversarial network based on the multi-scale homomorphic filtering labels, the original data set and the pseudo-enhanced data set to obtain the image enhancement model.

[0182] Specifically, in this embodiment, a number of random image data are collected in advance to form an original data set. For example, the original data set is used to train the image enhancement model;

[0183] Then an image generator G(r) is established, as shown in the following formula. G(r) consists of an encoder G dec (r) and a decoder G enc (r),

[0184]

[0185] where is the pseudo-enhanced image generated by the original image data f under the generator G(f). The generator is used to calculate the input image data to obtain a calculation result, train and update the generator in combination with the calculation result, and provide the calculation result to the discriminator for the discriminator to discriminate the calculation result;

[0186] Input a set of random vectors (θ1, θ2, θ3, …, θ k ) to initialize the generator in G(f), and a set of pseudo-enhanced datasets are generated from the original dataset under G(f);

[0187] Then, a discriminator is established. The discriminator is used to discriminate the input image data. When the input data is the enhanced image data, the discriminator outputs a high score (close to 1), and when the input data is the pseudo-enhanced image data, the discriminator outputs a low score (close to 0) to obtain a discrimination result, and the discriminator is trained and updated in combination with the discrimination result;

[0188] Based on the generator and the discriminator, a generative adversarial network is constructed;

[0189] Finally, the original dataset is enhanced based on the multi-scale homomorphic filtering label to obtain an enhanced dataset. The enhanced dataset and the pseudo-enhanced dataset are input into the generative adversarial network, and the generative adversarial network is trained in combination with the calculation result to obtain the image enhancement model.

[0190] Further, the step of training the generative adversarial network based on the multi-scale homomorphic filtering label, the original dataset, and the pseudo-enhanced dataset to obtain the image enhancement model includes:

[0191] First, combine the data in the original dataset and the corresponding data in the enhanced dataset to obtain an image pair dataset;

[0192] Then, input the enhanced dataset and the pseudo-enhanced dataset into the discriminator for the discriminator to discriminate the enhanced dataset and the pseudo-enhanced dataset to obtain a discrimination result;

[0193] Then, train and update the discriminator in combination with the discrimination result;

[0194] Then, input the image pair dataset into the generator for the generator to calculate the image pair dataset to obtain a first calculation result;

[0195] Then, train and update the generator in combination with the first calculation result;

[0196] Then, input the original dataset into the generator for the generator to calculate the original dataset to obtain a second calculation result;

[0197] Then, use the second result as the pseudo-augmented dataset; and return to execute the steps of inputting the augmented dataset and the pseudo-augmented dataset into the discriminator for the discriminator to discriminate between the augmented dataset and the pseudo-augmented dataset to obtain a discrimination result;

[0198] Finally, loop according to a preset number of loops until the loop is completed, terminate the training, and obtain the trained image enhancement model.

[0199] This embodiment creates an image enhancement model through the above scheme, specifically including:

[0200] Obtain the original dataset; establish a generator; input the first parameter into the generator to initialize the generator; input the original dataset into the generator to obtain the corresponding pseudo-augmented dataset; establish a discriminator; based on the generator and the discriminator, construct a generative adversarial network; based on the multi-scale homomorphic filtering label, the original dataset, and the pseudo-augmented dataset, train the generative adversarial network to obtain the image enhancement model. By using the generator in the generative adversarial network to enhance low-light images, the dependence on prior knowledge of images in traditional methods can be eliminated; through the trained image enhancement model, the real-time processing of underwater images and the preprocessing ability of large data streams can be improved, and the image enhancement efficiency can be enhanced.

[0201] Refer to Figure 6 , Figure 6 For the detailed flowchart of the steps of step S51 in the above Figure 5 shown embodiment, in step S51, the step of training the generative adversarial network based on the multi-scale homomorphic filtering label, the original dataset, and the pseudo-augmented dataset to obtain the image enhancement model includes:

[0202] Step S52: Based on the multi-scale homomorphic filtering label, enhance the data in the original dataset to obtain the corresponding augmented dataset;

[0203] Step S53: Based on the original dataset, the augmented dataset, and the pseudo-augmented dataset, train the generative adversarial network to obtain the trained generative adversarial network.

[0204] Further, in step S53, the step of training the generative adversarial network based on the original dataset, the augmented dataset, and the pseudo-augmented dataset to obtain the trained generative adversarial network includes:

[0205] Step S600: Combine the data in the original dataset and the corresponding data in the augmented dataset to obtain an image pair dataset;

[0206] Step S601: Input the augmented dataset and the pseudo-augmented dataset into the discriminator for the discriminator to discriminate the augmented dataset and the pseudo-augmented dataset to obtain a discrimination result;

[0207] Step S602: Train and update the discriminator in combination with the discrimination result;

[0208] Step S603: Input the image pair dataset into the generator for the generator to calculate the image pair dataset to obtain a first calculation result;

[0209] Step S604: Train and update the generator in combination with the first calculation result;

[0210] Step S605: Input the original dataset into the generator for the generator to calculate the original dataset to obtain a second calculation result;

[0211] Step S606: Use the second result as the pseudo-augmented dataset; and return to execute Step S601: Input the augmented dataset and the pseudo-augmented dataset into the discriminator for the discriminator to discriminate the augmented dataset and the pseudo-augmented dataset to obtain a discrimination result;

[0212] Step S607: Loop according to a preset number of loops until the loop is completed and terminate the training to obtain a trained image enhancement model.

[0213] Specifically, first use the original image dataset F and its corresponding homomorphic filtering enhanced image r(x, y) dataset R = {r1, r2, r3,... r n} in the multi-scale space as the image pair dataset.

[0214] M = {(f1, r1), (f2, r2), (f3, r3),...,(f n , r n )}

[0215] Then, input the augmented dataset and the pseudo-augmented dataset into the discriminator for the discriminator to discriminate the augmented dataset and the pseudo-augmented dataset. When the input data is the augmented dataset, the discriminator labels the augmented dataset as 1 and discriminates it as a high score (close to 1), and then trains and updates the discriminator in combination with the high score result; when the input data is the pseudo-augmented dataset when, the pseudo-augmented dataset Label it as 0, and determine it as a low score (close to 0), and then train and update the discriminator in combination with the high score result;

[0216] Then, the optimization problem has always been a very important field in machine learning and even deep learning. Especially for deep learning, therefore, in this embodiment, the Adam gradient descent algorithm is adopted. When calculating, it is based on the total adversarial loss function of the image pair dataset, which ensures a relatively low amount of calculation. The size of the parameter update in the generator does not change with the scaling of the gradient size; the boundary of the step size when updating the parameter is limited by the setting of the step size of the hyperparameter; there is no need for a fixed objective function. Therefore, input the image pair dataset into the generator for the generator to calculate the total loss function of the image pair dataset, and use the Adam gradient descent algorithm to obtain the updated parameters as the first calculation result;

[0217] Then, train and update the original parameters in the generator in combination with the first calculation result;

[0218] Then, input the original dataset into the generator for the generator to calculate the original dataset. The decoder in the generator encodes the input original image data into a low-dimensional vector. Among them, the low-dimensional vector contains the main information of the original image data. For example, the elements of the low-dimensional vector can represent the color, shape, size, etc. of any underwater creature. The encoder in the generator decodes the structural information of the low-dimensional vector about the image to generate a new pseudo-enhanced dataset as the second calculation result;

[0219] Then, use the second result as the pseudo-enhanced dataset; and return to execute the step: input the enhanced dataset and the pseudo-enhanced dataset into the discriminator for the discriminator to discriminate the enhanced dataset and the pseudo-enhanced dataset to obtain a discrimination result;

[0220] Finally, according to actual needs, loop in this way according to the preset number of loops until the loop is completed and the training is terminated to obtain the trained image enhancement model.

[0221] In this embodiment, through the above solution, specifically, by combining the data in the original dataset and the corresponding data in the enhanced dataset, an image pair dataset is obtained; the enhanced dataset and the pseudo-enhanced dataset are input into the discriminator for the discriminator to discriminate between the enhanced dataset and the pseudo-enhanced dataset to obtain a discrimination result; the discriminator is trained and updated in combination with the discrimination result; the image pair dataset is input into the generator for the generator to calculate the image pair dataset to obtain a first calculation result; the generator is trained and updated in combination with the first calculation result; the original dataset is input into the generator for the generator to calculate the original dataset to obtain a second calculation result; the second result is used as the pseudo-enhanced dataset; and the execution steps are returned: the enhanced dataset and the pseudo-enhanced dataset are input into the discriminator for the discriminator to discriminate between the enhanced dataset and the pseudo-enhanced dataset to obtain a discrimination result; and this loop is repeated according to a preset number of loop times until the loop is completed and the training is terminated to obtain a trained image enhancement model. By calculating the total loss function of the image pair dataset, using the total loss function and the Adam gradient descent algorithm, the difference between the predicted value and the true value of the image enhancement model is obtained, and the difference between the predicted value and the true value is used to measure the quality of the prediction of the image enhancement model, and the image enhancement model is trained and updated to obtain a trained image enhancement model. The real-time processing of underwater images and the preprocessing ability of large data streams are improved.

[0222] Refer to Figure 7 , Figure 7 is Figure 6 a detailed flowchart of the steps of step S52 in Figure 6 According to the embodiment shown above, step S52: The step of enhancing the data in the original dataset based on the multi-scale homomorphic filtering label to obtain a corresponding enhanced dataset includes:

[0223] Step S800: Calculate the noise reduction term corresponding to the data in the original dataset for denoising the data in the original dataset;

[0224] Specifically, during the process of collecting, transmitting, and receiving the original dataset, it is in a complex external environment and there are various interferences. Generally, it will be affected by noise, which will reduce the resolution of the image, and at the same time, the original fine structure of the image will also be damaged. And for processing digital images, denoising is the premise for various feature recognition and extraction.

[0225] Process the original image data f(x, y) in the original dataset, and realize denoising of the original image in the sliding window through a convolutional network. Calculate the image convolution noise reduction term I q (x, y).

[0226] I q (x,y) = med{f(x - k,y - l),(k,l ∈ CW)}

[0227] Where CW is the convolution kernel size.

[0228] Step S801: Calculate the spatial distance weight value of the data in the original dataset relative to the corresponding pixel point to the center point and the adjacent pixel value weight;

[0229] Specifically, the spatial distance weight value of pixel point I to the center point is calculated by the following formula.

[0230]

[0231] Then, the adjacent pixel value weight s(ξ,x) of the image is calculated by the following formula:

[0232]

[0233] Where ξ represents the spatial distance from pixel point I to the center point.

[0234] Step S802: Calculate the pixel weight sum based on the adjacent pixel value weight and the spatial distance weight value;

[0235] Specifically, the image pixel weight sum w(ξ,x,k,l) is calculated by the following formula:

[0236] w(ξ,x,k,l) = c(ξ,x)s(ξ,x);

[0237] Step S803: Calculate the reflected image based on the pixel weight sum and the denoising term;

[0238] Specifically, the reflected image L(x,y) is calculated by the following formula.

[0239]

[0240] Step S804: Calculate the reflected response image based on the reflected image and the preset Gaussian function for multi-scale spatial construction of the data in the original dataset;

[0241] Specifically, from near to far, it will cause the original image data to become more and more blurred, that is, in the process of the scale of the original image data becoming larger and larger. Therefore, it is necessary to perform multi-scale spatial construction on the original image data to obtain the best scale of the object of interest; and there are the same key points at different scales, so the key points can be detected and matched under the input image data of different scales.

[0242] The reflected image L(x,y) is calculated by the following formula.

[0243]

[0244] Then, calculate the reflection response image R(x, y, σ) by the following formula to construct a multi-scale space.

[0245] R(x, y, σ) = G(x, y, σ) * L(x, y);

[0246] In the formula, G(x, y, σ) is a variable Gaussian function, which is defined as follows:

[0247]

[0248] In the formula, σ is a scale space factor, which determines the degree of image blurring and smoothing processing.

[0249] Step S805: Based on the reflection response image and the noise reduction term, calculate the data in the original data set relative to the corresponding enhanced data, and so on, to obtain the enhanced data set.

[0250] Specifically, calculate the homomorphic filtering enhanced image r(x, y) in the multi-scale space

[0251]

[0252] In the formula, N is the number of scales of the constructed multi-scale space.

[0253] Finally, based on the multi-scale homomorphic filtering label, obtain the enhanced image r(x, y) corresponding to the original image data, and form an enhanced data set.

[0254] In this embodiment, through the above solution, specifically by calculating the data in the original data set relative to the corresponding noise reduction term; calculating the spatial distance weight value from the pixel point to the center point and the adjacent pixel value weight of the data in the original data set; calculating the pixel weight sum based on the adjacent pixel value weight and the spatial distance weight value; calculating the reflection image based on the pixel weight sum and the noise reduction term; calculating the reflection response image based on the reflection image and a preset Gaussian function; calculating the data in the original data set relative to the corresponding enhanced data based on the reflection response image and the noise reduction term, and so on, to obtain the enhanced data set. Based on the multi-scale homomorphic filtering label, denoising, multi-scale space construction, and image enhancement are performed on the original image data in the original data set, which can greatly reduce the difficulty of collecting the original data set, and there is no need to require that the underwater image data set has both underwater distorted images and clear images, nor is it required to collect images with and without water at the same position and the same parameters, thereby improving the real-time processing of underwater images and the preprocessing ability of large data streams, and enhancing the image enhancement efficiency.

[0255] Reference Figure 8 , Figure 8 is Figure 6 a detailed process schematic diagram of the steps in step S603 in Figure 6 Based on the embodiment shown above, step S603: inputting the image pair data set into the generator for the generator to calculate the image pair data set to obtain the first calculation result includes:

[0256] Step S700: Input the image pair data set into the generator, and calculate the adversarial loss function based on the image pair data set and the expected value of the distribution function;

[0257] Specifically, the following formula is used to calculate the adversarial loss function

[0258]

[0259] In the formula, represents the expected value of the distribution function, D represents the discriminator, G represents the generator, f represents the data in the original data set, and r represents the homomorphic filtering enhanced image in the multi-scale space.

[0260] By calculating the input image pair data set through the generator, the loss function of each image pair data is obtained, and the gap between the forward calculation result of each iteration and the true image data value is obtained, thereby guiding the next step of training in the correct direction.

[0261] Step S701: Extract the output features of the corresponding levels from the feature maps of several levels through the multi-level image block network, and calculate the features of the corresponding levels;

[0262] Specifically, features are the raw materials of the machine learning system, and their influence on the final image enhancement model is beyond doubt. When the data is well expressed as features, the image enhancement model can achieve satisfactory accuracy.

[0263] Select the feature maps of a total of L layers of interest, and pass them through the two-layer MLP network H l The generated features are:

[0264]

[0265] Among them, represents the output feature of the l-th layer, z l represents the feature of the l-th layer, l ∈ {1, 2, 3,..., L}, f represents the original data in the original data set, and H l represents a two-layer MLP network.

[0266] Step S702: Calculate the maximum mutual information and the features of the corresponding layer of the feature map based on the noise contrast estimation framework to obtain a noise contrast estimation repair function;

[0267] Specifically, the noise contrast estimation algorithm is a statistical model estimation method that can be used to solve complex calculation problems in generative adversarial networks.

[0268] Calculate the maximum mutual information through the noise contrast estimation (NCE) framework to generate a noise contrast estimation repair function.

[0269]

[0270] Among them, s represents the number of patches in each layer (s ∈ {1, 2, 3, …, S l}), where S l represents that there are S l spatial positions in the l-th layer, represents the probability that the positive sample is selected in NCE, represents that the dimension of the feature vector corresponding to the s-th patch in the l-th layer is C l , G represents the generator, F represents the original dataset, and H represents the two-layer multi-level image patch network.

[0271] At this time, the enhanced image can be expressed as:

[0272]

[0273] represents the output feature of the l-th layer, and f represents the original data in the original dataset.

[0274] By calculating for each image pair, each enhanced image forms an enhanced dataset R.

[0275] Step S703: Calculate the total loss function based on the adversarial loss function and the noise contrast estimation repair function;

[0276] Specifically, if the parameters in the generator are adjusted to completely satisfy that the output error of any image pair data is zero, usually the error of any other image pair data except the current image pair data will become larger. In this way, as the sum of errors, the value of the loss function will become larger. Therefore, after adjusting the weights according to the error of any image pair data, calculate the total loss function value of the image pair dataset to determine whether the image enhancement data has been trained to an acceptable state.

[0277] Then, calculate the total loss function through the above loss function

[0278]

[0279] In the formula, R represents the enhanced dataset, and D represents the discriminator.

[0280] Among them, according to the computing rate requirement, it can be set as: λ F = 1, λ R = 1 or λ F = 10, λ R = 0.

[0281] Step S704: Based on the gradient descent algorithm and the total loss function, calculate and obtain the second parameter;

[0282] Specifically, in this embodiment, according to the requirement, by setting the number of iterations, using the total loss function and the Adam gradient descent algorithm to obtain the second parameter (θ1, θ2, θ3, …, θ k ).

[0283]

[0284] By calculating the Adam gradient descent algorithm and the total loss function, it is possible to reduce oscillations, keep the general direction unchanged, thereby ensuring the efficiency and correct convergence of the calculation, and using relatively less memory.

[0285] Step S705: Use the second parameter as the first parameter; and return to execute the step: Step S704, based on the gradient descent algorithm and the total loss function, calculate and obtain the second parameter;

[0286] Specifically, use the above-mentioned second parameter as the first parameter, and return to execute Step S704 to continue the next iteration until the first parameter in the generator converges to obtain the first parameter after the iteration is completed.

[0287] Step S706: In this way, loop, perform parameter iteration according to the preset number of iterations until the iteration is completed, and use the first parameter as the first calculation result.

[0288] Loop in sequence, perform parameter iteration according to the preset number of iterations until the iteration is completed, use the first parameter after the iteration is completed as the first calculation result, and train and update the image enhancement model.

[0289] In this embodiment, through the above solution, specifically, by inputting the image pair dataset into the generator, an adversarial loss function is calculated based on the image pair dataset and the expected value of the distribution function; the output features of the corresponding levels are extracted from the feature maps of several levels through a multi-level image block network, and the features of the corresponding levels are calculated; the maximum mutual information and the features of the corresponding levels of the feature maps are calculated based on the noise contrast estimation framework to obtain a noise contrast estimation repair function; based on the adversarial loss function and the noise contrast estimation repair function, a total loss function is calculated; based on the gradient descent algorithm and the total loss function, a second parameter is calculated; the second parameter is used as the first parameter; and the execution steps are returned: based on the gradient descent algorithm and the total loss function, a second parameter is calculated; and so on in a loop, and parameter iteration is performed according to a preset number of iterations until the iteration is completed, and the first parameter is used as the first calculation result. By calculating the total loss function through the noise contrast estimation algorithm, the complex calculation problem in the generative adversarial network can be solved; through the gradient descent algorithm and the total loss function, the efficiency of the calculation and the correct convergence are ensured, and the memory used is relatively small. The real-time processing of underwater images and the preprocessing ability of large data streams are improved, and the image enhancement efficiency is enhanced.

[0290] It should be noted that in this article, the terms "including", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or system. Without further limitation, an element defined by the phrase "including a..." does not exclude the existence of additional identical elements in the process, method, article or system including that element.

[0291] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

[0292] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

[0293] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. An image enhancement method, characterized in that, The image enhancement method includes the following steps: Obtain the image data to be enhanced; Input the image data to be enhanced into a pre-created image enhancement model for enhancement to obtain enhanced image data, where the image enhancement model is trained based on a generative adversarial network and combined with a preset enhanced image algorithm; Before the step of inputting the data to be enhanced into a pre-created image enhancement model for enhancement to obtain enhanced data, the following steps are also included: Obtain the original data set; Establish a generator; Input a first parameter into the generator to initialize the generator; Input the original data set into the generator to obtain a corresponding pseudo-enhanced data set; Establish a discriminator; Based on the generator and the discriminator, construct a generative adversarial network; Based on the multi-scale homomorphic filtering label, the original data set, and the pseudo-enhanced data set, train the generative adversarial network to obtain an image enhancement model; The step of training the generative adversarial network based on the multi-scale homomorphic filtering label, the original data set, and the pseudo-enhanced data set to obtain an image enhancement model includes: Enhance the data in the original data set based on the multi-scale homomorphic filtering label to obtain a corresponding enhanced data set; Based on the original data set, the enhanced data set, and the pseudo-enhanced data set, train the generative adversarial network to obtain a trained generative adversarial network; The step of enhancing the data in the original data set based on the multi-scale homomorphic filtering label to obtain a corresponding enhanced data set includes: Calculate the noise reduction term corresponding to the data in the original data set for denoising the data in the original data set; Calculate the spatial distance weight value from the pixel point to the center point and the adjacent pixel value weight corresponding to the data in the original data set; Based on the adjacent pixel value weight and the spatial distance weight value, calculate the pixel weight sum; Based on the pixel weight sum and the noise reduction term, calculate a reflected image; Based on the reflected image and a preset Gaussian function, calculate a reflected response image for multi-scale spatial construction of the data in the original data set; Based on the reflected response image and the noise reduction term, calculate the enhanced data corresponding to the data in the original data set, and so on, to obtain the enhanced data set.

2. The image enhancement method according to claim 1, wherein The step of training the generative adversarial network based on the original data set, the enhanced data set, and the pseudo-enhanced data set to obtain a trained generative adversarial network includes: Combine the data in the original data set and the corresponding data in the enhanced data set to obtain an image pair data set; Input the enhanced data set and the pseudo-enhanced data set into the discriminator for the discriminator to discriminate the enhanced data set and the pseudo-enhanced data set to obtain a discrimination result; Train and update the discriminator in combination with the discrimination result; Input the image pair data set into the generator for the generator to calculate the image pair data set to obtain a first calculation result; Train and update the generator in combination with the first calculation result; Input the original data set into the generator for the generator to calculate the original data set to obtain a second calculation result; Use the second result as the pseudo-augmented data set; and return to execute the step: input the augmented data set and the pseudo-augmented data set into the discriminator for the discriminator to discriminate between the augmented data set and the pseudo-augmented data set to obtain a discrimination result; Loop in this way according to a preset number of cycles until the loop is completed and terminate the training to obtain a trained image enhancement model.

3. The image enhancement method according to claim 2, wherein The step of inputting the image pair data set into the generator for the generator to calculate the image pair data set to obtain a first calculation result includes: Input the image pair data set into the generator and calculate the adversarial loss function based on the image pair data set and the expected value of the distribution function; Extract the output features of the corresponding levels from the feature maps of several levels through a multi-level image block network and calculate the features of the corresponding levels; Calculate the maximum mutual information and the features of the corresponding levels of the feature maps based on the noise contrast estimation framework to obtain a noise contrast estimation repair function; Calculate the total loss function based on the adversarial loss function and the noise contrast estimation repair function; Calculate the second parameter based on the gradient descent algorithm and the total loss function; Use the second parameter as the first parameter; and return to execute the step: calculate the second parameter based on the gradient descent algorithm and the total loss function; Loop in this way and perform parameter iteration according to a preset number of iterations until the iteration is completed, and use the first parameter as the first calculation result.

4. An image enhancement device, characterized in that, The image data enhancement device includes: An acquisition module for acquiring image data to be enhanced; An enhancement module that inputs the image data to be enhanced into a pre-created image enhancement model for enhancement to obtain enhanced image data, where the image enhancement model is trained based on a generative adversarial network and in combination with a preset enhanced image algorithm; A creation module for acquiring an original data set, establishing a generator, inputting a first parameter into the generator, initializing the generator, inputting the original data set into the generator to obtain a corresponding pseudo-augmented data set; establishing a discriminator; constructing a generative adversarial network based on the generator and the discriminator; training the generative adversarial network based on multi-scale homomorphic filtering labels, the original data set, and the pseudo-augmented data set to obtain an image enhancement model; The creation module is further used for: Enhance the data in the original data set based on the multi-scale homomorphic filtering label to obtain a corresponding enhanced data set; Train the generative adversarial network based on the original data set, the enhanced data set, and the pseudo-augmented data set to obtain a trained generative adversarial network; The enhancing the data in the original data set based on the multi-scale homomorphic filtering label to obtain a corresponding enhanced data set includes: Calculate the data in the original dataset relative to the corresponding noise reduction term for denoising the data in the original dataset; Calculate the spatial distance weight value from the data in the original dataset to the corresponding pixel point to the center point and the adjacent pixel value weight; Based on the adjacent pixel value weight and the spatial distance weight value, calculate the pixel weight sum; Based on the pixel weight sum and the noise reduction term, calculate the reflected image; Based on the reflected image and a preset Gaussian function, calculate the reflected response image for multi-scale spatial construction of the data in the original dataset; Based on the reflected response image and the noise reduction term, calculate the enhanced data corresponding to the data in the original dataset, and so on, to obtain the enhanced dataset.

5. A terminal device, characterized in that, The terminal device includes: a memory, a processor, and an image enhancement program stored on the memory and executable on the processor, the image enhancement program being configured to implement the steps of the image enhancement method according to any one of claims 1 to 3.

6. A storage medium, characterized in that, An image enhancement program is stored on the storage medium, When the image enhancement program is executed by the processor, it implements the steps of the image enhancement method according to any one of claims 1 to 3.

Images