An image enhancement method, device and electronic equipment

Abstract

This application provides an image enhancement method, apparatus, and electronic device, comprising: acquiring an underwater image to be processed; performing image enhancement processing on the underwater image to be processed based on a target enhancement model to acquire an enhanced target underwater image; the target enhancement model is obtained by adjusting the parameters of a training enhancement model at least once based on a loss function, the loss function being constructed based on the adversarial loss corresponding to the output image with the same scale as the input image in at least two output images of the training enhancement model, and the pixel and loss corresponding to each of the at least two output images, wherein the at least two output images have different scales. The training enhancement model is optimized by jointly using the adversarial loss and the loss function constructed from at least two pixel-level losses to obtain the target enhancement model, thereby obtaining a clear and accurate target underwater image with the same content structure as the underwater image to be processed, thus improving the image enhancement effect of the underwater image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image enhancement method, apparatus, and electronic device. Background Technology

[0002] Underwater imagery serves as a primary medium for understanding, exploring, and developing the ocean, playing an irreplaceable role in marine engineering, resource development, and other marine fields. However, the unique underwater imaging environment often results in complex degradation phenomena in directly acquired underwater images, such as color distortion and blurred details, significantly reducing their usability in visual tasks like target detection and semantic segmentation.

[0003] Therefore, in order to improve the usability of underwater images in visual tasks such as target detection and semantic segmentation, image enhancement processing is usually performed on underwater images to obtain clear underwater images; thus, image enhancement processing of underwater images has become a hot topic in computer vision.

[0004] Currently, image enhancement methods for underwater images can be categorized into three types: restoration methods based on traditional imaging models, enhancement methods based on non-physical models, and methods based on deep learning. However, restoration methods based on traditional imaging models and enhancement methods based on non-physical models often suffer from high computational complexity and poor real-time performance when enhancing underwater images, exhibiting poor algorithm generalization ability and mediocre image enhancement results. While deep learning-based methods offer relatively lower computational complexity and better image enhancement results, they still suffer from issues such as incomplete color cast recovery and insufficient detail enhancement.

[0005] Therefore, improving the image enhancement effect of underwater images is a problem that needs to be solved when performing image enhancement processing on underwater images. Summary of the Invention

[0006] This application provides an image enhancement method, apparatus, and electronic device for improving the image enhancement effect of underwater images.

[0007] In a first aspect, embodiments of this application provide an image enhancement method, the method comprising:

[0008] Acquire underwater images to be processed;

[0009] Based on the target enhancement model, image enhancement processing is performed on the underwater image to be processed to obtain the enhanced underwater image of the target.

[0010] The target augmentation model is obtained by adjusting the parameters of the model to be trained at least once based on the loss function. The loss function is constructed based on the adversarial loss corresponding to the first type of output image with the same scale as the input image in at least two output images of the model to be trained, and the pixel-level loss corresponding to each of the at least two output images. The at least two output images have different scales.

[0011] In this embodiment, a loss function is first constructed based on the adversarial loss corresponding to the first type of output image with the same scale as the input image among at least two output images of the enhancement model to be trained, and the pixel-level loss corresponding to each of the at least two output images. At least one parameter adjustment is then performed on the enhancement model to be trained to jointly optimize it and obtain the target enhancement model. This ensures that the optimized target enhancement model can produce a clear, accurate, and consistent enhanced image with the content structure of the input image. Clearness and accuracy include the characteristics of complete color cast recovery and sufficient detail enhancement in the enhanced image. Then, based on the target enhancement model, image enhancement processing is performed on the acquired underwater image to be processed to obtain the enhanced target underwater image. At this point, the target underwater image is a clear, accurate, and consistent enhanced image with the content structure of the underwater image to be processed. Therefore, the image enhancement method provided in this application improves the image enhancement effect of underwater images.

[0012] In one possible implementation, the target augmentation model includes: a first encoder, a first attention network, and a first decoder;

[0013] Based on the target enhancement model, image enhancement processing is performed on the underwater image to be processed, including:

[0014] The first encoder extracts features from the underwater image to be processed, obtaining at least two initial feature maps.

[0015] Through the first attention network, attention processing is performed on at least two initial feature maps to obtain the corresponding target attention map;

[0016] Using the first decoder, based on the scale of at least two acquired target attention maps, the target attention maps are sequentially stitched with upsampled feature maps of the same scale in ascending order of scale until the stitched image is the same scale as the underwater image to be processed. Then, the target underwater image is determined and output based on the stitched image.

[0017] In this application embodiment, a specific implementation method for enhancing underwater images to be processed based on a target enhancement model is provided to ensure that a clear, accurate target underwater image that is consistent with the content structure of the underwater image to be processed is obtained based on the target enhancement model.

[0018] In one possible implementation, a first encoder extracts features from the underwater image to be processed, obtaining at least two initial feature maps, including:

[0019] The first encoder extracts features from the underwater image to be processed, obtaining a first type of initial feature map with the same scale as the underwater image to be processed, and a second type of initial feature map obtained by downsampling the first type of initial feature map at least once.

[0020] In this embodiment, feature extraction is performed on the underwater image to be processed to obtain a first type of initial feature map with the same scale as the image under the tree to be processed, and a second type of initial feature map is obtained by downsampling the first type of initial feature map at least once. Initial feature maps of different scales are extracted to extract more comprehensive feature information and ensure the accuracy of subsequent image enhancement processing.

[0021] In one possible implementation, attention processing is performed on at least two initial feature maps, including:

[0022] For any initial feature map, channel attention processing is performed on the initial feature map to obtain a channel attention map. The channel attention map is then multiplied with the initial feature map to obtain an intermediate feature map.

[0023] Spatial attention processing is applied to the intermediate feature map to obtain a spatial attention map. The spatial attention map is then multiplied with the intermediate feature map to obtain the corresponding target attention map.

[0024] In this embodiment, the initial feature map is sequentially processed by channel attention and spatial attention to obtain the corresponding target attention map, which enriches the detailed information of the underwater image to be processed, so as to ensure the accuracy of subsequent image enhancement processing.

[0025] In one possible implementation, the first decoder includes an error feedback network, and uses the input upsampled feature map corresponding to the stitched image as a high-scale feature map, and uses the feature map of the input upsampled feature map before upsampling as a low-scale feature map.

[0026] After concatenating the target attention map with an upsampled feature map of the same scale, the following operation is performed through the error feedback network:

[0027] The high-scale feature map is downsampled to obtain a downsampled feature map, and the feature error between the downsampled feature map and the low-scale feature map is determined. The downsampled feature map and the low-scale feature map have the same scale.

[0028] The feature error is deconvolved to obtain the high-scale error corresponding to the high-scale feature map, and the high-scale error is summed with the stitched image after the convolution operation.

[0029] In this embodiment, an error feedback network is introduced into the first decoder, and error feedback correction is achieved in the feature reconstruction process through the error feedback network. This maintains the inherent connection between feature maps of different scales, such as semantic consistency, and further ensures the clarity and accuracy of the output target underwater image, as well as the consistency with the content structure of the underwater image to be processed.

[0030] In one possible implementation, the augmentation model to be trained includes: a second encoder, a second attention network, a second decoder, a discriminant network, and a supervised optimization network; the target augmentation model is obtained as follows:

[0031] Training sample pairs are selected from the training dataset. The training sample pairs include: the original image and the corresponding synthetic underwater image.

[0032] The synthetic underwater image is used to extract features through a second encoder to obtain at least two training feature maps.

[0033] Through the second attention network, attention processing is performed on at least two training feature maps respectively to obtain the corresponding training attention maps;

[0034] The second decoder concatenates at least two training attention maps with corresponding training upsampled feature maps of the same scale to obtain at least two output images. The at least two output images include a first type of output image with the same scale as the synthetic underwater image and a second type of output image with a smaller scale than the synthetic underwater image.

[0035] The adversarial loss is determined by the discriminative network based on the first comparison result between the first type of output image and the original image;

[0036] By using a supervised optimization network, the corresponding pixel-level loss is determined based on the second comparison result between the first type of output image and the original image, and the third comparison result between the second type of output image and the reference image of the same scale. The reference image is obtained by downsampling the original image.

[0037] A loss function is constructed based on adversarial loss and at least two determined pixel-level losses. The parameters of the model to be trained are adjusted at least once using the loss function until the conditions are met, thus obtaining the target augmentation model.

[0038] In this embodiment, a specific method for training an augmentation network to obtain a target augmentation model for image augmentation processing is provided. When training the augmentation network, the method mainly involves constructing a loss function based on the adversarial loss corresponding to the output image with the same scale as the input image in at least two output images of the augmentation model to be trained, and the pixel-level loss corresponding to each of the at least two output images. The method adjusts the parameters of the augmentation model to be trained at least once to jointly optimize the augmentation model to be trained, so as to ensure that the target augmentation model obtained after optimization can obtain a clear and accurate augmented image that is consistent with the content structure of the input image. The clear and accurate features include the characteristics of complete color shift recovery and sufficient detail enhancement in the augmented image obtained after image augmentation.

[0039] In one possible implementation, the synthetic underwater image is determined from the original image based on a successfully trained underwater image physical imaging model.

[0040] In this embodiment of the application, the synthesized underwater image is determined based on the original image using a successfully trained underwater image physical imaging model. This accurately determines the training sample pairs in the training dataset. While ensuring the accuracy of the training sample pairs, the accuracy of the successfully trained target enhancement model is further guaranteed. That is, when the underwater image is enhanced by the target enhancement model, the accuracy of the enhanced underwater image is guaranteed, thereby improving the image enhancement effect of the underwater image.

[0041] In one possible implementation, the model to be trained and augmented is adjusted at least once based on the loss function, including:

[0042] Based on the loss function, the optimizer performs at least one parameter adjustment on the training enhancement model.

[0043] In the embodiments of this application, based on the loss function, the optimizer performs at least one parameter adjustment on the training augmentation model and optimizes the training augmentation model to ensure the accuracy of the optimization of the training augmentation model.

[0044] Secondly, embodiments of this application provide an image enhancement apparatus, the apparatus comprising:

[0045] The acquisition unit is used to acquire underwater images to be processed.

[0046] The processing unit is used to perform image enhancement processing on the underwater image to be processed based on the target enhancement model, and to obtain the enhanced underwater image of the target;

[0047] The target augmentation model is obtained by adjusting the parameters of the augmentation model to be trained at least once based on the loss function. The loss function is constructed based on the adversarial loss determined by the first type of output image with the same scale as the input image in at least two output images of the augmentation model to be trained, and the pixel-level loss corresponding to each of the at least two output images. The at least two output images have different scales.

[0048] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program to implement the steps of the image enhancement method provided in embodiments of this application.

[0049] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the image enhancement method provided in embodiments of this application.

[0050] Fifthly, embodiments of this application provide a computer program product, which includes a computer program stored in a computer-readable storage medium; when the processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of the image enhancement method provided in embodiments of this application.

[0051] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0054] Figure 2 A schematic diagram of a first type of training enhancement model provided in an embodiment of this application;

[0055] Figure 3 A schematic diagram of a second type of training enhancement model provided in an embodiment of this application;

[0056] Figure 4A schematic diagram of a second encoder provided for an embodiment of this application;

[0057] Figure 5 A schematic diagram of a second attention network provided in an embodiment of this application;

[0058] Figure 6 A schematic diagram of a first type of second decoder provided in an embodiment of this application;

[0059] Figure 7 A schematic diagram of a discrimination network provided for an embodiment of this application;

[0060] Figure 8 This is a schematic diagram of the specific structure of an enhancement model to be trained, provided in an embodiment of this application.

[0061] Figure 9 This application provides a flowchart of a method for training an augmentation model to be trained.

[0062] Figure 10 A schematic diagram of a second type of second decoder provided in an embodiment of this application;

[0063] Figure 11 A schematic diagram illustrating the specific structure of another training enhancement model provided in this application embodiment;

[0064] Figure 12 A flowchart illustrating another method for training a model to be enhanced, as provided in this application embodiment;

[0065] Figure 13 A schematic diagram of a target enhancement model provided in an embodiment of this application;

[0066] Figure 14 A flowchart of an image enhancement method provided in this application embodiment;

[0067] Figure 15 This is a schematic diagram of the enhancement results of various algorithms on a synthetic underwater image provided in an embodiment of this application;

[0068] Figure 16 This application provides a schematic diagram of the enhancement results of various algorithms on a real underwater image, as shown in the embodiments of this application.

[0069] Figure 17 A flowchart illustrating a specific implementation method for image enhancement provided in this application embodiment;

[0070] Figure 18 A structural diagram of an image enhancement device provided in an embodiment of this application;

[0071] Figure 19 This is a structural diagram of another image enhancement device provided in an embodiment of this application;

[0072] Figure 20 A structural diagram of an electronic device provided in an embodiment of this application;

[0073] Figure 21 This is another electronic device structure diagram provided for an embodiment of this application. Detailed Implementation

[0074] To make the objectives, technical solutions, and beneficial effects of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0075] The word “exemplary” as used below means “serving as an example, embodiment, or illustration.” Any embodiment illustrated as an “exemplary” need not be construed as superior to or better than other embodiments.

[0076] The terms "first" and "second" used below are for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0077] The design concept of the embodiments of this application will be briefly introduced below.

[0078] This application relates to the field of image processing technology, and mainly to the field of image enhancement processing of underwater images.

[0079] Underwater images acquired directly in underwater imaging environments suffer from complex degradation, reducing their usability in visual tasks such as target detection and semantic segmentation. Therefore, to improve the usability of underwater images in these tasks, image enhancement processing is typically performed to obtain clearer underwater images.

[0080] In related technologies, image enhancement methods for underwater images can be divided into three categories: restoration methods based on traditional imaging models, enhancement methods based on non-physical models, and methods based on deep learning.

[0081] However, restoration methods based on traditional imaging models and enhancement methods based on non-physical models often suffer from high computational complexity and poor real-time performance when enhancing underwater images. These methods also exhibit poor generalization ability and mediocre image enhancement results. Therefore, deep learning-based methods, which offer relatively lower computational complexity and better image enhancement effects, have emerged. While deep learning-based methods offer relatively lower computational complexity and better image enhancement results for underwater image enhancement, they still suffer from problems such as incomplete color cast recovery and insufficient detail enhancement. Color cast refers to a significant difference in hue and saturation between the image and the real image; incomplete color cast recovery refers to the presence of pixels in the enhanced image whose hue and saturation have not been restored. In other words, deep learning-based methods often result in images that are not clear and accurate after underwater image enhancement, leading to poor image enhancement effects.

[0082] Therefore, improving the image enhancement effect of underwater images is a problem that needs to be solved when performing image enhancement processing on underwater images.

[0083] In view of this, embodiments of this application provide an image enhancement method, apparatus, and electronic device, specifically relating to an underwater image multi-scale progressive enhancement algorithm based on encoding / decoding structure and attention mechanism, so as to improve the image enhancement effect of underwater images when performing image enhancement processing.

[0084] In this embodiment of the application, firstly, the parameters of the augmentation model to be trained are adjusted at least once to obtain a successfully trained target augmentation model; then, the underwater image to be processed is augmented using the successfully trained target augmentation model to obtain the augmented target underwater image.

[0085] Specifically, when performing at least one parameter adjustment on the model to be trained to obtain a successfully trained target augmentation model, the model to be trained is constructed using a second encoder, a second attention network, a second decoder, a discriminator network, and a supervised optimization network. The model to be trained is then trained in the following manner to obtain the successfully trained target augmentation model:

[0086] First, training sample pairs are selected from the training dataset. The training sample pairs include: the original image and the corresponding synthetic underwater image.

[0087] Secondly, the synthetic underwater image from the training sample pair is input into the second encoder, and the second encoder extracts features from the synthetic underwater image to obtain at least two training feature maps.

[0088] Then, the acquired at least two training feature maps are input into the second attention network. The second attention network performs attention processing on the at least two training feature maps respectively to obtain the corresponding training attention maps.

[0089] Next, the acquired at least two training attention maps are input into the second decoder. The second decoder concatenates the at least two training attention maps with the corresponding training upsampled feature maps of the same scale to obtain at least two output images. The at least two output images include a first type of output image with the same scale as the synthesized underwater image and a second type of output image with a smaller scale than the synthesized underwater image.

[0090] Finally, the first type of output image is input into the discriminative network. Based on the first comparison result between the first type of output image and the original image, the discriminative network determines the adversarial loss. At least two output images are input into the supervised optimization network. Based on the second comparison result between the first type of output image and the original image, and the third comparison result between the second type of output image and a reference image of the same scale, the supervised optimization network determines the corresponding pixel-level loss. The reference image is obtained by downsampling the original image. A loss function is constructed based on the adversarial loss and the determined at least two pixel-level losses. The parameters of the augmentation model to be trained are adjusted at least once through the loss function until the conditions are met, and the successfully trained target augmentation model is obtained.

[0091] It should be noted that the embodiments of this application involve artificial intelligence (AI) and machine learning (ML) technologies.

[0092] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence.

[0093] Artificial intelligence (AI) studies the design principles and implementation methods of various intelligent machines, enabling them to perceive, reason, and make decisions. AI technology mainly includes computer vision, natural language processing, and machine learning / deep learning. With the research and advancement of AI technology, it is being researched and applied in multiple fields.

[0094] Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Compared to data mining, which focuses on finding patterns in large datasets, machine learning emphasizes algorithm design, enabling computers to automatically "learn" patterns from data and use these patterns to predict unknown data.

[0095] Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning. Reinforcement learning (RL), also known as reward learning, evaluation learning, or enhancement learning, is one of the paradigms and methodologies of machine learning. It is used to describe and solve problems where an agent learns strategies to maximize rewards or achieve specific goals during its interaction with the environment.

[0096] In this embodiment, a loss function is first constructed based on the adversarial loss corresponding to the output image with the same scale as the input image from at least two output images of the enhancement model to be trained, and the pixel-level loss corresponding to each of the at least two output images. This function is used to perform at least one parameter adjustment on the enhancement model to be trained, jointly optimizing the model and obtaining the target enhancement model. This ensures that the optimized target enhancement model yields a clear, accurate, and structurally consistent enhanced image with the input image. Clearness and accuracy include complete color shift recovery and sufficient detail enhancement in the enhanced image. Then, based on the target enhancement model, image enhancement processing is performed on the acquired underwater image to be processed, obtaining the enhanced target underwater image. At this point, the target underwater image is a clear, accurate, and structurally consistent enhanced image with the underwater image to be processed. Therefore, the image enhancement method provided in this application improves the image enhancement effect of underwater images.

[0097] After introducing the design concept of the embodiments of this application, the following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application can be applied. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0098] See Figure 1 , Figure 1This is a schematic diagram of an application scenario according to an embodiment of this application. The application scenario includes a terminal device 110 and a server 120, and the terminal device 110 and the server 120 can communicate with each other through a communication network.

[0099] In one alternative implementation, the communication network can be a wired network or a wireless network. Therefore, the terminal device 110 and the server 120 can be connected directly or indirectly via wired or wireless communication. For example, the terminal device 110 can be indirectly connected to the server 120 via a wireless access point, or the terminal device 110 can be directly connected to the server 120 via the Internet; this application does not impose any limitations on this.

[0100] In this embodiment, the terminal device 110 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances, vehicle terminals, and other devices; the terminal device may be equipped with a client for acquiring and displaying underwater images, which may be a software application (such as a browser, video software, etc.) or a webpage, mini-program, etc.

[0101] Server 120 is a backend server corresponding to software, web pages, mini-programs, etc., or a server specifically used for image enhancement processing; this application does not impose specific limitations. Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0102] It should be noted that the image enhancement method in this application embodiment can be executed by an electronic device, which can be a server 120 or a terminal device 110. That is, the method can be executed by the server 120 or the terminal device 110 alone, or by the server 120 and the terminal device 110 together.

[0103] When the terminal device 110 executes alone, for example, the terminal device 110 can acquire the underwater image to be processed, and then perform image enhancement processing on the underwater image to be processed based on the target enhancement model to obtain the enhanced target underwater image.

[0104] When the server 120 is running alone, for example, the terminal device 110 can acquire the underwater image to be processed and then send the underwater image to be processed to the server 120. The server 120 performs image enhancement processing on the underwater image to be processed based on the target enhancement model to obtain the enhanced target underwater image.

[0105] When the server 120 and the terminal device 110 work together, for example, the terminal device 110 can obtain at least two initial feature maps of the underwater image to be processed and transmit the obtained at least two initial feature maps to the server 120, and the server 120 can obtain the enhanced target underwater image based on the at least two initial feature maps.

[0106] It should be noted that the following description mainly uses the example of a server executing the operation alone, and no specific limitations are made here. That is, in a specific implementation, the terminal device 110 acquires the underwater image to be processed and transmits the underwater image to be processed to the server 120. After acquiring the underwater image to be processed, the server 120 can use the image enhancement method of the present application embodiment to perform image enhancement processing on the underwater image to be processed, and obtain the enhanced target underwater image.

[0107] It should be noted that, Figure 1 The examples shown are merely illustrative; in reality, the number of terminal devices 110 and servers 120 is unlimited and is not specifically limited in this embodiment.

[0108] In this embodiment of the application, when there are multiple servers 120, the multiple servers 120 can form a blockchain, and the server 120 is a node on the blockchain; as disclosed in the image enhancement method of this embodiment, the training set involved can be stored on the blockchain.

[0109] Based on the above application scenarios, the image enhancement method provided by the exemplary embodiments of this application will be described below in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application. The embodiments of this application are not limited in any way in this respect, and the embodiments described herein are only for illustration and explanation of this application and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0110] The image enhancement method provided in this application is implemented through a target enhancement model. Therefore, to implement the image enhancement method provided in this application, the enhancement model to be trained should first be trained to obtain a successfully trained target enhancement model. The training method of the image enhancement model will be described in detail below with reference to the implementation method of Embodiment 1.

[0111] Example 1: Training method for image enhancement model, i.e., obtaining target image enhancement model.

[0112] refer to Figure 2 , Figure 2An exemplary schematic diagram of a first type of training enhancement model 20 is provided in this application embodiment. The training enhancement model 20 includes: a generator network 21, a discriminator network 22, and a supervised optimization network 23.

[0113] The generator network 21 outputs at least two images, including a first-class output image with the same scale as the input synthetic underwater image and a second-class output image with a smaller scale than the synthetic underwater image. The discriminator network 22 determines the adversarial loss based on the first-class output image and the corresponding original image. The supervised optimization network 22 determines the corresponding pixel-level loss based on the at least two output images and their corresponding reference images of the same scale, wherein the reference images include the original image and an image obtained by downsampling the original image. The original image is the original image corresponding to the synthetic underwater image, i.e., the image before the synthetic underwater image is synthesized. The training enhancement model is adjusted at least once using the loss function constructed based on the adversarial loss and the at least two pixel-level losses.

[0114] In one possible implementation, the generator network 21 consists of a second encoder 210, a second attention network 211, and a second decoder 212, with the aim of learning the mapping relationship between underwater images and clear atmospheric style images to generate high-quality target underwater images of underwater scenes. (Reference) Figure 3 , Figure 3 An exemplary schematic diagram of the second type of enhancement model to be trained is provided in an embodiment of this application;

[0115] The second encoder 210 is used to extract features from the input synthetic underwater image to obtain at least two training feature maps. The second encoder 210 includes at least two sets of convolutional layers, and downsampling operations are performed between adjacent sets of convolutional layers. Figure 4 , Figure 4 An exemplary schematic diagram of a second encoder according to an embodiment of this application is provided. This diagram illustrates a second encoder 210 comprising five sets of convolutional layers, wherein the five sets of convolutional layers undergo four downsampling operations. All convolutional layers in the second encoder are cascades of convolution operations, instance regularization operations, and linear rectified function activation operations. The downsampling operation is a max pooling operation. The convolution operation is at least one of a 3×3 kernel, a 5×5 kernel, or a 7×7 kernel. It should be noted that... Figure 4 This is merely an illustrative example; the second encoder provided in this application may also contain more convolutional layers, etc.

[0116] The second attention network 211 is used to perform attention processing on the training feature map output by the second encoder and obtain the corresponding training attention map. The second attention network is a hybrid domain second attention network, including a channel attention network and a spatial attention network; that is, the second attention network applies the attention mechanism sequentially to both the channel and spatial dimensions to obtain the corresponding training attention map. (Reference) Figure 5 , Figure 5 An exemplary schematic diagram of a second attention network in an embodiment of this application is provided, in which channel attention processing is performed first and then spatial attention processing is performed as an example; in this embodiment of the application, by adding the hybrid domain second attention network to the skip connection structure of the second encoder and the second decoder, the content information enhancement and underwater tone removal of the underwater scene are effectively achieved.

[0117] The second decoder 212 is used to perform feature concatenation processing on at least two training attention maps output by the second encoder and training upsampled feature maps of the same scale, and output at least two output images; wherein, the second decoder 212 includes at least two sets of feature concatenation layers, a set of convolutional layers connected to each set of feature concatenation layers, and an image output layer connected to at least two sets of convolutional layers, and an upsampling operation is performed between adjacent sets of convolutional layers, as referenced. Figure 6 , Figure 6 An exemplary schematic diagram of a first type of second decoder in this application embodiment is provided. This diagram illustrates a second decoder 212 comprising four sets of feature concatenation layers, five sets of convolutional layers, and three sets of image output layers, with the five sets of convolutional layers undergoing four upsampling operations. It should be noted that... Figure 6 This is merely an illustrative example; the second decoder provided in this application may also contain more convolutional layers, etc.

[0118] All convolutional layers in the second decoder are cascades of convolution operations, instance regularization operations, and linear rectified function activation operations. The convolution operation is at least one of the following: a convolution operation with a kernel size of 3×3, a convolution operation with a kernel size of 5×5, or a convolution operation with a kernel size of 7×7. The feature concatenation layer is a concatenation operation of two feature matrices of the same scale along the channel dimension. The image output layer is a convolution operation with a kernel size of 1×1. The upsampling operation is a deconvolution operation.

[0119] It should be noted that, in the embodiments of this application, the upsampling operation in the second decoder can be replaced by a bilinear interpolation operation.

[0120] In one possible implementation, the discriminator network 22 is used to compare the first-class output image, which is the same scale as the synthetic underwater image output by the generator network, with the original image to determine the corresponding adversarial loss, so as to adjust the network parameters of the training enhancement network through the adversarial loss, so that the output image after image enhancement is closer to the original image.

[0121] In this embodiment, the discriminant network uses a Markov discriminator architecture, referencing... Figure 7 , Figure 7 An exemplary schematic diagram of a discriminative network in an embodiment of this application is provided, wherein the discriminative network includes five convolutional layers as an example. The first four convolutional layers are cascades of convolution operations, instance regularization operations, and linear rectified function activation operations. The convolution operations are at least one of the following: a convolution operation with a kernel size of 3×3, a convolution operation with a kernel size of 5×5, and a convolution operation with a kernel size of 7×7. The last convolutional layer is a convolution operation with a kernel size of 3×3.

[0122] In this embodiment, the optimized supervision network is used to determine the corresponding pixel-level loss based on at least two output images and their corresponding reference images of the same scale. This allows for the adjustment of the network parameters of the training enhancement network through multiple pixel-level losses, thereby achieving multi-scale output and multi-supervision processes. This further enables high-resolution image reconstruction from coarse to fine, improving the clarity and accuracy of the target underwater image after underwater image enhancement.

[0123] For details, please refer to Figure 8 , Figure 8 The embodiment provides a schematic diagram of the specific structure of a training enhancement model according to the embodiment of this application. The various parts included in the training enhancement model and their corresponding functions are described above.

[0124] based on Figure 8 The illustrated augmentation model to be trained is further described in this embodiment of the application, along with a model training method for training the augmentation model to obtain a successfully trained target augmentation model. (Reference) Figure 9 , Figure 9 An exemplary flowchart of a method for training a model to be augmented, as described in this application, includes the following steps:

[0125] Step S900: Select training sample pairs from the training dataset, wherein the training sample pairs include: the original image and the synthetic underwater image corresponding to the original image;

[0126] In this embodiment, the enhancement model to be trained is implemented based on the PyTorch deep learning framework, and deep learning is a data-driven modeling approach. Therefore, during model training, the training dataset is first determined. To meet the model's requirements for training data, images from the NYU-V2 dataset are used as the original images, and a corresponding synthetic underwater image is generated for each original image using an underwater image physical imaging model. Then, the original images and the corresponding synthetic underwater images are used as training sample pairs to train the enhancement model.

[0127] Step S901: Input the synthesized underwater image into the second encoder, and extract features from the synthesized underwater image through the second encoder to obtain at least two training feature maps;

[0128] refer to Figure 8 When a synthetic underwater image is input into the encoder, it first performs feature extraction on the synthetic underwater image through a set of convolutional layers to obtain a first type of training feature map with the same scale as the synthetic underwater image. Then, it performs a downsampling operation on the first type of training feature map to obtain a training downsampled feature map. It then performs feature extraction on the training downsampled feature map through another set of convolutional layers to obtain a second type of training feature map with a scale smaller than the synthetic underwater image.

[0129] In one possible implementation, at least one downsampling operation is performed in the second encoder, so that at least two training feature maps can be obtained through the second encoder.

[0130] Step S902: Input the acquired at least two training feature maps into the second attention network. Through the second attention network, perform attention processing on the at least two training feature maps respectively to obtain the corresponding training attention maps.

[0131] Since the second attention network provided in this application embodiment consists of a channel attention network and a spatial attention network, when performing attention processing on the training feature map through the second attention network: first, the training feature map is processed by the channel attention network to obtain a channel training attention map; then, the channel training attention map is multiplied with the training feature map to obtain an intermediate training feature map; next, the intermediate training feature map is processed by the spatial attention network to obtain a spatial training attention map; finally, the spatial training attention map is multiplied with the intermediate training feature map to obtain the corresponding training attention map.

[0132] The following section will take the example of performing attention processing on a training feature map to obtain the corresponding training attention map, and provide a detailed explanation.

[0133] The training feature map in the second encoder is defined as F, with dimensions h×w×c, where h represents height, w represents width, and c represents the number of channels. First, channel attention processing is applied to F, followed by spatial attention processing on the result. Channel attention focuses on which channel features are meaningful. First, the training feature map is compressed in the spatial dimension (h×w), and spatial information is aggregated using max pooling and global average pooling to obtain two 1×1×c channel descriptors. Then, these two 1×1×c channel descriptors are input into a shared network consisting of a multilayer perceptron (MLP) to obtain the transformed output. Next, the two sets of feature vectors are element-wise added and merged, and then passed through a sigmoid function (σ) to obtain the weight coefficients M of the normalized channel training attention map. c This process can be represented by the following formula:

[0134] M c (F)=σ(MLP(MaxPool(F))+MLP(AvgPool(F))

[0135] Finally, the weighting coefficient M c Multiplying the intermediate training feature map F by the channel attention mechanism yields a new feature matrix corresponding to the intermediate training feature map F′, thus achieving the goal of focusing on meaningful channels and ignoring useless channels.

[0136] Spatial attention networks take an intermediate training feature map F′ (with dimensions h×w×c) that has already undergone channel attention processing as input, focusing on which locations in the feature map are regions of interest. Similar to the implementation principle of channel attention, spatial attention networks first perform max pooling and global average pooling operations along the channel dimension on the input intermediate training feature map F′, obtaining two feature maps of size h×w×1. These two h×w×1 feature maps are then concatenated along the channel dimension to generate a feature descriptor. Afterward, a 7×7 convolutional layer is applied, activated using the sigmoid function (σ), to obtain the weight coefficients M of a spatial training attention map. s The process is represented as:

[0137] M s (F)=σ(f 7×7 ([MaxPool(F);MLP(AvgPool(F)]))

[0138] Finally, the weighting coefficient M s Multiplying the input intermediate training feature map F′ yields a new feature matrix of the training attention map after spatial attention mechanism, achieving the goal of focusing on meaningful regions and ignoring useless regions.

[0139] Step S903: Input the acquired at least two training attention maps into the corresponding second decoder. Through the feature concatenation layer in the second decoder, based on the scale of the acquired at least two training attention maps, the training attention maps are concatenated with training upsampled feature maps of the same scale in order of increasing scale to obtain at least two output images.

[0140] Among them, at least two output images have different scales, that is, at least two output images have different resolutions; and the at least two output images include: a first type of output image with the same scale as the synthesized underwater image, and a second type of output image with a smaller scale than the synthesized underwater image.

[0141] It should be noted that in the embodiments of this application, scale can be replaced with resolution.

[0142] refer to Figure 8 Taking a set of feature concatenation layers as an example, the process of obtaining an output image is explained in detail: the training attention map and the training upsampled feature map are input into the feature concatenation layer, the training attention map and the training upsampled feature map are concatenated in the feature concatenation layer, the concatenated feature map is then processed through the convolutional layer, and finally the corresponding output image is output through the image output layer.

[0143] It should be noted that, Figure 8 The diagram shown illustrates the structure for obtaining three output images using three sets of convolutional layers. Figure 8 This is merely an illustrative example and not the only manifestation.

[0144] Step S904: The first type of output image, which is the same scale as the synthesized underwater image, is input into the discriminative network. Based on the first comparison result between the first type of output image and the original image, the adversarial loss is determined by the discriminative network.

[0145] During model training, the first-class output image from the generator network and its corresponding original image are input into the discriminator network to obtain a 1×16×16 discriminator matrix. Each element of the matrix corresponds to a relatively large receptive field on the two input images. The two input images are discriminated within each local receptive field, and the adversarial loss is determined. Through joint optimization of the discriminator network and the generator network, high-frequency supervision information such as style and local semantic content is provided for the underwater image enhancement task.

[0146] Step S905: Input at least two acquired output images into a supervised optimization network. Through the supervised optimization network, based on the second comparison result between the first type of output image and the original image, and the third comparison result between the second type of output image and a reference image of the same scale, determine the corresponding pixel-level loss respectively. The reference image is obtained by downsampling the original image.

[0147] In this embodiment, for the second type of low-resolution output images, reference images downsampled from the original image to their respective resolutions are used for supervision. For the first type of high-resolution output images, the original image is used for supervision. Through multi-scale output and multi-supervision processes, high-resolution image reconstruction from coarse to fine is achieved, improving the clarity and accuracy of the enhancement results.

[0148] Step S906: Construct a loss function based on the adversarial loss and at least two determined pixel-level losses;

[0149] In this embodiment, the loss function can be expressed as:

[0150] L = L adv +λL multi

[0151] Among them, L adv To combat the losses, L multi For multi-scale pixel loss, λ is the weight parameter corresponding to the multi-scale pixel loss, which is determined by cross-validation.

[0152] Specifically, L adv It can be represented as:

[0153] L adv =E{[logD(x i )]+[log(1-D(G(y i )))]}

[0154] Where, x i Represents the original image, y i For x i Synthetic underwater images generated using underwater physical imaging models, G(y i ) represents the enhanced result of the generated network output, D(G(y) i )) and D(x i The result is used to determine the discrimination result output by the discrimination network.

[0155] L multi It can be represented as:

[0156] L multi =τ1L p1 +τ2L p2 +τ3Lp3

[0157] Among them, L p1 L p2 L p3 The pixel-level loss of the model is represented at three scales. To ensure the clarity of the generated image, the L1 loss function is used for all pixel-level losses. τ1, τ2, and τ3 represent the weights of the pixel loss of the reconstruction results at different scales from high to low in the multi-scale pixel loss, and are determined using cross-validation.

[0158] It should be noted that in the embodiments of this application, the hyperparameters of the loss function are set to 10, 0.6, 0.3, and 0.1, respectively.

[0159] Step S907: Adjust the parameters of the training augmentation model at least once using the loss function until the conditions are met, and obtain the target augmentation model.

[0160] The conditions can be that the convergence condition is met, or that the number of adjustments reaches the upper limit. For example, if the number of adjustments is set to 100, the target augmentation model is obtained when the number of parameter adjustments reaches 100.

[0161] In this embodiment of the application, based on the loss function, the optimizer performs at least one parameter adjustment on the augmentation model to be trained; wherein, the optimizer is the ADAM optimizer, the momentum decay index β_1 and β_2 of the optimizer are set to 0.50 and 0.999 respectively, and the initial learning rate is 0.001.

[0162] In this application, a loss function consisting of adversarial loss and pixel loss is used to jointly complete the optimization process of the augmentation model to be trained, so that the augmentation model to be trained can obtain a clear and accurate augmented image that is consistent with the content structure of the original image.

[0163] In this embodiment of the application, in order to ensure the accuracy of model training, an error feedback network is introduced into the second decoder to realize error feedback correction in the feature reconstruction process, so as to maintain the semantic consistency and other intrinsic connections between feature maps at different scales as much as possible, and to use a multi-scale stepwise recovery and reconstruction error feedback network to prevent the loss of image detail information and improve the clarity and accuracy of the image obtained after underwater image enhancement processing.

[0164] refer to Figure 10 , Figure 10 An exemplary schematic diagram of a second type of second decoder is provided in an embodiment of this application. The second decoder introduces an error feedback network. The error feedback network is used to determine the error between images, and includes a convolutional layer and a deconvolutional layer.

[0165] Therefore, this application provides a trainable enhancement model including an error feedback network, for reference. Figure 11 , Figure 11 An exemplary schematic diagram of another training enhancement model is provided in this application embodiment.

[0166] based on Figure 11 The illustrated augmentation model to be trained is presented in this embodiment of the application, which proposes another model training method to train the augmentation model to obtain a successfully trained target augmentation model. (Reference) Figure 12 , Figure 12 An exemplary flowchart of a method for training a model to be augmented, as described in this application, includes the following steps:

[0167] Step S1200: Select training sample pairs from the training dataset, wherein the training sample pairs include: the original image and the synthetic underwater image corresponding to the original image.

[0168] Step S1201: Input the synthesized underwater image into the second encoder, and extract features from the synthesized underwater image through the second encoder to obtain at least two training feature maps.

[0169] In step S1202, the at least two training feature maps are input into the second attention network. The second attention network performs attention processing on the at least two training feature maps respectively to obtain the corresponding training attention maps.

[0170] Step S1203: Input the acquired at least two training attention maps into the corresponding second decoder. Through the feature concatenation layer in the second decoder, based on the scale of the acquired at least two training attention maps, the training attention maps are concatenated with training upsampled feature maps of the same scale in order of increasing scale to obtain at least two concatenated images.

[0171] Step S1204: After passing the acquired at least two stitched images through a convolutional layer, they are input into the error feedback network of the corresponding second decoder. After the stitched images after the convolutional operation are corrected by the error feedback network, at least two output images are output through the image output layer.

[0172] The following section uses a stitched image as an example to explain in detail how to correct the stitched image after convolution operation using an error feedback network.

[0173] In one possible implementation, the training upsampled feature map corresponding to the stitched image is used as the training high-scale feature map f. H′, that is, the high-scale feature map obtained only after upsampling and other operations, and the feature map of the training upsampled feature map corresponding to the stitched image before upsampling is used as the training low-scale feature map f. L This refers to the low-scale feature map output from the previous layer in the second decoder; at this point, the stitched image after the convolution operation is corrected through an error feedback network, including:

[0174] For training high-scale feature maps f H Perform downsampling to obtain the corresponding downsampled feature map, and determine the relationship between the downsampled feature map and the training low-scale feature map f. L The feature error between the two features, where the downsampled feature map and the low-scale feature map have the same scale;

[0175] Deconvolution is performed on the feature error to obtain the high-scale training feature map f. H The high-scale error corresponding to ′ is summed with the stitched image after convolution to obtain the training high-scale reconstructed feature map f after error feedback correction and optimization. H .

[0176] Because f L with f H Both ' and ' represent the enhanced feature maps, so the only difference between them is in scale. If the high-scale feature f is trained... H The low-scale features f are obtained again by downsampling through convolutional layers. L It should be compared with the original training low-scale feature map f. L Completely identical. However, in model training and application, due to feature loss and bias caused by processes such as upsampling, f L ′ and f L There will be errors. This is addressed by calculating the feature error E between two low-scale features. L By performing a deconvolution operation on it, we can obtain E. L The corresponding high-scale error E H E H By summing the images obtained from the convolution operation with the stitched images, feature correction of the upsampling process can be achieved, resulting in a high-scale reconstructed feature map f after error feedback correction and optimization. H This is to minimize feature loss during the decoding process and optimize model performance.

[0177] In step S1205, the first type of output image, which is the same scale as the synthesized underwater image, is input into the discriminant network. Based on the first comparison result between the first type of output image and the original image, the adversarial loss is determined by the discriminant network.

[0178] Step S1206: Input at least two acquired output images into a supervised optimization network. Through the supervised optimization network, based on the second comparison result between the first type of output image and the original image, and the third comparison result between the second type of output image and a reference image of the same scale, determine the corresponding pixel-level loss respectively. The reference image is obtained by downsampling the original image.

[0179] Step S1207: Construct a loss function based on the adversarial loss and at least two determined pixel-level losses.

[0180] Step S1208: Adjust the parameters of the training augmentation model at least once using the loss function until the conditions are met, and then obtain the target augmentation model.

[0181] It should be noted that, Figure 12 For steps not explained in the text, please refer to the reference. Figure 9 This will not be repeated here.

[0182] In this application, when training the augmentation network to be trained, the loss function is mainly constructed based on the adversarial loss corresponding to the output image with the same scale as the input image in at least two output images of the augmentation model to be trained, and the pixel-level loss corresponding to each of the at least two output images. The model to be trained is adjusted at least once to jointly complete the optimization of the augmentation model to be trained, so as to ensure that the target augmentation model obtained after optimization can obtain a clear and accurate augmented image that is consistent with the content structure of the input image. The clear and accurate features include the characteristics of complete color shift recovery and sufficient detail enhancement in the augmented image obtained after image augmentation.

[0183] Based on the above model training method, after obtaining a successfully trained target enhancement model, the underwater image to be processed is enhanced using the target enhancement model. Below, with reference to the implementation method of Example 2, the enhancement processing of underwater images based on the target image enhancement model will be described in detail.

[0184] Example 2: Image enhancement method, which enhances underwater images based on a target image enhancement model.

[0185] In this application embodiment, a target enhancement model for image enhancement processing is proposed, with reference to... Figure 13 , Figure 13An exemplary schematic diagram of a target enhancement model is provided in this application embodiment. The target enhancement model is obtained through the model training method provided in Embodiment 1 above. That is, the target enhancement model is obtained by adjusting the parameters of the enhancement model to be trained at least once based on the loss function. The loss function is constructed based on the adversarial loss corresponding to the output image with the same scale as the input image in at least two output images of the enhancement model to be trained, and the pixel-level loss corresponding to each of the at least two output images. The scales of the at least two output images are different. The specific training process will not be repeated here.

[0186] refer to Figure 14 , Figure 14 An exemplary flowchart of an image enhancement method according to an embodiment of this application is provided, which includes the following steps:

[0187] Step S1400: Obtain the underwater image to be processed;

[0188] The underwater image to be processed can be a real underwater image obtained through a terminal device, such as a real underwater image captured by the terminal device or a real underwater image obtained by the terminal device through a search method, or a synthetic underwater image generated using an underwater image physical imaging model; that is, the embodiments of this application are applicable to scenarios of image enhancement for real underwater images and synthetic underwater images.

[0189] Step S1401: Based on the target enhancement model, perform image enhancement processing on the underwater image to be processed to obtain the enhanced underwater image of the target.

[0190] To verify that the image enhancement method provided in this application is superior to other image enhancement methods in related technologies, in this embodiment, at least one synthetic underwater image is enhanced using a target enhancement model, and the target underwater image obtained after image enhancement is compared with underwater images enhanced by other image enhancement methods. Figure 15 , Figure 15 An illustrative diagram of the enhancement results of each algorithm on the synthetic underwater image is provided; and the enhancement index data of the target underwater image after enhancement processing by the target enhancement model is compared with the enhancement index data of the underwater image after enhancement processing by other image enhancement methods, as shown in Table 1, which provides an illustrative comparison result of the enhancement index data of each algorithm on the synthetic underwater image.

[0191] Table 1

[0192]

[0193] Similarly, in this embodiment, at least one real underwater image is enhanced using a target enhancement model, and the enhanced underwater image is compared with an underwater image enhanced using other image enhancement methods. Figure 16 , Figure 16 An illustrative diagram of the enhancement results of each algorithm on a real underwater image is provided; and the enhancement index data of the target underwater image after enhancement processing by the target enhancement model is compared with the enhancement index data of the underwater image after enhancement processing by other image enhancement methods, as shown in Table 2, which provides an illustrative comparison result of the enhancement index data of each algorithm on a real underwater image.

[0194] Table 2

[0195]

[0196] The comparative data above shows that the target enhancement model provided in this application achieves a peak signal-to-noise ratio (PSNR) of 23.9484 dB and a structural similarity index (SSIM) of 0.9032 for synthetic underwater images. For real underwater images, the Underwater Image Quality Measurement (UIQM) index reaches 4.5284, which is superior to the other image enhancement methods compared.

[0197] In this embodiment of the application, when obtaining the enhanced underwater image of the target based on the target enhancement model, the underwater image to be processed can be a real underwater image or a synthetic underwater image. However, regardless of whether it is a real underwater image or a synthetic underwater image, the operation flow of the image enhancement method provided in this embodiment of the application remains unchanged.

[0198] refer to Figure 17 , Figure 17 An exemplary flowchart of a specific implementation method for image enhancement in this application is provided, including the following steps:

[0199] Step S1700: Obtain the underwater image to be processed.

[0200] Step S1701: Input the underwater image to be processed into the first encoder of the target enhancement model, and extract features from the underwater image to be processed through the first encoder to obtain at least two initial feature maps.

[0201] In one possible implementation, a first encoder extracts features from the underwater image to be processed to obtain a first type of initial feature map with the same scale as the underwater image to be processed, and a second type of initial feature map is obtained by downsampling the first type of initial feature map at least once.

[0202] Step S1702: Input the acquired at least two initial feature maps into the first attention network of the target enhancement model. Through the first attention network, perform attention processing on the at least two initial feature maps respectively to obtain the corresponding target attention maps.

[0203] In one possible implementation, for any initial feature map, channel attention processing is performed on the initial feature map to obtain a channel attention map, and the channel attention map is fused with the initial feature map to obtain an intermediate feature map.

[0204] Spatial attention processing is performed on the intermediate feature map to obtain a spatial attention map. The spatial attention map is then fused with the intermediate feature map to obtain the corresponding target attention map.

[0205] Step S1703: Input the acquired at least two target attention maps into the first decoder of the target enhancement model, and respectively use the feature concatenation layer in the first decoder to concatenate the acquired target attention maps with the input upsampled feature maps of the same scale to obtain at least two concatenated images.

[0206] Step S1704: Perform convolution operation on at least two stitched images through the convolutional layer in the first decoder to determine the stitched image after the convolution operation;

[0207] Step S1705: Use the input upsampled feature map corresponding to the stitched image as the high-scale feature map, and use the feature map without upsampling corresponding to the input upsampled feature map as the low-scale feature map.

[0208] Step S1706: Downsample the high-scale feature map to obtain a downsampled feature map, and determine the feature error between the downsampled feature map and the low-scale feature map. The downsampled feature map and the low-scale feature map have the same scale.

[0209] Step S1707: Perform deconvolution on the feature error to obtain the high-scale error corresponding to the high-scale feature map, and sum the high-scale error with the stitched image after the convolution operation to obtain the feature reconstruction map.

[0210] Step S1708: After determining that the feature reconstruction map after summing is at the same scale as the underwater image to be processed, the target underwater image is determined and output based on the stitched image.

[0211] It should be noted that steps S1703 to S1707 are based on the scale of the acquired at least two target attention maps, and the target attention maps are processed sequentially in order of increasing scale.

[0212] In this application, a loss function is first constructed based on the adversarial loss corresponding to the output image with the same scale as the input image from at least two output images of the enhancement model to be trained, and the pixel-level loss corresponding to each of the at least two output images. At least one parameter adjustment is then performed on the enhancement model to be trained to jointly optimize it and obtain the target enhancement model. This ensures that the optimized target enhancement model yields a clear, accurate, and structurally consistent enhanced image with the input image. Clearness and accuracy include complete color shift recovery and sufficient detail enhancement in the enhanced image. Then, based on the target enhancement model, image enhancement processing is performed on the acquired underwater image to be processed to obtain the enhanced target underwater image. At this point, the target underwater image is a clear, accurate, and structurally consistent enhanced image with the underwater image to be processed. Therefore, the image enhancement method provided in this application improves the image enhancement effect of underwater images.

[0213] Example 3: Image enhancement device.

[0214] Based on the same inventive concept as the above-described method embodiments, this application also provides an image enhancement device. The principle of the device in solving the problem is similar to that of the above-described method. Therefore, the implementation of the device can refer to the implementation of the above-described method, and repeated details will not be described again.

[0215] Please refer to Figure 18 , Figure 18 An exemplary embodiment of this application provides an image enhancement device 1800, which includes:

[0216] Acquisition unit 1801 is used to acquire underwater images to be processed;

[0217] Processing unit 1802 is used to perform image enhancement processing on the underwater image to be processed based on the target enhancement model, and obtain the enhanced underwater image of the target.

[0218] The target augmentation model is obtained by adjusting the parameters of the augmentation model to be trained at least once based on the loss function. The loss function is constructed based on the adversarial loss determined by the first type of output image with the same scale as the input image in at least two output images of the augmentation model to be trained, and the pixel-level loss corresponding to each of the at least two output images. The at least two output images have different scales.

[0219] In one possible implementation, the target augmentation model includes: a first encoder, a first attention network, and a first decoder;

[0220] Processing unit 1802 is specifically used for:

[0221] The first encoder extracts features from the underwater image to be processed, obtaining at least two initial feature maps.

[0222] Through the first attention network, attention processing is performed on at least two initial feature maps to obtain the corresponding target attention map;

[0223] Using the first decoder, based on the scale of at least two acquired target attention maps, the target attention maps are sequentially stitched with upsampled feature maps of the same scale in ascending order of scale until the acquired stitched image has the same scale as the underwater image to be processed. Then, the target underwater image is determined and output based on the stitched image.

[0224] In one possible implementation, the processing unit 1802 is specifically used for:

[0225] The first encoder extracts features from the underwater image to be processed, obtaining a first type of initial feature map with the same scale as the underwater image to be processed, and a second type of initial feature map obtained by downsampling the first type of initial feature map at least once.

[0226] In one possible implementation, the processing unit 1802 is specifically used for:

[0227] For any initial feature map, channel attention processing is performed on the initial feature map to obtain a channel attention map. The channel attention map is then multiplied with the initial feature map to obtain an intermediate feature map.

[0228] Spatial attention processing is applied to the intermediate feature map to obtain a spatial attention map. The spatial attention map is then multiplied with the intermediate feature map to obtain the corresponding target attention map.

[0229] In one possible implementation, the first decoder includes an error feedback network, and uses the input upsampled feature map corresponding to the stitched image as a high-scale feature map, and uses the feature map of the input upsampled feature map before upsampling as a low-scale feature map.

[0230] Processing unit 1802 performs the following operations through the error feedback network:

[0231] The high-scale feature map is downsampled to obtain a downsampled feature map, and the feature error between the downsampled feature map and the low-scale feature map is determined. The downsampled feature map and the low-scale feature map have the same scale.

[0232] The feature error is deconvolved to obtain the high-scale error corresponding to the high-scale feature map, and the high-scale error is summed with the stitched image after the convolution operation.

[0233] In one possible implementation, the image enhancement device also includes a training unit 1803, such as... Figure 19 As shown, another image enhancement device 1900 provided in this application embodiment includes an enhancement model to be trained comprising: a second encoder, a second attention network, a second decoder, a discriminant network, and a supervised optimization network; the target enhancement model is obtained by the training unit 1803 in the following manner:

[0234] Training sample pairs are selected from the training dataset. The training sample pairs include: the original image and the corresponding synthetic underwater image.

[0235] The synthetic underwater image is used to extract features through a second encoder to obtain at least two training feature maps.

[0236] Through the second attention network, attention processing is performed on at least two training feature maps respectively to obtain the corresponding training attention maps;

[0237] The second decoder concatenates at least two training attention maps with corresponding training upsampled feature maps of the same scale to obtain at least two output images. The at least two output images include a first type of output image with the same scale as the synthetic underwater image and a second type of output image with a smaller scale than the synthetic underwater image.

[0238] The adversarial loss is determined by the discriminative network based on the first comparison result between the first type of output image and the original image;

[0239] By using a supervised optimization network, the corresponding pixel-level loss is determined based on the second comparison result between the first type of output image and the original image, and the third comparison result between the second type of output image and the reference image of the same scale. The reference image is obtained by downsampling the original image.

[0240] A loss function is constructed based on adversarial loss and at least two determined pixel-level losses. The parameters of the model to be trained are adjusted at least once using the loss function until the conditions are met, thus obtaining the target augmentation model.

[0241] In one possible implementation, the synthetic underwater image is determined from the original image based on a successfully trained underwater image physical imaging model.

[0242] In one possible implementation, training unit 1803 is specifically used for:

[0243] Based on the loss function, the optimizer performs at least one parameter adjustment on the training enhancement model.

[0244] In this embodiment, a loss function is first constructed based on the adversarial loss corresponding to the output image with the same scale as the input image from at least two output images of the enhancement model to be trained, and the pixel-level loss corresponding to each of the at least two output images. At least one parameter adjustment is then performed on the enhancement model to be trained to jointly optimize it and obtain the target enhancement model. This ensures that the optimized target enhancement model yields a clear, accurate, and structurally consistent enhanced image with the input image. Clearness and accuracy include complete color shift recovery and sufficient detail enhancement in the enhanced image. Then, based on the target enhancement model, image enhancement processing is performed on the acquired underwater image to be processed to obtain the enhanced target underwater image. At this point, the target underwater image is a clear, accurate, and structurally consistent enhanced image with the underwater image to be processed. Therefore, the image enhancement method provided in this application improves the image enhancement effect of underwater images.

[0245] For ease of description, the above sections are divided into units (or modules) according to their functions and described separately. Of course, in implementing this application, the functions of each unit (or module) can be implemented in one or more software or hardware components.

[0246] Example 4: Electronic device.

[0247] After introducing the image enhancement method and apparatus of exemplary embodiments of this application, based on the same inventive concept as the above-described method embodiments of this application, this application also provides an electronic device, which may be a terminal device or a server.

[0248] Next, an electronic device for image enhancement according to another exemplary embodiment of this application will be described.

[0249] In this embodiment, when the electronic device is a server, it can be... Figure 1 The structure of the server 120 shown can be as follows: Figure 20 As shown, it includes a memory 2001, a communication module 2003, and one or more processors 2002.

[0250] The memory 2001 is used to store computer programs executed by the processor 2002. The memory 2001 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0251] Memory 2001 may be volatile memory, such as random-access memory (RAM); memory 2001 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 2001 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 2001 may be a combination of the above-mentioned memories.

[0252] Processor 2002 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 2002 is used to implement the image enhancement method when calling computer programs stored in memory 2001.

[0253] The communication module 2003 is used to communicate with terminal devices and other servers.

[0254] This application embodiment does not limit the specific connection medium between the memory 2001, communication module 2003, and processor 2002. This application embodiment... Figure 20 The memory 2001 and the processor 2002 are connected via a bus 2004, which is in... Figure 20 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered as limiting information. The Bus 2004 can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 20 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0255] The memory 2001 stores a computer storage medium, which stores computer-executable instructions for implementing the image enhancement method of the embodiments of this application.

[0256] In this embodiment, when the electronic device is a terminal device, it can be... Figure 1 The terminal device 110 shown in the figure has the following structure: Figure 21 As shown, it includes components such as: communication component 2110, memory 2120, display unit 2130, camera 2140, sensor 2150, audio circuit 2160, Bluetooth module 2170, processor 2180, etc.

[0257] The communication component 2110 is used to communicate with the server. In some embodiments, it may include a Circuit-Wireless Fidelity (WiFi) module, which is a short-range wireless transmission technology. Electronic devices can use the WiFi module to help users send and receive information.

[0258] The memory 2120 can be used to store software programs and data. The processor 2180 executes various functions of the terminal device 110 and performs data processing by running the software programs or data stored in the memory 2120. The memory 2120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 2120 stores an operating system that enables the terminal device 110 to run. In this application, the memory 2120 may store the operating system and various applications, and may also store code that executes the image enhancement method of the embodiments of this application.

[0259] The display unit 2130 can also be used to display information input by the user or information provided to the user, as well as various menus of the terminal device 110, in a graphical user interface (GUI). Specifically, the display unit 2130 may include a display screen 2132 disposed on the front of the terminal device 110. The display screen 2132 may be configured as a liquid crystal display, a light-emitting diode, or the like. The display unit 2130 can be used to display the underwater image to be processed and the enhanced target underwater image, etc., in the embodiments of this application.

[0260] The display unit 2130 can also be used to receive input digital or character information and generate signal inputs related to user settings and function control of the terminal device 110. Specifically, the display unit 2130 may include a touch screen 2131 disposed on the front of the terminal device 110, which can collect touch operations of the user on or near it, such as clicking a button, dragging a scroll box, etc.

[0261] The touchscreen 2131 can be placed over the display screen 2132, or the touchscreen 2131 and the display screen 2132 can be integrated to realize the input and output functions of the terminal device 110. After integration, it can be referred to as a touch display screen. In this application, the display unit 2130 can display the application program and the corresponding operation steps.

[0262] Camera 2140 can be used to capture still images, and users can transmit underwater images captured by camera 2140 to other devices for processing. There can be one or multiple cameras 2140. An object is projected onto a photosensitive element through a lens, generating an optical image. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to processor 2180 for conversion into a digital image signal.

[0263] The terminal device may also include at least one sensor 2150, such as an accelerometer 2151, a proximity sensor 2152, a fingerprint sensor 2153, and a temperature sensor 2154. The terminal device may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, and motion sensor.

[0264] Audio circuitry 2160, speaker 2161, and microphone 2162 provide an audio interface between the user and terminal device 110. Audio circuitry 2160 converts received audio data into electrical signals, which are then transmitted to speaker 2161, where they are converted into sound signals for output. Terminal device 110 may also be equipped with volume buttons for adjusting the volume of the sound signal. On the other hand, microphone 2162 converts collected sound signals into electrical signals, which are then received by audio circuitry 2160, converted into audio data, and output to communication component 2110 for transmission to, for example, another terminal device 110, or to memory 2120 for further processing.

[0265] Bluetooth module 2170 is used to interact with other Bluetooth devices that also have Bluetooth modules via the Bluetooth protocol. For example, a terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smartwatch) that also has a Bluetooth module through Bluetooth module 2170, thereby exchanging data.

[0266] The processor 2180 is the control center of the terminal device, connecting various parts of the terminal through various interfaces and lines. It executes various functions and processes data by running or executing software programs stored in the memory 2120 and calling data stored in the memory 2120. In some embodiments, the processor 2180 may include one or more processing units; the processor 2180 may also integrate an application processor and a baseband processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the baseband processor mainly handles wireless communication. It is understood that the baseband processor may not be integrated into the processor 2180. In this application, the processor 2180 can run the operating system, applications, user interface display and touch response, as well as the image enhancement method of this embodiment. Furthermore, the processor 2180 is coupled to the display unit 2130.

[0267] Example 5: Program Product.

[0268] In some possible implementations, various aspects of the image enhancement method provided in this application may also be implemented in the form of a program product, which includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of the image enhancement method according to the various exemplary embodiments of this application described above.

[0269] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0270] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on a computing device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0271] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0272] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0273] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0274] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0275] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0276] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing a computer-usable computer program.

[0277] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0278] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image enhancement method characterized by, The method comprises: obtaining an underwater image to be processed; performing image enhancement processing on the underwater image to be processed based on a target enhancement model to obtain a target underwater image after image enhancement; wherein the target enhancement model is obtained by performing at least one parameter adjustment on a to-be-trained enhancement model based on a loss function, the loss function is constructed based on an adversarial loss corresponding to a first type of output image of the at least two output images output by the to-be-trained enhancement model and having the same scale as the input image, and a pixel-level loss corresponding to each of the at least two output images, the scales of the at least two output images are different, and the pixel-level loss of each output image is determined based on the output image and a corresponding reference image having the same scale, the reference image includes an original image and an image obtained by downsampling the original image; the target enhancement model comprises a first encoder, a first attention network, and a first decoder; the image enhancement processing on the underwater image to be processed based on the target enhancement model comprises: performing feature extraction on the underwater image to be processed by the first encoder to obtain at least two initial feature maps; performing attention processing on the at least two initial feature maps respectively by the first attention network to obtain corresponding target attention maps; sequentially concatenating the target attention maps with upsampled feature maps having the same scale based on the scale of the at least two target attention maps obtained in a manner from small to large by the first decoder until the obtained concatenated image has the same scale as the underwater image to be processed, and determining the target underwater image based on the concatenated image and outputting the target underwater image; the first decoder comprises an error feedback network, and the input upsampled feature map corresponding to the concatenated image is used as a high-scale feature map, and the feature map corresponding to the input upsampled feature map before being upsampled is used as a low-scale feature map; after the target attention map is concatenated with the upsampled feature map having the same scale, the error feedback network performs the following operations: downsampling the high-scale feature map to obtain a downsampled feature map and determining a feature error between the downsampled feature map and the low-scale feature map, the downsampled feature map and the low-scale feature map having the same scale; performing deconvolution operation on the feature error to obtain a high-scale error corresponding to the high-scale feature map, and performing addition processing on the high-scale error and the concatenated image after convolution operation.

2. The method of claim 1, wherein, the feature extraction on the underwater image to be processed by the first encoder to obtain at least two initial feature maps comprises: performing feature extraction on the underwater image to be processed by the first encoder to obtain a first type of initial feature map having the same scale as the underwater image to be processed, and a second type of initial feature map obtained by performing at least one downsampling on the first type of initial feature map.

3. The method of claim 1, wherein, the attention processing on the at least two initial feature maps respectively comprises: Channel attention processing is performed on any initial feature map to obtain a channel attention map, and the channel attention map is multiplied with the initial feature map to obtain an intermediate feature map; The intermediate feature map is subjected to spatial attention processing to obtain a spatial attention map, and the spatial attention map is multiplied with the intermediate feature map to obtain a corresponding target attention map.

4. The method according to any one of claims 1 to 3, characterized in that The to-be-trained enhancement model comprises a second encoder, a second attention network, a second decoder, a discriminator network and a supervised optimization network; and the target enhancement model is obtained in the following manner: A training sample pair is selected from a training data set, and the training sample pair comprises an original image and a corresponding synthesized underwater image; Feature extraction is performed on the synthesized underwater image by the second encoder to obtain at least two training feature maps; The at least two training feature maps are subjected to attention processing by the second attention network to obtain corresponding training attention maps; The at least two training attention maps are respectively spliced with corresponding training up-sampling feature maps of the same scale by the second decoder to obtain at least two output images, wherein the at least two output images comprise a first type of output image of the same scale as the synthesized underwater image and a second type of output image of a smaller scale than the synthesized underwater image; An adversarial loss is determined based on a first comparison result between the first type of output image and the original image by the discriminator network; Pixel-level losses are respectively determined based on a second comparison result between the first type of output image and the original image and a third comparison result between the second type of output image and a reference image of the same scale by the supervised optimization network, wherein the reference image is obtained by down-sampling the original image; A loss function is constructed based on the adversarial loss and the at least two determined pixel-level losses, and at least one parameter adjustment is performed on the to-be-trained enhancement model by the loss function until a condition is met, thereby obtaining the target enhancement model.

5. The method of claim 4, wherein, The synthesized underwater image is determined based on a successfully trained underwater image physical imaging model of the original image.

6. The method of claim 4, wherein, The at least one parameter adjustment performed on the to-be-trained enhancement model based on the loss function comprises: The at least one parameter adjustment performed on the to-be-trained enhancement model by the optimizer based on the loss function.

7. An image enhancement device, characterized by The method comprises the following steps: An acquisition unit is configured to acquire a to-be-processed underwater image; A processing unit is configured to perform image enhancement processing on the to-be-processed underwater image based on a target enhancement model to obtain a target underwater image after image enhancement. The target enhancement model is obtained by performing at least one parameter adjustment on a to-be-trained enhancement model based on a loss function, the loss function is constructed based on an adversarial loss determined based on a first type of output image of the at least two output images output by the to-be-trained enhancement model and a pixel-level loss corresponding to each of the at least two output images, the at least two output images have different scales, and the pixel-level loss of each output image is determined based on the output image and a reference image of the same scale, the reference image includes an original image and an image obtained by down-sampling the original image; The target enhancement model includes a first encoder, a first attention network, and a first decoder. The processing unit is specifically configured to: extract features of the to-be-processed underwater image through the first encoder to obtain at least two initial feature maps; perform attention processing on the at least two initial feature maps through the first attention network to obtain corresponding target attention maps; based on the scale of the at least two target attention maps, sequentially concatenate the target attention maps with up-sampled feature maps of the same scale in a descending order of scale, until the scale of the concatenated image is the same as that of the to-be-processed underwater image, determine the target underwater image based on the concatenated image, and output the target underwater image; The first decoder includes an error feedback network, and the input up-sampled feature map corresponding to the concatenated image is used as a high-scale feature map, and the feature map corresponding to the input up-sampled feature map before being up-sampled is used as a low-scale feature map. The processing unit performs the following operations through the error feedback network: down-sample the high-scale feature map to obtain a down-sampled feature map, and determine a feature error between the down-sampled feature map and the low-scale feature map, the down-sampled feature map and the low-scale feature map have the same scale; perform deconvolution on the feature error to obtain a high-scale error corresponding to the high-scale feature map, and add the high-scale error to the concatenated image after convolution.

8. An electronic device, comprising: comprise: a memory for storing a computer program executable by a processor; the processor is connected with the memory and is configured to execute the method of any one of claims 1-6.

Images