Real-time enhancement method and electronic equipment for high-resolution underwater images

Abstract

The present disclosure provides a method and electronic device for real-time enhancement of high-resolution underwater images, and relates to the technical field of underwater image processing. Underwater images are acquired in real time; a dual-branch underwater image enhancement network is constructed to enhance the underwater images, wherein the dual-branch underwater image enhancement network includes an upper branch network and a lower branch network; the lower branch network guides the upper branch network to perform image enhancement, and obtains the enhanced low-resolution image of the lower branch network and the enhanced high-resolution image of the upper branch network respectively. By guiding the upper branch through the lower branch to complete the overall underwater image enhancement, images with good visual effects can be obtained, and real-time enhancement of high-resolution underwater images and videos can be achieved.

Description

Background Art

[0002] During underwater exploration, underwater images directly acquired by underwater robots usually have problems such as color cast and unclearness. Compared with low-resolution underwater images, high-resolution underwater images and videos contain more detailed information, which helps to understand the marine environment more intuitively.

[0003] In order to solve the problem of unclear underwater images, with the development of deep learning, a variety of underwater image enhancement methods have been proposed, such as the fast underwater image restoration network based on CNN and GAN to achieve real-time enhancement of underwater images and videos. Although these existing models have good restoration effects, they rarely consider the importance of high-frequency information to the image, nor the impact of the number of channels on the model speed. As a result, the amount of calculation is too large or the model is too large, resulting in the inference speed being unable to break through real-time, and the real-time processing of high-resolution images cannot be achieved. It also cannot meet the needs of underwater robots to process high-resolution underwater images in real time.

[0004] Therefore, there is an urgent need for technical solutions that can enhance high-resolution underwater images in real time and efficiently.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide a method and electronic device for real-time enhancement of high-resolution underwater images, which at least to some extent overcome the problem that related technologies cannot process high-resolution underwater images in real time.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to one aspect of the present disclosure, a method for real-time enhancement of a high-resolution underwater image is provided, comprising:

[0009] Acquire underwater images in real time;

[0010] Constructing a dual-branch underwater image enhancement network to perform image enhancement on the underwater image, wherein the dual-branch underwater image enhancement network includes an upper branch network and a lower branch network; and

[0011] The upper branch network is guided by the lower branch network to perform image enhancement, and an enhanced low-resolution image of the lower branch network and an enhanced high-resolution image of the upper branch network are obtained respectively.

[0012] In one embodiment of the present disclosure, the dual-branch underwater image enhancement network includes: a convolution module, a sub-pixel convolution module, a multi-scale grouped convolution module, a simplified channel attention module and a feature enhancement guidance module, wherein the number of input channels of the multi-scale grouped convolution module and the simplified channel attention module is equal to the number of output channels.

[0013] In one embodiment of the present disclosure, the step of obtaining the enhanced high-resolution image of the upper branch network includes:

[0014] Inputting the underwater image into a first convolution module for feature extraction to obtain a first feature image;

[0015] Inputting the first feature image into a first sub-pixel convolution module to change the number of channels and perform image downsampling to obtain a second feature image;

[0016] Inputting the second feature image into the first multi-scale grouped convolution module to perform multi-scale feature extraction to obtain a third feature image;

[0017] Inputting the third feature image into the simplified channel attention module to extract high-frequency information in the third feature image and enhance the high-frequency information to obtain a fourth feature image;

[0018] Performing feature enhancement on the fourth feature image by using the feature enhancement guidance module and the second multi-scale grouped convolution module to obtain a fifth feature image;

[0019] Inputting the fifth feature image into the second sub-pixel convolution module to change the number of channels and perform image upsampling to obtain a sixth feature image;

[0020] The sixth feature image is input into the second convolution module to obtain the enhanced high-resolution image.

[0021] In one embodiment of the present disclosure, the step of obtaining the enhanced low-resolution image of the lower branch network includes:

[0022] Downsampling the underwater image to obtain a low-resolution underwater image;

[0023] Inputting the low-resolution underwater image into a third convolution module for feature extraction to obtain a seventh feature image;

[0024] Inputting the seventh feature image into a third sub-pixel convolution module to change the number of channels and image downsampling, thereby obtaining an eighth feature image;

[0025] Inputting the eighth feature image into a third multi-scale grouped convolution module to perform multi-scale feature extraction to obtain a ninth feature image;

[0026] Extracting image feature information from the ninth feature image through the feature enhancement guidance module, so as to guide the upper branch network to perform image enhancement according to the image feature information;

[0027] Inputting the ninth feature into the fourth sub-pixel convolution module, changing the number of channels, and obtaining a tenth feature image;

[0028] The tenth feature image is input into a fourth convolution module to obtain the enhanced low-resolution image.

[0029] In one embodiment of the present disclosure, the method further includes:

[0030] Calculate the loss function by taking the enhanced high-resolution image and the enhanced low-resolution image as target data sets;

[0031] Training the dual-branch underwater image enhancement network according to the loss function to obtain an enhanced high-resolution underwater image in real time through the trained dual-branch underwater image enhancement network;

[0032] In one embodiment of the present disclosure, the step of calculating the loss function by taking the enhanced high-resolution image and the enhanced low-resolution image as target data sets includes:

[0033] Acquire a clear image data set corresponding to the underwater image, wherein the clear image data set corresponds to images in the target data set;

[0034] According to the enhanced high-resolution image and the corresponding clear image, a first loss function of the upper branch network is obtained, and the expression of the first loss function is:

[0035]

[0036] Among them, E l represents the enhanced high-resolution image, y is the clear image, L 1 is the preset regression loss function, represents the first loss function;

[0037] According to the enhanced low-resolution image and the downsampled clear image, a second loss function of the lower branch network is obtained, and the expression of the second loss function is:

[0038]

[0039] Among them, E S represents the enhanced low-resolution image, y s represents the clear image after downsampling, represents the second loss function;

[0040] The loss function is determined according to the first loss function and the second loss function, and the expression of the loss function is:

[0041] In one embodiment of the present disclosure, the multi-scale grouped convolution module has two branches, namely a first branch and a second branch;

[0042] The processing process in the multi-scale grouped convolution module includes:

[0043] Splitting the input target feature image into two groups of feature images with equal numbers of channels, and inputting the two groups of feature images into the first branch and the second branch respectively;

[0044] Extracting features from the first group of feature images through the first branch to obtain a first branch feature image;

[0045] Down-sampling and feature extraction are performed on the second group of feature images through the second branch, and then up-sampling is performed to obtain the second branch feature map;

[0046] The first branch feature map and the second branch feature map are channel-superimposed to obtain a multi-scale feature image.

[0047] In one embodiment of the present disclosure, the third feature image is input into the simplified channel attention module to extract high-frequency information in the third feature image and enhance the high-frequency information to obtain a fourth feature image, including

[0048] Acquiring low-frequency information in the third feature image;

[0049] Subtract the overall information from the low-frequency information to obtain the high-frequency information;

[0050] The high-frequency information is enhanced by the simplified channel attention module, wherein the simplified channel attention module adopts a simplified channel attention CA mechanism, and the function expression of the simplified channel attention mechanism SCA is:

[0051] SC A(f h ) = f h *Wpool(f h )

[0052] Among them, f h represents the high-frequency information, W represents the fully connected layer, and pool represents pooling.

[0053] In one embodiment of the present disclosure, extracting image feature information from the ninth feature image by the feature enhancement guidance module to guide the upper branch network to perform image enhancement according to the image feature information includes:

[0054] In the upper branch network, a feature enhanced image is obtained according to the second feature image and the fourth feature image;

[0055] Performing size compression on the feature enhanced image to obtain a first compressed feature;

[0056] In the lower branch network, the ninth feature image is size-compressed to obtain a second compressed feature;

[0057] Performing convolution processing according to the first compressed feature and the second compressed feature to obtain feature enhancement information;

[0058] The upper branch network is guided to perform image enhancement through the feature enhancement information.

[0059] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0060] Processor; and

[0061] A memory, configured to store executable instructions of the processor;

[0062] The processor is configured to execute any one of the above-mentioned methods for real-time enhancement of high-resolution underwater images by executing the executable instructions.

[0063] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the real-time enhancement method for high-resolution underwater images described above is implemented.

[0064] The embodiment of the present disclosure provides a method for real-time enhancement of high-resolution underwater images. After acquiring the underwater image in real time, a dual-branch underwater image enhancement network is constructed to enhance the underwater image, wherein the dual-branch underwater image enhancement network includes an upper branch network and a lower branch network; the lower branch network guides the upper branch network to perform image enhancement, and obtains the enhanced low-resolution image of the lower branch network and the enhanced high-resolution image of the upper branch network respectively. The present disclosure provides a network model with a dual-branch structure for underwater image enhancement. The upper and lower branches perform feature extraction of different sizes on the same image, and then the lower branch guides the upper branch to complete the overall underwater image enhancement, thereby achieving real-time enhancement of high-resolution underwater images and videos, obtaining good visual effects, and improving enhancement performance.

[0065] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0067] Figure 1 A schematic diagram of a process flow of a high-resolution underwater image real-time enhancement method in an embodiment of the present disclosure is shown;

[0068] Figure 2 A schematic diagram showing the structure of a dual-branch underwater image enhancement network in an embodiment of the present disclosure is shown;

[0069] Figure 3 A schematic diagram showing a specific architecture of a dual-branch underwater image enhancement network in an embodiment of the present disclosure is shown;

[0070] Figure 4 A schematic diagram of a multi-scale grouped convolution module in an embodiment of the present disclosure is shown;

[0071] Figure 5 A schematic diagram showing the flow of another method for real-time enhancement of high-resolution underwater images in an embodiment of the present disclosure is shown;

[0072] Figure 6 A schematic diagram of a process flow of another high-resolution underwater image real-time enhancement method according to an embodiment of the present disclosure is shown;

[0073] Figure 7 A schematic diagram of a conventional channel attention mechanism in an embodiment of the present disclosure is shown;

[0074] Figure 8 A simplified channel attention mechanism schematic diagram is shown in an embodiment of the present disclosure;

[0075] Fig. 9 A schematic diagram of a process flow of another high-resolution underwater image real-time enhancement method in an embodiment of the present disclosure is shown;

[0076] Fig.10 A schematic diagram showing a dynamic convolution kernel module in an embodiment of the present disclosure is shown;

[0077] Fig.11 A structural block diagram of a computer device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0078] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0079] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0080] The solution provided by this application relates to underwater image enhancement technology. For ease of understanding, several terms involved in this application are first explained below.

[0081] Group convolution is used to split the network so that the model can run in parallel on 2 GPUs.

[0082] PixelShuffle is an efficient, fast, non-parameter pixel rearrangement upsampling method, an image upsampling method and sub-pixel convolution algorithm. It can convert low-resolution images into high-resolution images.

[0083] PixUnshuffle is the inverse operation of PixelShuffle, which downsamples the feature map, making the feature map smaller and the number of channels larger.

[0084] CA (Channel Attention) mechanism: is an attention mechanism widely used in deep learning, mainly used in fields such as image processing and natural language processing. It can help the model focus on the important parts of the input data and ignore the unimportant parts, which can effectively improve the performance of the model.

[0085] Convolutional Neural Networks (CNN) is a type of feedforward neural network with a deep structure that includes convolution calculations. It is a deep learning model based on convolution operations and can achieve tasks such as data classification and recognition by automatically learning and extracting data features.

[0086] The solution provided in the embodiment of the present application involves technologies such as image enhancement and channel attention, which are specifically described by the following embodiments:

[0087] In one embodiment, a high-resolution underwater image real-time enhancement method is proposed. Figure 1 The figure shows a flow chart of a method for real-time enhancement of high-resolution underwater images, including:

[0088] S101, real-time acquisition of underwater images;

[0089] Specifically, when exploring underwater, underwater images are acquired by underwater robots, which are usually unclear. Underwater robots usually need to process cameras in real time when performing tasks to ensure timely transmission of information. The current existing models used in underwater machines cannot solve the problem of real-time enhancement of high-resolution images.

[0090] S102, constructing a dual-branch underwater image enhancement network to enhance the underwater image, wherein the dual-branch underwater image enhancement network includes an upper branch network and a lower branch network;

[0091] In order to solve the above problems, this embodiment proposes a network capable of achieving real-time enhancement of high-resolution underwater images. The network adopts a dual-branch structure, namely the dual-branch underwater image enhancement network. Figure 2 Shown is a schematic diagram of the structure of a dual-branch underwater image enhancement network.

[0092] S103, guiding the upper branch network to perform image enhancement through the lower branch network, and obtaining an enhanced low-resolution image of the lower branch network and an enhanced high-resolution image of the upper branch network respectively;

[0093] Combination Figure 2 and Figure 3 As shown in the figure, x is an underwater image. The upper and lower branch networks simultaneously extract features of different sizes of the same underwater image. At the same time, the lower branch guides the upper branch to complete the overall underwater image enhancement. After image restoration, the upper branch network finally outputs the enhanced high-resolution image E. l , the lower branch network outputs the enhanced low-resolution image E s , so as to obtain a better enhanced feature map.

[0094] In this embodiment, a network model with a dual-branch structure is provided for underwater image enhancement. The upper and lower branches extract features of different sizes for the same image, and then the lower branch guides the upper branch to complete the overall underwater image enhancement, thereby achieving real-time enhancement of high-resolution underwater images and videos, obtaining good visual effects, and improving enhancement performance.

[0095] In a specific example, the dual-branch underwater image enhancement network includes: a convolution module, a sub-pixel convolution module, a multi-scale grouped convolution module, a simplified channel attention module and a feature enhancement guidance module, wherein the number of input channels of the multi-scale grouped convolution module and the simplified channel attention module is equal to the number of output channels.

[0096] Specifically, combined Figure 2 The figure shows a schematic diagram of the structure of a dual-branch underwater image enhancement network. The upper branch of the dual-branch underwater image enhancement network 200 includes a convolution module, a multi-scale grouped convolution module, and a simplified channel attention module, and the lower branch includes a convolution module and a multi-scale convolution module. The lower branch of the feature enhancement guidance module guides the enhancement of the upper branch. Optionally, sub-pixel convolution modules are used in both the upper and lower branches.

[0097] In an optional example, the convolution module uses a convolution layer with a convolution kernel of 3 and an output channel number of 8 to perform rough feature extraction. In order to extract more feature maps while increasing the number of feature maps to ensure that the network's reasoning speed is still fast enough, a sub-pixel convolution module is used to simultaneously transform the feature map size and the number of channels of the feature map. Optionally, the sub-pixel convolution module uses PixShuffle and PixUnshuffle strategies. The multi-scale grouped convolution module performs further feature extraction. Since high-frequency information is often ignored in image restoration tasks, a simplified channel attention module is used to enhance and optimize the extraction of high-frequency information during image restoration. The feature enhancement guidance module extracts the feature information of small-size images in the lower branch to guide the enhancement of large-size images in the upper branch.

[0098] In addition, the network is most efficient when the input channels of all convolutional layers in the network are equal to the output channels. In order to maintain the efficiency of the model, the number of input channels of all modules in this embodiment is equal to the number of output channels, specifically the multi-scale grouped convolution module and the high-frequency simplified channel attention module, except for the feature transformation convolution layer and sub-pixel convolution module used for input and output images in the network. The strategy of equal number of channels is adopted to improve the speed of the model and make the model performance better.

[0099] The following is an explanation of the processing of the above modules in the upper and lower branch networks:

[0100] In the upper branch network, the large-size image is finely extracted to obtain a better feature map and finally a good enhancement effect map.

[0101] Reference Figure 5 As shown, in a specific example, the step of obtaining the enhanced high-resolution image of the upper branch network includes:

[0102] S501, inputting the underwater image into a first convolution module for feature extraction to obtain a first feature image;

[0103] Combination Figure 3 As shown, the underwater image x is input into a convolution layer with a convolution kernel of 3 and an output channel number of 8 according to the original size, that is, the first convolution module 201, and the features are extracted to obtain H represents the image height, W represents the image width, C represents the number of channels, and the formula for the first feature image f is:

[0104] f=conv1(x) Formula 1

[0105] Among them, conv1 is the convolution layer of the first convolution module 201, and the number of channels is 8 in an optional implementation manner.

[0106] S502, inputting the first feature image into a first sub-pixel convolution module to change the number of channels and perform image downsampling to obtain a second feature image;

[0107] In order to extract more feature maps to improve the richness of feature information extraction and ensure that the network reasoning speed is still fast enough while increasing the number of feature maps, the first sub-pixel convolution module 202 expands the number of output channels while downsampling the image, using the PixUnshuffle strategy, the formula of which is:

[0108] pixcup L = torch.nn.PixelUnshuffle(r) Formula 2

[0109] f 4c =pixcup L (f) Formula 3

[0110] PixUnshuffle is encapsulated in Pytorch. Formula 2 is its implementation function in Pytorch. The PixUnshuffle function encapsulated in Pytorch is directly called to perform upsampling. In the optional implementation, r is set to 2, such as Figure 3 2↑ in f 4c Represents the second feature image, and sets the number of sub-pixel convolution output channels to r times the number of input channels 2 times, and obtain the features after channel transformation

[0111] S503, inputting the second feature image into a first multi-scale grouping convolution module to perform multi-scale feature extraction to obtain a third feature image;

[0112] Specifically, the second feature image f 4cFurther feature extraction is performed, where the first multi-scale grouped convolution module 203 performs multi-scale feature extraction twice.

[0113] In a specific example, referring to Figure 4 A schematic diagram of a multi-scale grouped convolution module is shown, wherein the multi-scale grouped convolution module has two branches, namely a first branch and a second branch;

[0114] Specifically, the first branch is 401 and the second branch is 402 .

[0115] Reference Figure 6 As shown, the processing process in the multi-scale grouped convolution module includes:

[0116] S601, splitting the input target feature image into two groups of feature images with equal numbers of channels, and inputting the two groups of feature images into the first branch and the second branch respectively;

[0117] Specifically, taking the above process as an example, the feature f after convolution in the first sub-pixel convolution module 202 is 4c , split the channel into two branches with equal number of channels f 1 and f 2 , the feature map is divided into two halves, the first branch is The second branch is

[0118] S602, extracting features from a first group of feature images through the first branch to obtain a first branch feature image;

[0119] Specifically, half of the feature map f 1 The feature information of the original size image is further extracted through the first branch to obtain In the first branch 401, activation function, convolution and activation function are sequentially performed to obtain the first branch feature image, which is expressed as:

[0120] f′ 1 =conv Gtop2 (δ(conv Gtop1 (f 1 ))) Formula 4

[0121] Among them, conv Gtop1 and conv Gtop2 It is a convolutional layer with 3 convolution kernels and 2C input and output channels. δ(·) represents the ReLU activation function.

[0122] S603, downsampling and feature extraction are performed on the second group of feature images through the second branch, and then upsampling is performed to obtain the second branch feature map;

[0123] The other half of the feature map f 2 The second branch 402 performs scale transformation and feature information extraction, and sequentially performs strided convolution, convolution, and upsampling in the second branch 402. First, downsampling is performed to extract features, and then upsampling is performed to obtain the same information as f. 2 Features of the same size The second branch feature image is obtained, and its expression is:

[0124] f 2 ′=(s(conv Gbottom1 (f 2 )) 2↑ Formula 5

[0125] Formula (5)conv Gbottom1 It is a strided convolution with 3 convolution kernels, 2 input and output channels, and a stride of 2. This paper uses strided convolution to implement downsampling operations. s(·) represents the LeakyRelu activation function. 2↑ represents 2x upsampling, which is achieved through interpolation upsampling.

[0126] S604: Channel-superimpose the first branch feature map and the second branch feature map to obtain a multi-scale feature image.

[0127] After various convolution sampling for multi-scale feature extraction, the extracted information needs to be transformed into a channel to obtain features with the same number of channels as the feature map before channel splitting. Channel superposition to obtain a multi-scale feature image, the formula is as follows:

[0128]

[0129] in, The concat layer is a concat operation, which is mostly used to utilize the semantic information of feature maps of different scales and increase the channels to achieve better performance.

[0130] The first multi-scale grouped convolution module 203 performs two multi-scale feature extractions and then obtains the feature f′ e It can be considered as the third feature image in the above step S503.

[0131] S504, inputting the third feature image into the simplified channel attention module to extract high-frequency information in the third feature image and enhance the high-frequency information to obtain a fourth feature image;

[0132] Specifically, considering that high-frequency information is often ignored in image restoration tasks, the simplified channel attention module 204 is used to enhance high-frequency features to optimize the extraction of high-frequency information during image restoration.

[0133] In a specific example, the third feature image is input into the simplified channel attention module to extract high-frequency information in the third feature image and enhance the high-frequency information to obtain a fourth feature image, including

[0134] Acquiring low-frequency information in the third feature image;

[0135] Specifically, first, e ′Operate to obtain f e ′ low-frequency information The specific operations are as follows:

[0136] f l =mean(f e ′) Formula 7

[0137] Wherein, mean(·) is a function for obtaining low-frequency information, and mean blurring is optionally used to obtain low-frequency information.

[0138] Subtract the overall information from the low-frequency information to obtain the high-frequency information;

[0139] Specifically, after obtaining the low-frequency information, the overall information is subtracted from the low-frequency information to obtain the high-frequency information. The formula expression is as follows:

[0140] f h =1-f l Formula 8

[0141] The high-frequency information is enhanced by the simplified channel attention module, wherein the simplified channel attention module adopts a simplified channel attention CA mechanism, and the function expression of the simplified channel attention mechanism SCA is:

[0142] SCA(f h ) = f h *Wpool(f h ) Formula 9

[0143] Among them, f h represents the high-frequency information, W represents the fully connected layer, and pool represents pooling.

[0144] In obtaining high frequency information f h Finally, the channel attention mechanism is combined to focus on high-frequency information f h To enhance, the formula is as follows:

[0145] f′ h =SCA(f h ) Formula 10

[0146] Reference Figure 7 A schematic diagram of a conventional channel attention mechanism and Figure 8 The simplified channel attention mechanism diagram shown explains how to obtain the simplified channel attention mechanism.

[0147] The expression of the conventional channel attention mechanism is:

[0148] CA(X)=X*σ(W 2 max(0,W 1 pool(X))) Formula 11

[0149] Where X represents the feature map, pool represents the global average pooling operation that aggregates spatial information into channels. σ is a nonlinear activation function Sigmoid, W 1 , W 2 is a fully connected layer, and ReLU is used between two fully connected layers. Finally, * is a channelization operation.

[0150] By retaining the two most important functions of channel attention, namely aggregating global information and channel information interaction, a simplified channel attention mechanism is obtained, which is expressed as:

[0151] SCA(X)=X*Wpool(X) Formula 12 By comparison, it can be seen that the simplified channel attention mechanism is simpler than the conventional channel attention mechanism. Therefore, the simplified channel attention module 204 uses the simplified channel attention mechanism to enhance the high-frequency information.

[0152] S505, performing feature enhancement on the fourth feature image by using the feature enhancement guidance module and the second multi-scale group convolution module to obtain a fifth feature image;

[0153] Specifically, after the enhancement of the high-frequency information is completed in the above step S505, a feature enhancement guidance module 205 and a second multi-scale group convolution module 206 are continuously used to continuously transform and enhance the features to obtain a fifth feature image

[0154] f add =f h ′+f 4c Formula 13

[0155] For the enhanced fourth feature image With the second feature image f 4c Perform a shortcut to get To enhance the fourth feature image f h ′ Features related to the original image to improve the effectiveness of information.

[0156] After passing through the second multi-scale group convolution module 206, the fifth feature image f′ is obtained 4c , whose expression is:

[0157] f′ 4c =MGB(FEGB(f add ,f s ″))) Formula 14

[0158] is the ninth image feature obtained by a series of operations in the lower branch, FEGB represents the expression of the feature enhancement guidance module 205. MGB represents the expression corresponding to the multi-scale grouped convolution module processing, which can be obtained according to the above Figure 6 The embodiment shown is known. Formula 14 will be specifically explained in conjunction with the following lower branch network.

[0159] S506, inputting the fifth feature image into a second sub-pixel convolution module to change the number of channels and perform image upsampling to obtain a sixth feature image;

[0160] The channels are transformed again by the second sub-pixel convolution module 207 to obtain a sixth feature image, and the operation expression of the second sub-pixel convolution module 207 is:

[0161] pixcdown L =torch.nn.PixelShuffle(r) Formula 15

[0162] Here, PixelShuffle is used in the second sub-pixel convolution module 207 for upsampling, r is the desired upsampling multiple, r is optionally set to 2, and the number of input channels is r times the number of output channels. 2 times.

[0163] S507: Input the sixth feature image into a second convolution module to obtain the enhanced high-resolution image.

[0164] Specifically, the enhanced image E is obtained through the second convolution module 208, which is a convolution layer with a convolution kernel of 3 and an output channel number of 3. l , the formula is as follows:

[0165] E l =conv2(pixcdown L (f′ 4c )) Formula 16

[0166] Among them, conv2 is the convolution layer of the second convolution module 208.

[0167] In the lower branch network, the small-size image is roughly extracted, and then the extracted information is used to guide the upper branch to obtain a better feature map, and finally a good enhanced effect map is obtained.

[0168] Reference Fig. 9 As shown, in a specific example, the step of obtaining the enhanced low-resolution image of the lower branch network includes:

[0169] S901, downsampling the underwater image to obtain a low-resolution underwater image;

[0170] Specifically, the downsampling module 209 first performs Interpolate and downsample to get a low-resolution image x s .

[0171] S902, inputting the low-resolution underwater image into a third convolution module for feature extraction to obtain a seventh feature image;

[0172] x s Input a convolution layer conv3 with 3 convolution kernels and 8 output channels to extract features. Optionally, C is set to 8, and the seventh feature image f s The expression is as follows:

[0173] f s =conv3(x s ) Formula 17

[0174] Among them, the third convolution module 210 corresponds to the convolution layer conv3.

[0175] S903, inputting the seventh feature image into a third sub-pixel convolution module to change the number of channels and image downsampling, to obtain an eighth feature image;

[0176] Similar to the upper branch, the third sub-pixel convolution module 211 is used to perform channel expansion and size reduction, and the formula is expressed as follows:

[0177] pixcup S = torch.nn.PIxelUnshuffle(r) Formula 18

[0178] f s ′=pixcup S (f s ) Formula 19

[0179] Among them, r is set to 2, setting the number of sub-pixel convolution output channels to r times the number of input channels 2 times, and obtain the eighth image feature after channel transformation

[0180] S904, inputting the eighth feature image into a third multi-scale grouping convolution module to perform multi-scale feature extraction to obtain a ninth feature image;

[0181] Then, the features are further extracted through i MGBs to obtain the ninth feature image f s ″, the expression is as follows:

[0182] f s =MGB i (f s ′) Formula 20

[0183] Among them, MGB i (·) is the operation of the multi-scale group convolution module. The subscript i is set to 6 in this embodiment. The third multi-scale convolution module performs 6 multi-scale convolutions, the multi-scale convolution module 212 performs 3 times, and the multi-scale convolution module 213 performs 3 times.

[0184] S905, extracting image feature information from the ninth feature image through the feature enhancement guidance module, so as to guide the upper branch network to perform image enhancement according to the image feature information;

[0185] Specifically, the feature information of the small-size image of the lower branch is extracted to guide the enhancement of the large-size image of the upper branch.

[0186] S906, inputting the ninth feature into a fourth sub-pixel convolution module, changing the number of channels, and obtaining a tenth feature image;

[0187] The enhanced image E is obtained by transforming the channel through the fourth sub-pixel convolution module 214. s , the formula is as follows:

[0188] pixcdown S =torch.nn.PixelShuffle(r) Formula 21

[0189] Among them, the PixelShuffle strategy is used for upsampling, r is set to 2, and the number of input channels is r times the number of output channels. 2 times.

[0190] S907, input the tenth feature image into a fourth convolution module to obtain the enhanced low-resolution image.

[0191] Specifically, the convolution layer conv4 corresponding to the fourth convolution module 215 performs convolution, and obtains the enhanced low-resolution image E through a convolution layer conv4 with a convolution kernel of 3 and an output channel number of 3. s , whose expression is:

[0192] Es =conv4(pixcdown S (f s ″)) Formula 22

[0193] In a specific example, the method further includes:

[0194] Calculate the loss function by taking the enhanced high-resolution image and the enhanced low-resolution image as target data sets;

[0195] Among them, the target data set includes the enhanced high-resolution images obtained by the upper branch and the enhanced low-resolution images obtained by the lower branch. The enhanced images in the target data set and the clear images are used as supervision to calculate the loss function, which can effectively realize update training for the dual-branch underwater image enhancement network.

[0196] The dual-branch underwater image enhancement network is trained according to the loss function to obtain enhanced high-resolution underwater images in real time through the trained dual-branch underwater image enhancement network.

[0197] In this example, the dual-branch underwater image enhancement network trained with the loss function can achieve better underwater image enhancement effects. The upper and lower branch networks extract different sizes of the same image, and the low-resolution image features obtained by the lower branch guide the restoration of the high-resolution image of the upper branch. While obtaining high-resolution enhanced images in real time, better image enhancement effects are further obtained with better stability.

[0198] In a specific example, the step of calculating the loss function by taking the enhanced high-resolution image and the enhanced low-resolution image as target data sets includes:

[0199] Acquire a clear image data set corresponding to the underwater image, wherein the clear image data set corresponds to images in the target data set;

[0200] According to the enhanced high-resolution image and the corresponding clear image, a first loss function of the upper branch network is obtained, and the expression of the first loss function is:

[0201] The loss function consists of two parts, which come from the upper branch network supervision loss and the lower branch supervision loss. The upper branch network loss function, the first loss function expression is as follows:

[0202]

[0203] Among them, E l represents the enhanced high-resolution image, y is the clear image, L 1 is the preset regression loss function, represents the first loss function;

[0204] According to the enhanced low-resolution image and the downsampled clear image, a second loss function of the lower branch network is obtained, and the expression of the second loss function is:

[0205]

[0206] Among them, E S represents the enhanced low-resolution image, y s represents the clear image after downsampling, represents the second loss function;

[0207] in, To perform Interpolated downsampled sharp image.

[0208] The loss function is determined according to the first loss function and the second loss function, and the expression of the loss function is:

[0209]

[0210] In a specific example, extracting the image feature information in the ninth feature image by the feature enhancement guidance module to guide the upper branch network to perform image enhancement according to the image feature information includes:

[0211] In the upper branch network, a feature enhanced image is obtained according to the second feature image and the fourth feature image;

[0212] For the enhanced fourth feature image With the second feature image f 4c Perform shortcut to obtain feature enhanced image To enhance f′ h Features related to the original image improve the effectiveness of information. The above formula 13 gives f add .

[0213] Performing size compression on the feature enhanced image to obtain a first compressed feature;

[0214] f add After sub-pixel convolution, the size is compressed and the first compression feature formula is expressed as follows:

[0215] pix down2 = torch.nn.PixelUnshuffle(r) Formula 26

[0216] L=pix down2 (f add) Formula 27

[0217] Where r is set to 2, and Formula 26 is the first compression feature obtained after size compression

[0218] In the lower branch network, the ninth feature image is size-compressed to obtain a second compressed feature;

[0219] The ninth feature image feature f after the lower branch is extracted by 6 MGBs s The size is compressed by sub-pixel convolution, and the second compression feature formula is expressed as follows:

[0220] pix down1 = torch.nn.PixelUnshuffle(r) Formula 28

[0221] S=Pix down1 (f s ″) Formula 29

[0222] Among them, r is set to 4, and the feature after size compression is the second compressed feature Then pass through a dynamic convolution kernel module (DCKG) to get the convolution kernel sequence As the action guide for the upper branch image restoration. Fig.10 Schematic diagram of the dynamic convolution kernel module, in which average pooling, 3×3 convolution, maximum pooling, 3×3 convolution, adaptive pooling and 1×1 convolution are performed in sequence.

[0223] Performing convolution processing according to the first compressed feature and the second compressed feature to obtain feature enhancement information;

[0224] Convolution is performed to obtain The FEGB expression corresponding to the feature enhancement information guidance module is:

[0225] f g =(DCKG(S)*L)↑ 4 Formula 30

[0226] The upper branch network is guided to perform image enhancement through the feature enhancement information.

[0227] Among them, the feature enhancement information can be considered as the above-mentioned FEGB expression.

[0228] In another embodiment, an electronic device is provided, comprising:

[0229] Processor; and

[0230] A memory, configured to store executable instructions of the processor;

[0231] The processor is configured to execute any one of the above-mentioned methods for real-time enhancement of high-resolution underwater images by executing the executable instructions.

[0232] This embodiment provides an electronic device to implement the above-mentioned high-resolution underwater image real-time enhancement method. Please refer to the method embodiment for details, which will not be repeated here.

[0233] In yet another embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the real-time enhancement method for high-resolution underwater images described above is implemented.

[0234] In this embodiment, a computer-readable storage medium is provided to implement the above-mentioned high-resolution underwater image real-time enhancement method. Please refer to the method embodiment for details, which will not be repeated here.

[0235] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0236] Refer to the following Fig.11 The electronic device 1100 according to this embodiment of the present invention is described. Fig.11 The electronic device 1100 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0237] like Fig.11 As shown, the electronic device 1100 is in the form of a general computing device. The components of the electronic device 1100 may include but are not limited to: at least one processing unit 1110, at least one storage unit 1120, and a bus 1130 connecting different system components (including the storage unit 1120 and the processing unit 1110).

[0238] The storage unit stores program codes, which can be executed by the processing unit 1110, so that the processing unit 1110 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 1110 can perform the following steps: Figure 1 The real-time enhancement method for high-resolution underwater images shown in.

[0239] The storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 11201 and / or a cache storage unit 11202 , and may further include a read-only storage unit (ROM) 11203 .

[0240] The storage unit 1120 may also include a program / utility 11204 having a set (at least one) of program modules 11205, such program modules 11205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0241] Bus 1130 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0242] The electronic device 1100 may also communicate with one or more external devices 1200 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1150. Furthermore, the electronic device 1100 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1160. As shown, the network adapter 1160 communicates with other modules of the electronic device 1100 via a bus 1130. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0243] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0244] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0245] A program product for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0246] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0247] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0248] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0249] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0250] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0251] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0252] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0253] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A method for real-time enhancement of high-resolution underwater images, characterized in that: include: Acquire underwater images in real time; Constructing a dual-branch underwater image enhancement network to perform image enhancement on the underwater image, wherein the dual-branch underwater image enhancement network includes an upper branch network and a lower branch network; and The upper branch network is guided by the lower branch network to perform image enhancement, so as to obtain an enhanced low-resolution image of the lower branch network and an enhanced high-resolution image of the upper branch network respectively; The dual-branch underwater image enhancement network includes: a convolution module, a sub-pixel convolution module, a multi-scale grouped convolution module, a simplified channel attention module and a feature enhancement guidance module, wherein the number of input channels of the multi-scale grouped convolution module and the simplified channel attention module is equal to the number of output channels; Among them, the upper branch network includes the convolution module, the sub-pixel convolution module, the multi-scale grouped convolution module and the simplified channel attention module; the lower branch network includes the convolution module, the sub-pixel convolution module and the multi-scale grouped convolution module.

2. The method for real-time enhancement of high-resolution underwater images according to claim 1, characterized in that: The step of obtaining the enhanced high-resolution image of the upper branch network comprises: Inputting the underwater image into a first convolution module for feature extraction to obtain a first feature image; Inputting the first feature image into a first sub-pixel convolution module to change the number of channels and perform image downsampling to obtain a second feature image; Inputting the second feature image into the first multi-scale grouped convolution module to perform multi-scale feature extraction to obtain a third feature image; Inputting the third feature image into the simplified channel attention module to extract high-frequency information in the third feature image and enhance the high-frequency information to obtain a fourth feature image; Performing feature enhancement on the fourth feature image by using the feature enhancement guidance module and the second multi-scale grouped convolution module to obtain a fifth feature image; Inputting the fifth feature image into the second sub-pixel convolution module to change the number of channels and perform image upsampling to obtain a sixth feature image; The sixth feature image is input into the second convolution module to obtain the enhanced high-resolution image.

3. The method for real-time enhancement of high-resolution underwater images according to claim 2, characterized in that: The step of obtaining the enhanced low-resolution image of the lower branch network comprises: Downsampling the underwater image to obtain a low-resolution underwater image; Inputting the low-resolution underwater image into a third convolution module for feature extraction to obtain a seventh feature image; Inputting the seventh feature image into a third sub-pixel convolution module to change the number of channels and image downsampling to obtain an eighth feature image; Inputting the eighth feature image into a third multi-scale grouped convolution module for multi-scale feature extraction to obtain a ninth feature image; Extracting image feature information from the ninth feature image through the feature enhancement guidance module, so as to guide the upper branch network to perform image enhancement according to the image feature information; Inputting the ninth feature into the fourth sub-pixel convolution module, changing the number of channels, and obtaining a tenth feature image; The tenth feature image is input into a fourth convolution module to obtain the enhanced low-resolution image.

4. The method for real-time enhancement of high-resolution underwater images according to claim 1, characterized in that: The method further comprises: Calculate the loss function by taking the enhanced high-resolution image and the enhanced low-resolution image as target data sets; The dual-branch underwater image enhancement network is trained according to the loss function to obtain enhanced high-resolution underwater images in real time through the trained dual-branch underwater image enhancement network.

5. The method for real-time enhancement of high-resolution underwater images according to claim 4, characterized in that: The step of calculating the loss function by taking the enhanced high-resolution image and the enhanced low-resolution image as target data sets includes: Acquire a clear image data set corresponding to the underwater image, wherein the clear image data set corresponds to images in the target data set; According to the enhanced high-resolution image and the corresponding clear image, a first loss function of the upper branch network is obtained, and the expression of the first loss function is: Among them, E l represents the enhanced high-resolution image, y is the clear image, L1 is the preset regression loss function, represents the first loss function; According to the enhanced low-resolution image and the downsampled clear image, a second loss function of the lower branch network is obtained, and the expression of the second loss function is: Among them, E S represents the enhanced low-resolution image, y s represents the clear image after downsampling, represents the second loss function; The loss function is determined according to the first loss function and the second loss function, and the expression of the loss function is:

6. The method for real-time enhancement of high-resolution underwater images according to claim 1, characterized in that: The multi-scale grouped convolution module has two branches, namely a first branch and a second branch; The processing process in the multi-scale grouped convolution module includes: Splitting the input target feature image into two groups of feature images with equal numbers of channels, and inputting the two groups of feature images into the first branch and the second branch respectively; Extracting features from the first group of feature images through the first branch to obtain a first branch feature image; Down-sampling and feature extraction are performed on the second group of feature images through the second branch, and then up-sampling is performed to obtain the second branch feature map; The first branch feature map and the second branch feature map are channel-superimposed to obtain a multi-scale feature image.

7. The method for real-time enhancement of high-resolution underwater images according to claim 2, characterized in that: The step of inputting the third feature image into the simplified channel attention module to extract high-frequency information in the third feature image and enhancing the high-frequency information to obtain a fourth feature image comprises: Acquiring low-frequency information in the third feature image; Subtract the overall information from the low-frequency information to obtain the high-frequency information; The high-frequency information is enhanced by the simplified channel attention module, wherein the simplified channel attention module adopts a simplified channel attention CA mechanism, and the function expression of the simplified channel attention mechanism SCA is: SCA(f h )=f h *Wpool(f h ) Among them, f h represents the high-frequency information, W represents the fully connected layer, and pool represents pooling.

8. The method for real-time enhancement of high-resolution underwater images according to claim 3, characterized in that: The extracting the image feature information in the ninth feature image by the feature enhancement guidance module to guide the upper branch network to perform image enhancement according to the image feature information includes: In the upper branch network, a feature enhanced image is obtained according to the second feature image and the fourth feature image; Performing size compression on the feature enhanced image to obtain a first compressed feature; In the lower branch network, the ninth feature image is size-compressed to obtain a second compressed feature; Performing convolution processing according to the first compressed feature and the second compressed feature to obtain feature enhancement information; The upper branch network is guided to perform image enhancement through the feature enhancement information.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the high-resolution underwater image real-time enhancement method according to any one of claims 1 to 8 by executing the executable instructions.

Images