Image processing method, device, equipment and readable storage medium

Abstract

The application discloses an image processing method and device, equipment and a readable storage medium, and relates to the technical field of image processing, so as to improve the image enhancement effect. The method comprises the following steps: acquiring an image to be processed; and processing the image to be processed through a cross-domain dense connection neural network to obtain a target image. The embodiment of the application can improve the image enhancement effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, device, and readable storage medium. Background Technology

[0002] Deep learning-based image enhancement techniques can generate results that are close to the ground truth, but compared with real images, images generated in this way lack high-frequency details.

[0003] Image enhancement techniques can include spatial domain image enhancement and wavelet domain image enhancement. Spatial domain image enhancement extracts features from the entire image without dividing it into frequency bands. Wavelet domain image enhancement decomposes an image into different low-frequency sub-bands and high-frequency wavelet sub-bands. The high-frequency sub-bands can preserve the texture details of an image, while the low-frequency sub-bands preserve the basic structure of objects in the image. Therefore, many existing techniques use the wavelet domain for feature extraction.

[0004] However, the texture details of images generated using existing techniques still differ significantly from those of real images. Therefore, a method to improve the image enhancement effect is needed. Summary of the Invention

[0005] This application provides an image processing method, apparatus, device, and readable storage medium to improve image enhancement effects.

[0006] In a first aspect, embodiments of this application provide an image processing method, including:

[0007] Obtain the image to be processed;

[0008] The image to be processed is processed through a cross-domain densely connected neural network to obtain the target image;

[0009] The cross-domain dense connection neural network includes n sub-networks, where n is an integer and n≥1;

[0010] The sub-network includes at least one spatial domain processing module and at least one wavelet domain processing module. Dense connections are formed between the spatial domain processing module and the corresponding wavelet domain processing module. Dense connections are formed within the spatial domain processing module and within the wavelet domain processing module.

[0011] The spatial domain processing module performs spatial domain processing on the first input image to obtain a first output image, wherein the resolution of the first output image is greater than the resolution of the first input image, and the first input image is the image to be processed or the output image of the previous sub-network.

[0012] The wavelet domain processing module performs wavelet domain processing on the first output image and the cascade information of the spatial domain processing module to obtain a second output image, wherein the second output image is either the target image or the input image of the spatial domain processing module of the next sub-network.

[0013] The spatial domain processing module performs spatial domain processing on the first input image to obtain a first output image, including:

[0014] The spatial domain processing module upsamples the first input image to obtain the first output image, the resolution of which is four times that of the first input image.

[0015] The spatial domain processing module includes one or more nested residual dense blocks, each nested residual dense block including one or more residual dense blocks; the wavelet domain processing module includes one or more nested residual dense blocks, each nested residual dense block including one or more residual dense blocks.

[0016] The y-th residual dense module of the x-th nested residual dense block of the spatial domain processing module and the y-th residual dense module of the x-th nested residual dense block of the wavelet domain processing module are cascaded.

[0017] Where x and y are integers, 1≤x≤P, 1≤y≤Q; P is an integer greater than 0, representing the number of nested residual dense blocks included in the spatial domain processing module or the wavelet domain processing module; Q is an integer greater than 0, representing the number of residual dense blocks included in a nested residual dense block of the spatial domain processing module or the wavelet domain processing module.

[0018] The wavelet domain processing module performs wavelet domain processing on the cascaded information of the first output image and the spatial domain processing module to obtain a second output image, including:

[0019] The wavelet domain processing module performs wavelet decomposition on the first output image to obtain multiple first wavelet sub-bands;

[0020] Feature extraction is performed on the plurality of first wavelet sub-bands and the cascaded information to obtain a plurality of wavelet domain sub-bands;

[0021] The second output image is obtained by performing inverse wavelet transform on the multiple wavelet domain subbands.

[0022] For the nth subnetwork, the method further includes:

[0023] Calculate the first loss function;

[0024] Calculate the second loss function;

[0025] Using the first loss function and the second loss function, the third loss function of the cross-domain densely connected neural network is calculated.

[0026] The calculation of the first loss function includes:

[0027] Calculate the Manhattan distance or Euclidean distance between the target image and the real image in the spatial domain;

[0028] The Manhattan distance or Euclidean distance is used as the first loss function.

[0029] The step of calculating the Manhattan distance or Euclidean distance between the target image and the real image in the spatial domain includes:

[0030] The target image is processed by the Visual Image Generator (VGG) network to obtain the first intermediate target image;

[0031] The spatial domain real image is processed through a VGG network to obtain a first intermediate spatial domain real image.

[0032] Calculate the Manhattan distance or Euclidean distance between the first intermediate target image and the first intermediate spatial domain real image.

[0033] The calculation of the first loss function includes:

[0034] Based on the result of processing the target image and the real spatial domain image by the discriminator of a Generative Adversarial Network (GAN), the first loss function is obtained according to the following formula:

[0035] L adv,G1 = -log(1-D Ra (D,I GT ))-log(D Ra (I GT ,D));

[0036] Among them, L adv,G1 Let D represent the first loss function. Ra Let D represent the discriminator of the GAN, and let I represent the target image. GT This represents the true image of the spatial domain.

[0037] The calculation of the second loss function includes:

[0038] The spatial domain real image is decomposed into multiple second wavelet sub-bands, and the multiple second wavelet sub-bands correspond to multiple wavelet domain sub-bands;

[0039] For any first target wavelet sub-band among multiple second wavelet sub-bands, calculate the Manhattan distance or Euclidean distance between the first target wavelet sub-band and the corresponding wavelet domain sub-band;

[0040] The Manhattan distance or Euclidean distance is used as the loss function for the first target wavelet subband;

[0041] The second loss function is obtained based on multiple loss functions.

[0042] The step of calculating the Manhattan distance or Euclidean distance between the first target wavelet sub-band and the corresponding wavelet domain sub-band includes:

[0043] The first target wavelet subband is processed through a VGG network to obtain the first intermediate target wavelet subband.

[0044] The wavelet domain sub-band corresponding to the first target wavelet sub-band is processed by the VGG network to obtain the first intermediate wavelet domain sub-band.

[0045] Calculate the Manhattan distance or Euclidean distance between the first intermediate target wavelet sub-band and the first intermediate wavelet domain sub-band.

[0046] The calculation of the second loss function includes:

[0047] The spatial domain real image is decomposed into multiple second wavelet sub-bands, and the multiple second wavelet sub-bands correspond to multiple wavelet domain sub-bands;

[0048] For any first target wavelet subband among multiple second wavelet subbands, the first target wavelet subband and its corresponding wavelet domain subband are processed by the discriminator of the GAN network, and the loss function of the first target wavelet subband is obtained according to the following formula:

[0049] L adv,G2 = -log(1-D Ra (I GT ,F(I SR )))-log(D Ra (F(I SR ),I GT ))

[0050] Among them, L adv,G2 Let D represent the second loss function. Ra I represents the discriminator of a GAN. GT F(I) represents the first target wavelet subband. SR () represents the wavelet domain sub-band corresponding to the first target wavelet sub-band;

[0051] The second loss function is obtained based on the multiple loss functions.

[0052] The step of calculating the third loss function of the cross-domain densely connected neural network using the first loss function and the second loss function includes:

[0053] The third loss function is obtained by combining the product of the first loss function and the first coefficient with the product of the second loss function and the second coefficient.

[0054] Secondly, embodiments of this application also provide an image processing apparatus, comprising:

[0055] The first acquisition module is used to acquire the image to be processed;

[0056] The first processing module is used to process the image to be processed through a cross-domain dense connection neural network to obtain the target image;

[0057] The cross-domain dense connection neural network includes n sub-networks, where n is an integer and n≥1;

[0058] The sub-network includes at least one spatial domain processing module and at least one wavelet domain processing module. Dense connections are formed between the spatial domain processing module and the corresponding wavelet domain processing module. Dense connections are formed within the spatial domain processing module and within the wavelet domain processing module.

[0059] The spatial domain processing module performs spatial domain processing on the first input image to obtain a first output image, wherein the resolution of the first output image is greater than the resolution of the first input image, and the first input image is the image to be processed or the output image of the previous sub-network.

[0060] The wavelet domain processing module performs wavelet domain processing on the first output image and the cascade information of the spatial domain processing module to obtain a second output image, wherein the second output image is either the target image or the input image of the spatial domain processing module of the next sub-network.

[0061] The spatial domain processing module upsamples the first input image to obtain the first output image, and the resolution of the first output image is four times that of the first input image.

[0062] The spatial domain processing module includes one or more nested residual dense blocks, each nested residual dense block including one or more residual dense blocks; the wavelet domain processing module includes one or more nested residual dense blocks, each nested residual dense block including one or more residual dense blocks.

[0063] The y-th residual dense module of the x-th nested residual dense block of the spatial domain processing module and the y-th residual dense module of the x-th nested residual dense block of the wavelet domain processing module are cascaded.

[0064] Where x and y are integers, 1≤x≤P, 1≤y≤Q; P is an integer greater than 0, representing the number of nested residual dense blocks included in the spatial domain processing module or the wavelet domain processing module; Q is an integer greater than 0, representing the number of residual dense blocks included in a nested residual dense block of the spatial domain processing module or the wavelet domain processing module.

[0065] The wavelet domain processing module is used for:

[0066] Wavelet decomposition is performed on the first output image to obtain multiple first wavelet sub-bands;

[0067] Feature extraction is performed on the plurality of first wavelet sub-bands and the cascaded information to obtain a plurality of wavelet domain sub-bands;

[0068] The second output image is obtained by performing inverse wavelet transform on the multiple wavelet domain subbands.

[0069] The device further includes:

[0070] The first calculation module is used to calculate the first loss function for the nth sub-network;

[0071] The second calculation module is used to calculate the second loss function;

[0072] The third calculation module is used to calculate the third loss function of the cross-domain densely connected neural network using the first loss function and the second loss function.

[0073] The first calculation module includes:

[0074] The first calculation submodule is used to calculate the Manhattan distance or Euclidean distance between the target image and the spatial domain real image;

[0075] The second calculation submodule is used to use the Manhattan distance or Euclidean distance as the first loss function.

[0076] The first calculation submodule includes:

[0077] The first acquisition unit is used to process the target image through a VGG network to obtain a first intermediate target image;

[0078] The second acquisition unit is used to process the spatial domain real image through a VGG network to obtain a first intermediate spatial domain real image.

[0079] The first calculation unit is used to calculate the Manhattan distance or Euclidean distance between the first intermediate target image and the first intermediate spatial domain real image.

[0080] The first calculation module is used to obtain the first loss function based on the result of processing the target image and the spatial domain real image by the discriminator of a Generative Adversarial Network (GAN) according to the following formula:

[0081] L adv,G1 = -log(1-D Ra (D,I GT ))-log(D Ra (I GT ,D));

[0082] Among them, L adv,G1 Let D represent the first loss function. Ra Let D represent the discriminator of the GAN, and let I represent the target image. GT This represents the true image of the spatial domain.

[0083] The second calculation module includes:

[0084] The first decomposition submodule is used to decompose the spatial domain real image into multiple second wavelet sub-bands, wherein the multiple second wavelet sub-bands correspond to multiple wavelet domain sub-bands.

[0085] The first calculation submodule is used to calculate the Manhattan distance or Euclidean distance between the first target wavelet subband and the corresponding wavelet domain subband for any first target wavelet subband among a plurality of second wavelet subbands;

[0086] The first acquisition submodule is used to use the Manhattan distance or Euclidean distance as the loss function of the first target wavelet subband;

[0087] The second acquisition submodule is used to obtain the second loss function based on multiple loss functions.

[0088] The first calculation submodule includes:

[0089] The first processing unit is used to process the first target wavelet subband through a VGG network to obtain the first intermediate target wavelet subband.

[0090] The second processing unit is used to process the wavelet domain sub-band corresponding to the first target wavelet sub-band through the VGG network to obtain the first intermediate wavelet domain sub-band.

[0091] The first calculation unit is used to calculate the Manhattan distance or Euclidean distance between the first intermediate target wavelet sub-band and the first intermediate wavelet domain sub-band.

[0092] The second calculation module includes:

[0093] The second decomposition submodule is used to decompose the spatial domain real image into multiple second wavelet sub-bands, and the multiple second wavelet sub-bands correspond to multiple wavelet domain sub-bands.

[0094] The second calculation submodule is used to process any first target wavelet subband among multiple second wavelet subbands, passing the first target wavelet subband and its corresponding wavelet domain subband through the discriminator of the GAN network, and obtaining the loss function of the first target wavelet subband according to the following formula:

[0095] L adv,G2 = -log(1-D Ra (I GT ,F(I SR )))-log(D Ra (F(I SR ),I GT ))

[0096] Among them, L adv,G2 Let D represent the second loss function. Ra I represents the discriminator of a GAN. GT F(I) represents the first target wavelet subband. SR () represents the wavelet domain sub-band corresponding to the first target wavelet sub-band;

[0097] The third acquisition submodule is used to obtain the second loss function based on the multiple loss functions.

[0098] The third calculation module is used to use the sum of the product of the first loss function and the first coefficient, and the product of the second loss function and the second coefficient, as the third loss function.

[0099] Thirdly, embodiments of this application also provide an electronic device, including: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the image processing method described above.

[0100] Fourthly, embodiments of this application also provide a readable storage medium storing a program that, when executed by a processor, implements the steps in the image processing method described above.

[0101] In this embodiment, a cross-domain densely connected neural network is used to process the image to be processed to obtain the target image. Since the sub-networks of the cross-domain densely connected neural network include at least one spatial domain processing module and at least one wavelet domain processing module, and wavelet domain processing is performed based on the spatial domain processing module, the features of the spatial domain can be superimposed onto the wavelet domain, thereby improving the feature diversity of the wavelet domain. Therefore, the solution using this embodiment can improve the image enhancement effect. Attached Figure Description

[0102] Figure 1 This is one of the flowcharts of the image processing method provided in the embodiments of this application;

[0103] Figure 2(a) is a schematic diagram of a cross-domain densely connected neural network provided in an embodiment of this application;

[0104] Figure 2(b) is a schematic diagram of dense connections;

[0105] Figure 3 This is the second flowchart of the image processing method provided in the embodiments of this application;

[0106] Figure 4 This is a schematic diagram illustrating multi-level (multi-scale) image enhancement using the cross-domain densely connected neural network proposed in the embodiments of this application;

[0107] Figure 5 This is a schematic diagram of a subnetwork in a cross-domain densely connected neural network provided in an embodiment of this application;

[0108] Figure 6 This shows the dense connections between the first-stage spatial domain feature extraction module and the second-stage wavelet domain feature extraction module in the sub-network;

[0109] Figure 7 This is the third flowchart of the image processing method provided in the embodiments of this application;

[0110] Figures 8 to 10 The illustration shows the method using embodiments of this application and the image effect obtained by the method using prior art;

[0111] Figure 11 This is one of the structural diagrams of the image processing apparatus provided in the embodiments of this application;

[0112] Figure 12 This is a second structural diagram of the image processing apparatus provided in the embodiments of this application;

[0113] Figure 13 This is a structural diagram of the first computing module provided in an embodiment of this application;

[0114] Figure 14 This is a structural diagram of the first computing submodule provided in the embodiments of this application;

[0115] Figure 15 This is one of the structural diagrams of the second computing module provided in the embodiments of this application;

[0116] Figure 16 This is a structural diagram of the first computing submodule provided in the embodiments of this application;

[0117] Figure 17 This is the second structural diagram of the second computing module provided in the embodiments of this application;

[0118] Figure 18 This is a structural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0119] In the embodiments of this application, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0120] In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0121] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0122] See Figure 1 , Figure 1 This is a flowchart of the image processing method provided in the embodiments of this application, such as... Figure 1 As shown, it includes the following steps:

[0123] Step 101: Obtain the image to be processed.

[0124] In this embodiment of the application, the image to be processed may be a low-quality image, such as an image taken by a mobile phone.

[0125] Step 102: Process the image to be processed through a cross-domain dense connection neural network to obtain the target image.

[0126] The cross-domain densely connected neural network comprises n sub-networks, where n is an integer and n≥1. Each sub-network includes at least one spatial domain processing module and at least one wavelet domain processing module. Dense connections are formed between the spatial domain processing module and its corresponding wavelet domain processing module. Dense connections are also formed within each spatial domain processing module and each wavelet domain processing module. In this embodiment, one spatial domain processing module may correspond to one wavelet domain processing module.

[0127] When a cross-domain densely connected neural network includes two or more subnetworks, the subnetworks can be connected sequentially.

[0128] Figure 2(a) shows a schematic diagram of the cross-domain densely connected neural network in an embodiment of this application. The cross-domain densely connected neural network includes n sub-networks 201. Each sub-network includes at least one spatial domain processing module 2011 and at least one wavelet domain processing module 2012. Dense connections are formed between the spatial domain processing module and the corresponding wavelet domain processing module. Dense connections are also formed within each spatial domain processing module and each wavelet domain processing module. Each sub-network can be considered as a level, and each level includes two stages: the first stage is the spatial domain image pre-enhancement stage performed by the spatial domain processing module, which extracts features in the spatial domain; the second stage is the wavelet domain feature extraction stage performed by the wavelet domain processing module.

[0129] Dense connections are formed as follows: In a neural network, the input of a convolutional layer is the output of all convolutional layers preceding it, and the output of that convolutional layer serves as the input of all subsequent convolutional layers. Dense connections are formed through the connections of all convolutional layers in the neural network as described above. Figure 2(b) illustrates a schematic diagram of dense connections in a neural network. To ensure maximum information flow between convolutional layers in the network and to preserve feedforward characteristics, each convolutional layer obtains additional input from all previous convolutional layers and passes its own feature map to all subsequent convolutional layers. In this embodiment, dense connections are manifested in two aspects: intra-domain dense connections and inter-domain dense connections. In this embodiment, both the spatial domain processing module and the wavelet domain processing module can be considered as neural networks. Intra-domain dense connections refer to situations where, within the spatial domain processing module and the wavelet domain processing module, the input of a convolutional layer is the output of all convolutional layers preceding it, and the output of that convolutional layer serves as the input of all subsequent convolutional layers. Alternatively, it can be interpreted as the output of a convolutional layer being connected to the outputs of all convolutional layers following it. Dense connections between domains refer to the dense connections formed between spatial domain processing modules and wavelet domain processing modules. That is, within a certain sub-network, the output of the spatial domain processing module and the cascade information of the spatial domain processing module are the inputs of each convolutional layer in its corresponding wavelet domain processing module.

[0130] The spatial domain processing module performs spatial domain processing on the first input image to obtain a first output image, wherein the resolution of the first output image is greater than the resolution of the first input image, and the first input image is the image to be processed or the output image of the previous sub-network; the wavelet domain processing module performs wavelet domain processing on the first output image and the cascade information of the spatial domain processing module to obtain a second output image, wherein the second output image is the target image or the input image of the spatial domain processing module of the next sub-network.

[0131] Specifically, for the first subnetwork of the entire cross-domain densely connected neural network, the first input image of its spatial domain processing module can be the image to be processed; while for subsequent subnetworks, the first input image of its spatial domain processing module is the output image of the previous subnetwork. For the last subnetwork of the entire cross-domain densely connected neural network, the second output image of its wavelet domain processing module is the target image; while for other subnetworks, the second output image of its wavelet domain processing module is the input image of the spatial domain processing module of the next subnetwork.

[0132] For any sub-network, when the spatial domain processing module performs spatial domain processing on the first input image, the spatial domain processing module upsamples the first input image to obtain the first output image. This spatial domain processing can also be called image pre-enhancement. In the spatial domain, a deep neural network is used to improve the image quality of the input image of the spatial domain processing module to a certain extent. Typically, this can make the resolution of the first output image four times that of the first input image (with the length and width each being twice that of the first output image). Therefore, after n levels of enhancement, the resolution of the resulting image is (2... n ×H)*(2 n ×W).

[0133] For any subnetwork, combined with Figure 5 As shown, the spatial domain processing module includes one or more nested residual dense blocks (RRDBs), each nested residual dense block including one or more residual dense blocks (RDBs); the wavelet domain processing module includes one or more nested residual dense blocks, each nested residual dense block including one or more residual dense blocks. Furthermore, the spatial domain processing module may also include convolutional layers, etc. The cascaded information originates from the residual dense blocks of the spatial domain processing module.

[0134] The y-th residual dense module of the x-th nested residual dense block of the spatial domain processing module and the y-th residual dense module of the x-th nested residual dense block of the wavelet domain processing module are cascaded; where x and y are integers, 1≤x≤P, 1≤y≤Q; P is an integer greater than 0, representing the number of nested residual dense blocks included in the spatial domain processing module or the wavelet domain processing module; Q is an integer greater than 0, representing the number of residual dense blocks included in a nested residual dense block of the spatial domain processing module or the wavelet domain processing module.

[0135] Through the above methods, a dense connection is formed between the spatial domain processing module and the wavelet domain processing module. Each RDB includes convolutional layers, LReLU, etc. During the cascading process, the output of any convolutional layer of the y-th residual dense module of the x-th nested residual dense block of the spatial domain processing module is connected to the output of each convolutional layer of the y-th residual dense module of the x-th nested residual dense block of the wavelet domain processing module, thus forming the cascading information of the y-th residual dense module of the x-th nested residual dense block of the wavelet domain processing module.

[0136] Of course, in practical applications, other modules can be used to replace RRDB or RDB.

[0137] In the wavelet domain processing module, the first output image is decomposed using wavelet decomposition to obtain multiple first wavelet sub-bands. Simultaneously, cascade information is obtained from the spatial domain processing module. Then, feature extraction is performed on the multiple first wavelet sub-bands and the cascade information to obtain multiple wavelet domain sub-bands. Finally, an inverse wavelet transform is performed on the multiple wavelet domain sub-bands to obtain the second output image. The inverse wavelet transform includes wavelet reconstruction, etc.

[0138] In this embodiment, a cross-domain densely connected neural network is used to process the image to be processed to obtain the target image. Since the sub-networks of the cross-domain densely connected neural network include at least one spatial domain processing module and at least one wavelet domain processing module, and wavelet domain processing is performed based on the spatial domain processing module, the features of the spatial domain can be superimposed onto the wavelet domain, thereby improving the feature diversity of the wavelet domain. Therefore, the solution using this embodiment can improve the image enhancement effect.

[0139] See Figure 3 , Figure 3 This is a flowchart of the image processing method provided in the embodiments of this application, such as... Figure 3 As shown, it includes the following steps:

[0140] Step 301: Obtain the image to be processed.

[0141] In this embodiment of the application, the image to be processed may be a low-quality image, such as an image taken by a mobile phone.

[0142] Step 302: Process the image to be processed through a cross-domain dense connection neural network to obtain the target image.

[0143] The descriptions of steps 301 and 302 can be found in the descriptions of the foregoing embodiments.

[0144] Step 303: Calculate the first loss function.

[0145] Specifically, in this step, the Manhattan distance (L1) or Euclidean distance (L2 distance) between the target image and the spatial domain real image can be calculated, and the Manhattan distance or Euclidean distance can be used as the first loss function.

[0146] Specifically, when calculating the Manhattan distance or Euclidean distance, the target image can be processed through a VGG (Visual Graphics Generator) network to obtain a first intermediate target image; the spatial domain ground truth image can also be processed through a VGG network to obtain a first intermediate spatial domain ground truth image. Then, the Manhattan distance or Euclidean distance between the first intermediate target image and the first intermediate spatial domain ground truth image is calculated.

[0147] For example, the loss function can be based on the spatial domain ground truth image. GT It is obtained by calculating the L1 distance (Manhattan distance) or L2 distance (Euclidean distance) between the target image and the pixels.

[0148] At this point, the L1 distance can be defined as: L pixel =||I GT -D‖1. Where D represents the target image.

[0149] For example, it can also be based on spatial domain ground truth images. GT The L1 or L2 distance is calculated from the result of the target image after passing through the VGG network.

[0150] At this point, the L1 distance can be defined as: L VGG =‖φ(I GT )-φ(D)‖1, where φ() represents the output of a certain layer of the VGG19 network.

[0151] Specifically, in this step, the first loss function can also be obtained based on the result of processing the target image and the real spatial domain image by the discriminator of GAN (Generative Adversarial Networks), according to the following formula:

[0152] L adv,G1 = -log(1-D Ra (D,I GT ))-log(D Ra (I GT ,D));

[0153] Among them, L adv,G1 Let D represent the first loss function. Ra Let D represent the discriminator of the GAN, and let I represent the target image. GT This represents the true image of the spatial domain.

[0154] Specifically, this is based on the spatial domain ground truth image (I) GT The calculation is performed using the result of the discriminant network output from the target image after passing through the generative adversarial network.

[0155] The loss function of a discriminant network can be defined as:

[0156] L adv,D =-log(D Ra (D,I GT )))-log(1-D Ra (I GT ,D));

[0157] The loss function (i.e., the first loss function) of the generator network is:

[0158] L adv,G1 = -log(1-D Ra (D,I GT ))-log(D Ra (I GT ,D));

[0159] Among them, D Ra D represents the discriminator, and I represents the target image. GT This represents the true image of the spatial domain.

[0160] It should be noted that the above formula calculates the loss function for a single image. If multiple images are used for training, the loss function is the average of the loss functions for all images.

[0161] Step 304: Calculate the second loss function.

[0162] In this step, the spatial domain real image can be decomposed into multiple second wavelet sub-bands, each corresponding to a wavelet domain sub-band. Then, for any first target wavelet sub-band among the multiple second wavelet sub-bands, the Manhattan distance or Euclidean distance between the first target wavelet sub-band and the corresponding wavelet domain sub-band is calculated. This Manhattan distance or Euclidean distance is used as the loss function for the first target wavelet sub-band, and the second loss function is obtained based on these multiple loss functions. For example, the second loss function can be obtained by averaging the multiple loss functions, or by multiplying each loss function by its corresponding weight and then averaging the sum of the products.

[0163] When calculating the Manhattan or Euclidean distance between the first target wavelet sub-band and its corresponding wavelet domain sub-band, the first target wavelet sub-band can be processed through a VGG network to obtain a first intermediate target wavelet sub-band. Then, the wavelet domain sub-band corresponding to the first target wavelet sub-band is processed through a VGG network to obtain a first intermediate wavelet domain sub-band. Finally, the Manhattan or Euclidean distance between the first intermediate target wavelet sub-band and the first intermediate wavelet domain sub-band is calculated.

[0164] Specifically, in this step, firstly, the spatial domain ground truth image (I) is... GT First, decompose it into different wavelet subbands LL GT LH GT HL GT HH GT Then, it is compared with the different wavelet subbands F(LL) output by the corresponding wavelet domain processing module. SR ),F(LH SR ),F(HL SR ),F(HH SR The calculation is performed to obtain LL. GT ,LH GT HL GT ,HH GT These represent the low-frequency sub-band, vertical high-frequency sub-band, horizontal high-frequency sub-band, and diagonal high-frequency sub-band of the real image in the spatial domain, respectively. F() represents the low-frequency sub-band, vertical high-frequency sub-band, horizontal high-frequency sub-band, and diagonal high-frequency sub-band output by the wavelet domain processing module.

[0165] For example, the loss function can be calculated based on the L1 distance (Manhattan distance) or L2 distance (Euclidean distance) of the pixels in the two wavelet subbands mentioned above.

[0166] At this point, taking the LH subband as an example, the L1 distance can be defined as: L pixel=‖F(LH SR )-LH GT ‖1.

[0167] For example, the L1 or L2 distance can also be calculated based on the results of the two wavelet subbands after passing through the VGG network.

[0168] Taking the LH subband as an example, the L1 distance can be defined as: L VGG_LH =‖φ(F(LH SR )-φ(LH GT )‖1, where φ() represents the output of a certain layer of the VGG19 network.

[0169] Furthermore, in this step, the spatial domain real image can be decomposed into multiple second wavelet sub-bands, which correspond to multiple wavelet domain sub-bands. Then, for any first target wavelet sub-band among the multiple second wavelet sub-bands, the first target wavelet sub-band and its corresponding wavelet domain sub-band are processed by the discriminator of the GAN network, and the loss function of the first target wavelet sub-band is obtained according to the following formula:

[0170] L adv,G2 = -log(1-D Ra (I GT ,F(I SR )))-log(D Ra (F(I SR ),I GT ))

[0171] Among them, L adv,G2 Let D represent the second loss function. Ra I represents the discriminator of a GAN. GT F(I) represents the first target wavelet subband. SR ) represents the wavelet domain sub-band corresponding to the first target wavelet sub-band.

[0172] Finally, the second loss function is obtained based on the multiple loss functions. For example, the second loss function can be obtained by averaging the multiple loss functions, or the multiple loss functions can be multiplied by their corresponding weights respectively, and then the sum of the multiple products can be averaged to obtain the second loss function.

[0173] Specifically, the calculation is based on the results of the discriminant network output in the generative adversarial network for the two wavelet subbands mentioned above.

[0174] Taking the LH subband as an example, the loss function of the discriminant network can be defined as:

[0175] L adv，D =-log(D Ra(LH GT ,F(LH SR )))-log(1-D Ra (F(LH SR ), LH GT ));

[0176] The loss function (i.e., the second loss function) of the generator network is:

[0177] L adv,G2 = -log(1-D Ra (LH GT , F(LH SR )))-log(D Ra (F(LH SR ),LH GT ));

[0178] Among them, D Ra LH represents a relative discriminator. GT Indicates LH subband, F(LH) SR ) represents the wavelet domain subband corresponding to the LH subband.

[0179] It should be noted that the above formula is a loss function calculated for a single image. If multiple images are trained together, the loss function is the average of the loss functions of all images.

[0180] In the embodiments of this application, there is no strict order between steps 303 and 304. Step 303 can be executed first and then step 304, or step 304 can be executed first and then step 303.

[0181] Step 305: Calculate the third loss function of the cross-domain dense connection neural network using the first loss function and the second loss function.

[0182] Specifically, in this step, the product of the first loss function and the first coefficient, and the sum of the product of the second loss function and the second coefficient, can be used as the third loss function.

[0183] Assume the loss function of the entire network is: L total We can obtain:

[0184] L total =αL spatial +βL wavelet Where α and β are the coefficients of the spatial domain loss function and the wavelet domain loss function, respectively, and L spatial L represents the spatial domain loss function. wavelet This represents the wavelet domain loss function.

[0185] As can be seen from the above description, in this embodiment, features from the spatial domain are superimposed onto the wavelet domain, thereby improving the feature diversity in the wavelet domain. Therefore, compared to simply performing dense connections within the spatial domain or wavelet domain, the solution using this embodiment yields better image enhancement results.

[0186] Furthermore, the scheme using the embodiments of this application not only calculates the loss function in the spatial domain but also in the wavelet domain. Therefore, the loss function obtained in the wavelet domain can help to extract features for different wavelet bands (LL, LH, HL, and HH) respectively, thereby enhancing image details in a targeted manner.

[0187] Figure 4 This paper illustrates a schematic diagram of multi-level (multi-scale) image enhancement using a cross-domain densely connected neural network proposed in an embodiment of this application. Each level (or each sub-network) comprises two stages: the first stage (Stage-1) is a spatial domain image pre-enhancement stage, where feature extraction is performed in the spatial domain; the second stage (Stage-2) is a wavelet domain feature extraction stage. The two stages are connected by dense connections, linking the outputs of the convolutional layers in the modules. Compared to the feature image resolution H×W of the first stage spatial domain feature extraction module, the feature image resolution of the second stage wavelet domain feature extraction stage is 2H×2W. In the multi-level structure, image enhancement is achieved through super-resolution technology. Each stage increases the resolution of the input image by 4 times (the length and width are each doubled). The resolution of an input image after n stages of enhancement is (2... n ×H)*(2 n ×W).

[0188] Figure 5 This illustration shows a schematic diagram of a subnetwork in the cross-domain densely connected neural network provided in an embodiment of this application. Each level comprises two stages: the first stage is an image pre-enhancement stage in the spatial domain, and the second stage is a feature extraction stage in the wavelet domain. In the image pre-enhancement stage, the image quality of the input image is improved to a certain extent in the spatial domain using a deep neural network. Then, in the second stage, the image output from the first stage is decomposed into many wavelet sub-bands using wavelet decomposition, and features are extracted from these wavelet sub-bands using the wavelet domain deep neural network in the second stage. Finally, the output of the wavelet domain feature extraction module in the second stage is reconstructed to obtain an output image in the spatial domain.

[0189] from Figure 5As can be seen, the first stage includes convolutional layers (Conv) and CDDB-I (spatial domain dense connections). Each CDDB-I module can contain multiple convolutional layers, LReLU, and RRDB (Residual-in-Residual Dense Blocks). Each RRDB contains multiple RDBs (Residual Dense Blocks). The second stage includes CDDB-W (wavelet domain dense connections). Each CDDB-W module can contain multiple convolutional layers, LReLU, and RRDB. Each RRDB contains multiple RDBs. Of course, in practical applications, other modules can be used to replace RRDBs.

[0190] For input image I LR (Resolution is H×W), after the first stage of processing, the following is generated: (Resolution is 2H×2W). After that, It is input into the second stage and divided into multiple subbands LL2 SR LH2 SR ...etc. Afterwards, through wavelet domain feature extraction, the corresponding wavelet domain subband F(LL2) is obtained. SR ), F(LH2 SR ...The output of a convolutional layer within any RDB of any RRDB in CDDB-I is connected to the output of every convolutional layer within the corresponding RDB of the RRDB in CDDB-W. For example, the output of the first convolutional layer within the first RDB of the first RRDB in CDDB-I is connected to the output of every convolutional layer within the first RDB of the first RRDB in CDDB-W. GT Represents the ground truth image in the spatial domain.

[0191] Figure 6 This diagram illustrates the dense connections between the first-stage spatial domain feature extraction module and the second-stage wavelet domain feature extraction module in a given level. Both the spatial and wavelet domains contain three densely connected modules. The diagram only shows details of the first densely connected module in each stage; the structures of the other densely connected modules are identical to those shown.

[0192] This application's embodiments include two types of dense connections: intra-domain dense connections and cross-domain dense connections. Specifically, these are dense connections between feature images within the spatial domain or wavelet domain, and dense connections between feature images in the spatial domain and feature images in the wavelet domain.

[0193] Cross-domain dense connections are achieved as follows: The feature image output from a convolutional layer of the first-stage (CDDB-I) spatial domain feature extraction module is first upsampled to increase its resolution by four times (by a factor of two in both length and width). The output of the first stage is then input into the second stage (CDDB-W). Wavelet decomposition is used to decompose the improved spatial domain feature image into different wavelet sub-bands. Finally, these different sub-bands are concatenated with the wavelet domain feature images output from the corresponding convolutional layers in the second-stage wavelet domain feature extraction module, thereby achieving cross-domain dense connections.

[0194] exist Figure 6 In the diagram, 61 and 62 indicate the structures of the RDB in the spatial domain and the RDB in the wavelet domain, respectively. From... Figure 6 As can be seen, the output of a certain convolutional layer (or LReLu) in the spatial domain RDB is the input of each convolutional layer after that convolutional layer in the spatial domain RDB; the output (i.e., the cascade information) of a certain convolutional layer (or LReLu) in the spatial domain RDB is connected to the output of each convolutional layer (or LReLu) in the wavelet domain RDB, thus forming a dense connection across domains.

[0195] Figure 5 and Figure 6 Only a two-level wavelet decomposition is shown, with a total of 8 wavelet subbands, 4 at each level: LL, LH, HL, and HH. However, since LL1 is further decomposed into a second-level wavelet subband, generally only the highest-level LL wavelet subband is retained. Therefore, in the two-level wavelet decomposition, generally only LL2 is retained. Thus, in the embodiments of this application, there are a total of 7 wavelet subbands: LL2, LH1, HL1, HH1, LH2, HL2, and HH2.

[0196] Figure 7 A flowchart illustrating the processing of a specific level within a multi-level structure is provided. This flowchart describes a schematic diagram of the training process for the method proposed in the embodiments of this application. Figure 7 As shown, the method includes: s

[0197] Step 701: The input image is transmitted to the first-stage spatial domain feature extraction module CDDB_I.

[0198] Step 702: In each densely connected module, the feature image output by a certain convolutional layer is first upsampled to increase its resolution to 4 times its original value (2 times in length and width).

[0199] Step 703: Decompose the improved spatial domain feature image into multi-level wavelet sub-bands using wavelet decomposition.

[0200] Step 704: Store the wavelet subband for use in subsequent dense connection operations.

[0201] Step 705: Decompose the output wavelet of the first-stage spatial domain feature extraction module into different wavelet subbands, and then transmit them to the second-stage wavelet domain feature extraction module CDDB_W.

[0202] Step 706: The feature extraction modules corresponding to CDDB_I and CDDB_W are connected in a dense connection manner.

[0203] Step 707: Calculate the loss function between the real image and the generated image in the spatial domain and the wavelet domain respectively. The loss function obtained by superimposing the loss functions of the two domains is used as the loss function of the entire network.

[0204] Step 708: Perform wavelet reconstruction on the output of the second-stage wavelet domain feature extraction module to obtain an output image in the spatial domain. This output image serves as the input for the next stage.

[0205] As can be seen from the above description, in this embodiment, features from the spatial domain are superimposed onto the wavelet domain, thereby improving the feature diversity in the wavelet domain. Therefore, compared to simply performing dense connections within the spatial domain or wavelet domain, the solution using this embodiment yields better image enhancement results.

[0206] Figures 8 to 10 The illustration shows the method using embodiments of this application and the image effect obtained by comparing it with that obtained using prior art methods. Wherein, in Figures 8 to 10 In the image, the leftmost image in the first row is the original image, the middle image is the image obtained using existing methods, and the rightmost image is the enhanced image obtained using the method described in this application. Figures 8 to 10 In the image, the second row is a magnified view of the area within the box in the first row. The comparison shows that the solution implemented in this application enhances image details.

[0207] This application also provides an image processing apparatus. See [link to relevant documentation]. Figure 11 , Figure 11 This is a structural diagram of the image processing apparatus provided in an embodiment of this application. Figure 11 As shown, the image processing apparatus 1100 includes:

[0208] The first acquisition module 1101 is used to acquire the image to be processed; the first processing module 1102 is used to process the image to be processed through a cross-domain dense connection neural network to obtain the target image.

[0209] The cross-domain dense connection neural network includes n sub-networks, where n is an integer and n≥1;

[0210] The sub-network includes at least one spatial domain processing module and at least one wavelet domain processing module. Dense connections are formed between the spatial domain processing module and the corresponding wavelet domain processing module. Dense connections are formed within the spatial domain processing module and within the wavelet domain processing module.

[0211] The spatial domain processing module performs spatial domain processing on the first input image to obtain a first output image, wherein the resolution of the first output image is greater than the resolution of the first input image, and the first input image is the image to be processed or the output image of the previous sub-network.

[0212] The wavelet domain processing module performs wavelet domain processing on the first output image and the cascade information of the spatial domain processing module to obtain a second output image, wherein the second output image is either the target image or the input image of the spatial domain processing module of the next sub-network.

[0213] The spatial domain processing module upsamples the first input image to obtain the first output image, and the resolution of the first output image is four times that of the first input image.

[0214] The spatial domain processing module includes one or more nested residual dense blocks, each nested residual dense block including one or more residual dense blocks; the wavelet domain processing module includes one or more nested residual dense blocks, each nested residual dense block including one or more residual dense blocks.

[0215] The y-th residual dense module of the x-th nested residual dense block of the spatial domain processing module and the y-th residual dense module of the x-th nested residual dense block of the wavelet domain processing module are cascaded.

[0216] Where x and y are integers, 1≤x≤P, 1≤y≤Q; P is an integer greater than 0, representing the number of nested residual dense blocks included in the spatial domain processing module or the wavelet domain processing module; Q is an integer greater than 0, representing the number of residual dense blocks included in a nested residual dense block of the spatial domain processing module or the wavelet domain processing module.

[0217] The wavelet domain processing module is used for:

[0218] Wavelet decomposition is performed on the first output image to obtain multiple first wavelet sub-bands;

[0219] Feature extraction is performed on the plurality of first wavelet sub-bands and the cascaded information to obtain a plurality of wavelet domain sub-bands;

[0220] The second output image is obtained by performing inverse wavelet transform on the multiple wavelet domain subbands.

[0221] Among them, such as Figure 12 As shown, the device further includes:

[0222] The first calculation module 1103 is used to calculate the first loss function for the nth sub-network;

[0223] The second calculation module 1104 is used to calculate the second loss function;

[0224] The third calculation module 1105 is used to calculate the third loss function of the cross-domain dense connection neural network using the first loss function and the second loss function.

[0225] Among them, such as Figure 13 As shown, the first computing module 1103 includes:

[0226] The first calculation submodule 11031 is used to calculate the Manhattan distance or Euclidean distance between the target image and the spatial domain real image;

[0227] The second calculation submodule 11032 is used to use the Manhattan distance or Euclidean distance as the first loss function.

[0228] Among them, such as Figure 14 As shown, the first calculation submodule 11031 includes:

[0229] The first acquisition unit 11031a is used to process the target image through a VGG network to obtain a first intermediate target image;

[0230] The second acquisition unit 11031b is used to process the spatial domain real image through a VGG network to obtain a first intermediate spatial domain real image.

[0231] The first calculation unit 11031c is used to calculate the Manhattan distance or Euclidean distance between the first intermediate target image and the first intermediate spatial domain real image.

[0232] The first calculation module 1103 is further configured to obtain the first loss function based on the result of processing the target image and the spatial domain real image by the discriminator of a Generative Adversarial Network (GAN) according to the following formula:

[0233] L adv,G1 = -log(1-D Ra (D,I GT ))-log(D Ra (I GT ,D));

[0234] Among them, L adv,G1Let D represent the first loss function. Ra Let D represent the discriminator of the GAN, and let I represent the target image. GT This represents the true image of the spatial domain.

[0235] Among them, such as Figure 15 As shown, the second calculation module 1104 includes:

[0236] The first decomposition submodule 11041 is used to decompose the spatial domain real image into multiple second wavelet sub-bands, wherein the multiple second wavelet sub-bands correspond to multiple wavelet domain sub-bands.

[0237] The first calculation submodule 11042 is used to calculate the Manhattan distance or Euclidean distance between the first target wavelet subband and the corresponding wavelet domain subband for any first target wavelet subband among a plurality of second wavelet subbands.

[0238] The first acquisition submodule 11043 is used to use the Manhattan distance or Euclidean distance as the loss function of the first target wavelet subband;

[0239] The second acquisition submodule 11044 is used to obtain the second loss function based on multiple loss functions.

[0240] Among them, such as Figure 16 As shown, the first calculation submodule 11042 includes:

[0241] The first processing unit 110421 is used to process the first target wavelet subband through a VGG network GAN to obtain the first intermediate target wavelet subband.

[0242] The second processing unit 110422 is used to process the wavelet domain sub-band corresponding to the first target wavelet sub-band through a VGG network to obtain the first intermediate wavelet domain sub-band.

[0243] The first calculation unit 110423 is used to calculate the Manhattan distance or Euclidean distance between the first intermediate target wavelet sub-band and the first intermediate wavelet domain sub-band.

[0244] Among them, such as Figure 17 As shown, the second calculation module 1104 includes:

[0245] The second decomposition submodule 11045 is used to decompose the spatial domain real image into multiple second wavelet sub-bands, wherein the multiple second wavelet sub-bands correspond to multiple wavelet domain sub-bands.

[0246] The second calculation submodule 11046 is used to process any first target wavelet subband among multiple second wavelet subbands, passing the first target wavelet subband and its corresponding wavelet domain subband through the discriminator of the GAN network, and obtaining the loss function of the first target wavelet subband according to the following formula:

[0247] L adv,G2 = -log(1-D Ra (I GT ,F(I SR )))-log(D Ra (F(I SR ),I GT ))

[0248] Among them, L adv,G2 Let D represent the second loss function. Ra I represents the discriminator of a GAN. GT F(I) represents the first target wavelet subband. SR () represents the wavelet domain sub-band corresponding to the first target wavelet sub-band;

[0249] The third acquisition submodule 11047 is used to obtain the second loss function based on the multiple loss functions.

[0250] The third calculation module is used to use the sum of the product of the first loss function and the first coefficient, and the product of the second loss function and the second coefficient, as the third loss function.

[0251] The apparatus provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0252] It should be noted that the division of units in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0253] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0254] See Figure 18 This application also provides a hardware structure for an electronic device. For example... Figure 18 As shown, the electronic device 1800 includes:

[0255] Processor 1802; and

[0256] Memory 1804 stores program instructions, wherein when the processor executes the program instructions, the processor 1802 performs the following steps:

[0257] Obtain the image to be processed;

[0258] The image to be processed is processed through a cross-domain densely connected neural network to obtain the target image;

[0259] The cross-domain dense connection neural network includes n sub-networks, where n is an integer and n≥1;

[0260] The sub-network includes at least one spatial domain processing module and at least one wavelet domain processing module. Dense connections are formed between the spatial domain processing module and the corresponding wavelet domain processing module. Dense connections are formed within the spatial domain processing module and within the wavelet domain processing module.

[0261] The spatial domain processing module performs spatial domain processing on the first input image to obtain a first output image, wherein the resolution of the first output image is greater than the resolution of the first input image, and the first input image is the image to be processed or the output image of the previous sub-network.

[0262] The wavelet domain processing module performs wavelet domain processing on the first output image and the cascade information of the spatial domain processing module to obtain a second output image, wherein the second output image is either the target image or the input image of the spatial domain processing module of the next sub-network.

[0263] Furthermore, such as Figure 18 As shown, the electronic device 1800 may also include a network interface 1801, an input device 1803, a hard disk 1805, and a display device 1806.

[0264] The various interfaces and devices described above can be interconnected via a bus architecture. The bus architecture can include any number of interconnected buses and bridges. Specifically, various circuits representing one or more central processing units (CPUs), represented by processor 1802, and one or more memories, represented by memory 1804, are connected together. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. It is understood that the bus architecture is used to implement communication between these components. In addition to the data bus, the bus architecture also includes a power bus, a control bus, and a status signal bus, which are well known in the art and will not be described in detail herein.

[0265] The network interface 1801 can be connected to a network (such as the Internet, local area network, etc.), receive data from the network, and save the received data to the hard disk 1805.

[0266] The input device 1803 can receive various instructions input by the operator and send them to the processor 1802 for execution. The input device 1803 may include a keyboard or a clicking device (e.g., a mouse, trackball, touchpad, or touchscreen).

[0267] The display device 1806 can display the results obtained by the processor 1802 executing instructions.

[0268] The memory 1804 is used to store programs and data necessary for the operation of the operating system, as well as intermediate results and other data during the calculation process of the processor 1802.

[0269] It is understood that the memory 1804 in the embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. The memory 1804 of the apparatus and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0270] In some implementations, memory 1804 stores elements, executable modules or data structures, or subsets thereof, or extended sets thereof: operating system 18041 and application program 18042.

[0271] The operating system 18041 includes various system programs, such as a framework layer, a core library layer, and a driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 18042 includes various applications, such as a browser, used to implement various application functions. Programs implementing the methods of the embodiments of this application can be included in application program 18042.

[0272] The image processing method disclosed in the above embodiments of the present invention can be applied to or implemented by the processor 1802. The processor 1802 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above image processing method can be completed by the integrated logic circuit of the hardware in the processor 1802 or by instructions in the form of software. The processor 1802 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1804. Processor 1802 reads the information in memory 1804 and, in conjunction with its hardware, completes the steps of the above method.

[0273] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0274] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.

[0275] Specifically, when the program is executed by the processor 1802, it can also perform the following steps:

[0276] The first input image is upsampled to obtain the first output image, and the resolution of the first output image is 4 times that of the first input image.

[0277] The spatial domain processing module includes one or more nested residual dense blocks, each nested residual dense block including one or more residual dense blocks; the wavelet domain processing module includes one or more nested residual dense blocks, each nested residual dense block including one or more residual dense blocks.

[0278] The y-th residual dense module of the x-th nested residual dense block of the spatial domain processing module and the y-th residual dense module of the x-th nested residual dense block of the wavelet domain processing module are cascaded.

[0279] Where x and y are integers, 1≤x≤P, 1≤y≤Q; P is an integer greater than 0, representing the number of nested residual dense blocks included in the spatial domain processing module or the wavelet domain processing module; Q is an integer greater than 0, representing the number of residual dense blocks included in a nested residual dense block of the spatial domain processing module or the wavelet domain processing module.

[0280] Specifically, when the program is executed by the processor 1802, it can also perform the following steps:

[0281] Wavelet decomposition is performed on the first output image to obtain multiple first wavelet sub-bands;

[0282] Feature extraction is performed on the plurality of first wavelet sub-bands and the cascaded information to obtain a plurality of wavelet domain sub-bands;

[0283] The second output image is obtained by performing inverse wavelet transform on the multiple wavelet domain subbands.

[0284] Specifically, when the program is executed by the processor 1802, it can also perform the following steps:

[0285] Calculate the first loss function;

[0286] Calculate the second loss function;

[0287] Using the first loss function and the second loss function, the third loss function of the cross-domain densely connected neural network is calculated.

[0288] Specifically, when the program is executed by the processor 1802, it can also perform the following steps:

[0289] Calculate the Manhattan distance or Euclidean distance between the target image and the real image in the spatial domain;

[0290] The Manhattan distance or Euclidean distance is used as the first loss function.

[0291] Specifically, when the program is executed by the processor 1802, it can also perform the following steps:

[0292] The target image is processed by the Visual Image Generator (VGG) network to obtain the first intermediate target image;

[0293] The spatial domain real image is processed through a VGG network to obtain a first intermediate spatial domain real image.

[0294] Calculate the Manhattan distance or Euclidean distance between the first intermediate target image and the first intermediate spatial domain real image.

[0295] Specifically, when the program is executed by the processor 1802, it can also perform the following steps:

[0296] Based on the result of processing the target image and the real spatial domain image by the discriminator of a Generative Adversarial Network (GAN), the first loss function is obtained according to the following formula:

[0297] L adv,G1 = -log(1-D Ra (D,I GT ))-log(D Ra (I GT ,D));

[0298] Among them, L adv,G1 Let D represent the first loss function. Ra Let D represent the discriminator of the GAN, and let I represent the target image. GT This represents the true image of the spatial domain.

[0299] Specifically, when the program is executed by the processor 1802, it can also perform the following steps:

[0300] The spatial domain real image is decomposed into multiple second wavelet sub-bands, and the multiple second wavelet sub-bands correspond to multiple wavelet domain sub-bands;

[0301] For any first target wavelet sub-band among multiple second wavelet sub-bands, calculate the Manhattan distance or Euclidean distance between the first target wavelet sub-band and the corresponding wavelet domain sub-band;

[0302] The Manhattan distance or Euclidean distance is used as the loss function for the first target wavelet subband;

[0303] The second loss function is obtained based on multiple loss functions.

[0304] Specifically, when the program is executed by the processor 1802, it can also perform the following steps:

[0305] The first target wavelet subband is processed through a VGG network to obtain the first intermediate target wavelet subband.

[0306] The wavelet domain sub-band corresponding to the first target wavelet sub-band is processed by the VGG network to obtain the first intermediate wavelet domain sub-band.

[0307] Calculate the Manhattan distance or Euclidean distance between the first intermediate target wavelet sub-band and the first intermediate wavelet domain sub-band.

[0308] Specifically, when the program is executed by the processor 1802, it can also perform the following steps:

[0309] The spatial domain real image is decomposed into multiple second wavelet sub-bands, and the multiple second wavelet sub-bands correspond to multiple wavelet domain sub-bands;

[0310] For any first target wavelet subband among multiple second wavelet subbands, the first target wavelet subband and its corresponding wavelet domain subband are processed by the discriminator of the GAN network, and the loss function of the first target wavelet subband is obtained according to the following formula:

[0311] L adv,G2 = -log(1-D Ra (I GT ,F(I SR )))-log(D Ra (F(I SR ),I GT ))

[0312] Among them, L adv,G2 Let D represent the second loss function. RaI represents the discriminator of a GAN. GT F(I) represents the first target wavelet subband. SR () represents the wavelet domain sub-band corresponding to the first target wavelet sub-band;

[0313] The second loss function is obtained based on the multiple loss functions.

[0314] Specifically, when the program is executed by the processor 1802, it can also perform the following steps:

[0315] The third loss function is obtained by combining the product of the first loss function and the first coefficient with the product of the second loss function and the second coefficient.

[0316] The electronic device provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0317] This application also provides a readable storage medium storing a program. When executed by a processor, this program implements the various processes of the above-described image processing method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0318] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0319] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0320] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. An image processing method, characterized by, include: Obtain the image to be processed; The image to be processed is processed through a cross-domain densely connected neural network to obtain the target image; The cross-domain dense connection neural network includes n sub-networks, where n is an integer and n≥1; The sub-network includes at least one spatial domain processing module and at least one wavelet domain processing module. Dense connections are formed between the spatial domain processing module and the corresponding wavelet domain processing module. Dense connections are formed within the spatial domain processing module and within the wavelet domain processing module. The spatial domain processing module performs spatial domain processing on the first input image to obtain a first output image, wherein the resolution of the first output image is greater than the resolution of the first input image, and the first input image is the image to be processed or the output image of the previous sub-network. The wavelet domain processing module performs wavelet domain processing on the first output image and the cascade information of the spatial domain processing module to obtain a second output image, wherein the second output image is either the target image or the input image of the spatial domain processing module of the next sub-network.

2. The method of claim 1, wherein, The spatial domain processing module performs spatial domain processing on the first input image to obtain a first output image, including: The spatial domain processing module upsamples the first input image to obtain the first output image, the resolution of which is four times that of the first input image.

3. The method of claim 1, wherein, The spatial domain processing module includes one or more nested residual dense blocks, each nested residual dense block including one or more residual dense blocks; the wavelet domain processing module includes one or more nested residual dense blocks, each nested residual dense block including one or more residual dense blocks. The y-th residual dense module of the x-th nested residual dense block of the spatial domain processing module and the y-th residual dense module of the x-th nested residual dense block of the wavelet domain processing module are cascaded. Where x and y are integers, 1≤x≤P, 1≤y≤Q; P is an integer greater than 0, representing the number of nested residual dense blocks included in the spatial domain processing module or the wavelet domain processing module; Q is an integer greater than 0, representing the number of residual dense blocks included in a nested residual dense block of the spatial domain processing module or the wavelet domain processing module.

4. The method of claim 3, wherein, The wavelet domain processing module performs wavelet domain processing on the first output image and the cascaded information of the spatial domain processing module to obtain a second output image, including: The wavelet domain processing module performs wavelet decomposition on the first output image to obtain multiple first wavelet sub-bands; Feature extraction is performed on the plurality of first wavelet sub-bands and the cascaded information to obtain a plurality of wavelet domain sub-bands; The second output image is obtained by performing inverse wavelet transform on the multiple wavelet domain subbands.

5. The method of claim 1, wherein, For the nth subnetwork, the method further includes: Calculate the first loss function; Calculate the second loss function; Using the first loss function and the second loss function, the third loss function of the cross-domain densely connected neural network is calculated.

6. The method of claim 5, wherein, The calculation of the first loss function includes: Calculate the Manhattan distance or Euclidean distance between the target image and the real image in the spatial domain; The Manhattan distance or Euclidean distance is used as the first loss function.

7. The method of claim 6, wherein, The calculation of the Manhattan distance or Euclidean distance between the target image and the spatial domain ground truth image includes: The target image is processed by the Visual Image Generator (VGG) network to obtain the first intermediate target image; The spatial domain real image is passed through a VGG network to obtain a first intermediate spatial domain real image; Calculate the Manhattan distance or Euclidean distance between the first intermediate target image and the first intermediate spatial domain real image.

8. The method of claim 5, wherein, The calculation of the first loss function includes: Based on the result of processing the target image and the real spatial domain image by the discriminator of a Generative Adversarial Network (GAN), the first loss function is obtained according to the following formula: L adv,G1 = -log(l - D Ra (D, I GT )) - log(D Ra (I GT , D)); wherein L adv,G1 represents the first loss function, D Ra represents a discriminator of the GAN, D represents the target image, I GT represents the spatial domain real image.

9. The method of claim 5, wherein, The calculation of the second loss function includes: The spatial domain real image is decomposed into multiple second wavelet sub-bands, and the multiple second wavelet sub-bands correspond to multiple wavelet domain sub-bands; For any first target wavelet sub-band among multiple second wavelet sub-bands, calculate the Manhattan distance or Euclidean distance between the first target wavelet sub-band and the corresponding wavelet domain sub-band; The Manhattan distance or Euclidean distance is used as the loss function for the first target wavelet subband; The second loss function is obtained based on the multiple loss functions.

10. The method of claim 9, wherein, The calculation of the Manhattan distance or Euclidean distance between the first target wavelet sub-band and the corresponding wavelet domain sub-band includes: The first target wavelet subband is processed through a VGG network to obtain the first intermediate target wavelet subband. The wavelet domain sub-band corresponding to the first target wavelet sub-band is processed by the VGG network to obtain the first intermediate wavelet domain sub-band. Calculate the Manhattan distance or Euclidean distance between the first intermediate target wavelet sub-band and the first intermediate wavelet domain sub-band.

11. The method according to claim 5, characterized in that, The calculation of the second loss function includes: The spatial domain real image is decomposed into multiple second wavelet sub-bands, and the multiple second wavelet sub-bands correspond to multiple wavelet domain sub-bands; For any first target wavelet subband among multiple second wavelet subbands, the first target wavelet subband and its corresponding wavelet domain subband are processed by the discriminator of the GAN network, and the loss function of the first target wavelet subband is obtained according to the following formula: L adv,G2 = -log(l - D Ra (I GT , F(I SR )))-log(D Ra (F(I SR ), I GT )) wherein, L adv,G2 represents the second loss function, D Ra represents a discriminator of the GAN, I GT represents the first target wavelet subband, F(I SR ) represents a wavelet domain subband corresponding to the first target wavelet subband; The second loss function is obtained based on the multiple loss functions.

12. The method of claim 5, wherein, The step of calculating the third loss function of the cross-domain densely connected neural network using the first loss function and the second loss function includes: The third loss function is obtained by combining the product of the first loss function and the first coefficient with the product of the second loss function and the second coefficient.

13. An image processing apparatus characterized by comprising: include: The first acquisition module is used to acquire the image to be processed; The first processing module is used to process the image to be processed through a cross-domain dense connection neural network to obtain the target image; The cross-domain dense connection neural network includes n sub-networks, where n is an integer and n≥1; The sub-network includes at least one spatial domain processing module and at least one wavelet domain processing module. Dense connections are formed between the spatial domain processing module and the corresponding wavelet domain processing module. Dense connections are formed within the spatial domain processing module and within the wavelet domain processing module. The spatial domain processing module performs spatial domain processing on the first input image to obtain a first output image, wherein the resolution of the first output image is greater than the resolution of the first input image, and the first input image is the image to be processed or the output image of the previous sub-network. The wavelet domain processing module performs wavelet domain processing on the first output image and the cascade information of the spatial domain processing module to obtain a second output image, wherein the second output image is either the target image or the input image of the spatial domain processing module of the next sub-network.

14. An electronic device comprising: A memory, a processor, and a program stored in the memory and executable on the processor; characterized in that the processor is configured to read the program from the memory to implement the steps of the image processing method as described in any one of claims 1 to 12.

15. A readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps of the image processing method according to any one of claims 1 to 12.

Images