Image processing method and device, neural network training method, and storage medium

Abstract

An image processing method and apparatus, a neural network training method, and a storage medium. The image processing method includes: obtaining an input image; and processing the input image using a generative network to generate an output image; the generative network includes a first subnetwork and at least one second subnetwork. Processing the input image using the generative network to generate an output image includes: processing the input image using the first subnetwork to obtain multiple first feature maps; performing branching and weight sharing processing on the multiple first feature maps using the at least one second subnetwork to obtain multiple second feature maps; and processing the multiple second feature maps to obtain the output image. The image processing method combines branching and weight sharing for image enhancement, which can reduce the number of parameters and facilitate gradient calculation during backpropagation. Thus, while outputting high-quality images, it can also improve processing speed and convergence speed.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to an image processing method and apparatus, a neural network training method, and a storage medium. Background Art

[0002] Currently, deep learning technology based on artificial neural networks has made tremendous progress in fields such as object classification, text processing, recommendation engines, image search, facial recognition, age and voice recognition, human-computer interaction, and affective computing. With the deepening of artificial neural network structures and the improvement of algorithms, deep learning technology has achieved breakthroughs in the field of human-like data perception. Deep learning technology can be used to describe image content, identify objects in complex environments within images, and perform speech recognition in noisy environments. Deep learning technology can also solve problems such as image generation and fusion. Summary of the Invention

[0003] At least one embodiment of the present disclosure provides an image processing method, comprising: obtaining an input image; and processing the input image using a generative network to generate an output image; wherein the generative network comprises a first subnetwork and at least one second subnetwork, and processing the input image using the generative network to generate the output image comprises: processing the input image using the first subnetwork to obtain a plurality of first feature maps; performing branching processing and weight sharing processing on the plurality of first feature maps using the at least one second subnetwork to obtain a plurality of second feature maps; and processing the plurality of second feature maps to obtain an output image.

[0004] For example, in the image processing method provided in some embodiments of the present disclosure, each second sub-network includes a first branch network, a second branch network, and a third branch network, and the branch processing of each second sub-network includes: dividing the input of each second sub-network into a first branch input, a second branch input, and a third branch input; and using the first branch network to process the first branch input to obtain a first branch output, using the second branch network to process the second branch input to obtain a second branch output, and using the third branch network to process the third branch input to obtain a third branch output; wherein, the at least one second sub-network includes a first second sub-network, the first second sub-network is connected to the first sub-network, and the multiple first feature maps serve as inputs of the first second sub-network.

[0005] For example, in the image processing method provided in some embodiments of the present disclosure, each second sub-network also includes a first backbone network, and the weight sharing processing of each second sub-network includes: connecting the first branch output, the second branch output and the third branch output to obtain a first intermediate output; and using the first backbone network to process the first intermediate output to obtain the output of each second sub-network.

[0006] For example, in the image processing method provided in some embodiments of the present disclosure, the processing of the first branch network includes standard convolution processing, the processing of the second branch network includes standard convolution processing, the processing of the third branch network includes standard convolution processing, and the processing of the first trunk network includes standard convolution processing and downsampling processing.

[0007] For example, in the image processing method provided in some embodiments of the present disclosure, the generation network also includes a third sub-network, which processes the multiple second feature maps to obtain the output image, including: processing the multiple second feature maps to obtain multiple third feature maps; using the third sub-network to process the multiple third feature maps to obtain multiple fourth feature maps; and synthesizing the multiple fourth feature maps to obtain the output image.

[0008] For example, in the image processing method provided in some embodiments of the present disclosure, the third subnetwork includes a second trunk network, a fourth branch network, a fifth branch network and a sixth branch network, and the third subnetwork is used to process the multiple third feature maps to obtain the multiple fourth feature maps, including: using the second trunk network to process the multiple third feature maps to obtain multiple fifth feature maps; dividing the multiple fifth feature maps into fourth branch input, fifth branch input and sixth branch input; and using the fourth branch network to process the fourth branch input to obtain the fourth feature map corresponding to the fourth branch network, using the fifth branch network to process the fifth branch input to obtain the fourth feature map corresponding to the fifth branch network, and using the sixth branch network to process the sixth branch input to obtain the fourth feature map corresponding to the sixth branch network; wherein the multiple fourth feature maps include the fourth feature map corresponding to the fourth branch network, the fourth feature map corresponding to the fifth branch network and the fourth feature map corresponding to the sixth branch network.

[0009] For example, in the image processing method provided in some embodiments of the present disclosure, the processing of the second trunk network includes upsampling processing, the processing of the fourth branch network includes standard convolution processing, the processing of the fifth branch network includes standard convolution processing, and the processing of the sixth branch network includes standard convolution processing.

[0010] For example, in the image processing method provided in some embodiments of the present disclosure, the processing of the fourth branch network also includes upsampling processing, the processing of the fifth branch network also includes upsampling processing, and the processing of the sixth branch network also includes upsampling processing.

[0011] For example, in the image processing method provided in some embodiments of the present disclosure, the processing of the first sub-network includes standard convolution processing, and the input image is processed using the first sub-network to obtain the multiple first feature maps, including: using the first sub-network to perform standard convolution processing on the input image to obtain the multiple first feature maps.

[0012] For example, in the image processing method provided in some embodiments of the present disclosure, the input image has a first color channel, a second color channel and a third color channel, the first sub-network includes a conversion module, a seventh branch network, an eighth branch network, a ninth branch network and a third backbone network, and the input image is processed using the first sub-network to obtain the multiple first feature maps, including: using the conversion module to convert the data information of the first color channel, the second color channel and the third color channel of the input image into the data information of the first luminance channel, the first color difference channel and the second color difference channel of the intermediate input image; using the seventh branch network to process the data information of the first luminance channel of the intermediate input image to obtain a seventh branch output, using the eighth branch network to process the data information of the first color difference channel of the intermediate input image to obtain an eighth branch output, and using the ninth branch network to process the data information of the second color difference channel of the intermediate input image to obtain a ninth branch output; connecting the seventh branch output, the eighth branch output and the ninth branch output to obtain a second intermediate output; and using the third backbone network to process the second intermediate output to obtain the multiple first feature maps.

[0013] For example, in the image processing method provided in some embodiments of the present disclosure, the processing of the seventh branch network includes standard convolution processing and downsampling processing, the processing of the eighth branch network includes standard downsampling processing, and the processing of the ninth branch network includes standard downsampling processing.

[0014] For example, in the image processing method provided in some embodiments of the present disclosure, the processing of the fourth branch network includes standard convolution processing and upsampling processing, the processing of the fifth branch network includes standard convolution processing and standard upsampling processing, and the processing of the sixth branch network includes standard convolution processing and standard upsampling processing.

[0015] For example, in the image processing method provided in some embodiments of the present disclosure, the generation network also includes a dense subnetwork, and the dense subnetwork includes N dense modules, which process the multiple second feature maps to obtain the multiple third feature maps, including: using the dense subnetwork to process the multiple second feature maps to obtain the multiple third feature maps; wherein, the multiple second feature maps are used as inputs of the first dense module among the N dense modules, and the multiple second feature maps are connected to the outputs of the i-1 dense modules before the i-th dense module among the N dense modules as inputs of the i-th dense module, and the multiple second feature maps are connected to the outputs of each of the dense modules as the multiple third feature maps, N and i are integers, N≥2, i≥2 and i≤N.

[0016] For example, in the image processing method provided in some embodiments of the present disclosure, the processing of each dense module includes dimensionality reduction processing and convolution processing.

[0017] For example, in the image processing method provided in some embodiments of the present disclosure, the generation network also includes a synthesis module, which performs synthesis processing on the multiple fourth feature maps to obtain the output image, including: using the synthesis module to perform synthesis processing on the multiple fourth feature maps to obtain the output image.

[0018] For example, in the image processing method provided in some embodiments of the present disclosure, the synthesis module includes a first conversion matrix, and the synthesis module is used to perform synthesis processing on the multiple fourth feature maps to obtain the output image, including: using the first conversion matrix to convert the data information of the fourth feature map corresponding to the fourth branch network, the data information of the fourth feature map corresponding to the fifth branch network, and the data information of the fourth feature map corresponding to the sixth branch network into data information of the first color channel, data information of the second color channel, and data information of the third color channel of the output image to obtain the output image.

[0019] At least one embodiment of the present disclosure further provides a neural network training method, comprising: training a discriminant network based on a generative network to be trained; training the generative network to be trained based on the discriminant network; and alternately performing the above training processes to obtain the generative network in the image processing method provided by any embodiment of the present disclosure; wherein, training the generative network to be trained based on the discriminant network comprises: using the generative network to be trained to process a first training input image to generate a first training output image; based on the first training output image, calculating the system loss value of the generative network to be trained by a system loss function; and correcting the parameters of the generative network to be trained based on the system loss value.

[0020] For example, in the training method provided in some embodiments of the present disclosure, the system loss function includes a generative network adversarial loss function, and the system loss value includes a generative network adversarial loss value; the generative network adversarial loss function is expressed as:

[0021]

[0022] Among them, L G represents the generative network adversarial loss function, z1 represents the first training input image, P z1 (z1) represents the set of the first training input images, G(z1) represents the first training output image, D(G(z1)) represents the output of the discriminant network for the first training output image, It represents finding the expectation for the set of the first training input images to obtain the generative network adversarial loss value.

[0023] For example, in the training method provided in some embodiments of the present disclosure, the system loss function further includes a content loss function, and the system loss value further includes a content loss value;

[0024] Calculating a system loss value of the generator network to be trained based on the first training output image using a system loss function, including: extracting a first content feature map of the first training input image and a second content feature map of the first training output image using an analysis network, and calculating the content loss value of the generator network using the content loss function based on the first content feature map and the second content feature map.

[0025] Wherein, the analysis network comprises at least one convolution module for extracting the first content feature map and the second content feature map;

[0026] The content loss function is expressed as:

[0027]

[0028] Among them, L content Denotes the content loss function, C m represents the single-layer content loss function of the mth convolutional module in the at least one convolutional module, w 1m Indicates C m The weight of

[0029] The single-layer content loss function is expressed as:

[0030]

[0031] Where S1 is a constant, represents the value of the j-th position in the first content feature map of the first training input image extracted by the i-th convolution kernel in the m-th convolution module, Represents the value of the j-th position in the second content feature map of the first training output image extracted by the i-th convolution kernel in the m-th convolution module.

[0032] For example, in the training method provided in some embodiments of the present disclosure, the system loss function also includes a color loss function, and the system loss value also includes a color loss value; the color loss function is expressed as:

[0033] L color =abs(gaussian(G(z1))-gaussian(I1))

[0034] Among them, L color represents the color loss function, G(z1) represents the first training output image, I1 represents the second training input image, gaussian() represents Gaussian blur operation, and abs() represents absolute value operation;

[0035] The quality of the second training input image is higher than that of the first training input image.

[0036] For example, in the training method provided in some embodiments of the present disclosure, the first training output image has a first color channel, a second color channel, and a third color channel;

[0037] The system loss function also includes a contrast loss function, and the system loss value also includes a contrast loss value; the contrast loss function is expressed as:

[0038] L L1 =0.299*abs(F G(z1) -F I2 )+0.587*abs(S G(z1) -S I2 )+0.299*abs(T G(z1) -T I2 )

[0039] Among them, L L1 represents the contrast loss function, G(z1) represents the first training output image, I2 represents the third training input image, F G(z1) 、S G(z1) and T G(z1) Respectively represent the data information of the first color channel, the second color channel and the third color channel of the first training output image, F I2 、S I2 and T I2Respectively represent the data information of the first color channel, the second color channel and the third color channel of the third training input image, and abs() represents an absolute value operation;

[0040] The third training input image has the same scene as the first training input image, and the quality of the third training input image is higher than that of the first training input image.

[0041] For example, in the training method provided in some embodiments of the present disclosure, the discriminant network is trained based on the generative network to be trained, including: using the generative network to be trained to process the fourth training input image to generate a second training output image; based on the second training output image and the fifth training input image, calculating the discriminant network adversarial loss value through the discriminant network adversarial loss function; and correcting the parameters of the discriminant network according to the discriminant network adversarial loss value; wherein the quality of the fifth training input image is higher than the quality of the fourth training input image.

[0042] For example, in the training method provided in some embodiments of the present disclosure, the discriminant network adversarial loss function is expressed as:

[0043]

[0044] Among them, L D represents the discriminant network adversarial loss function, x represents the fifth training input image, P data (x) represents the set of the fifth training input images, D(x) represents the output of the discriminant network for the fifth training input image, represents the expectation for the set of the fifth training input images, z2 represents the fourth training input image, P z2 (z2) represents the set of the fourth training input images, G(z2) represents the second training output image, D(G(z2)) represents the output of the discriminant network for the second training output image, It represents finding the expectation for the set of the fourth training input images.

[0045] At least one embodiment of the present disclosure further provides an image processing device, comprising: a memory for non-temporarily storing computer-readable instructions; and a processor for executing the computer-readable instructions, wherein the computer-readable instructions, when executed by the processor, execute the image processing method provided by any embodiment of the present disclosure.

[0046] At least one embodiment of the present disclosure further provides a storage medium that non-temporarily stores computer-readable instructions. When the computer-readable instructions are executed by a computer, the image processing method provided by any embodiment of the present disclosure can be executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure.

[0048] Figure 1 A schematic diagram of a convolutional neural network;

[0049] Figure 2A A schematic diagram of the structure of a convolutional neural network;

[0050] Figure 2B A schematic diagram of the working process of a convolutional neural network;

[0051] Figure 3 This is a schematic diagram of the structure of another convolutional neural network;

[0052] Figure 4 A flowchart of an image processing method provided by at least one embodiment of the present disclosure;

[0053] Figure 5 For a corresponding Figure 4 An exemplary flow chart of step S200 is shown in FIG.

[0054] Figure 6A A method corresponding to at least one embodiment of the present disclosure is provided Figure 4 A schematic block diagram of the architecture of the generative network of the image processing method shown in ;

[0055] Figure 6B Another embodiment of the present disclosure provides a method corresponding to Figure 4 A schematic block diagram of the architecture of the generative network of the image processing method shown in ;

[0056] Figure 7 A schematic diagram of the structure of a dense sub-network provided by at least one embodiment of the present disclosure;

[0057] Figure 8A A schematic diagram of an upsampling layer provided in at least one embodiment of the present disclosure;

[0058] Figure 8B A schematic diagram of another upsampling layer provided in at least one embodiment of the present disclosure;

[0059] Figure 9A is a schematic diagram of an input image;

[0060] Figure 9B Based on Figure 6A The generative network shown is Figure 9ASchematic diagram of an output image obtained by processing the input image shown;

[0061] Figure 9C Based on Figure 6B The generative network shown is Figure 9A Schematic diagram of an output image obtained by processing the input image shown;

[0062] Figure 10 A flowchart of a neural network training method provided in at least one embodiment of the present disclosure;

[0063] Figure 11A A method corresponding to at least one embodiment of the present disclosure is provided Figure 10 A schematic block diagram of the architecture of a generative network to be trained using the training method shown in FIG.

[0064] Figure 11B A schematic flowchart of a process for training a generative network to be trained provided in at least one embodiment of the present disclosure;

[0065] Figure 12 A schematic diagram of the structure of a discriminant network provided in at least one embodiment of the present disclosure;

[0066] Figure 13 A schematic diagram of the structure of an analysis network provided by at least one embodiment of the present disclosure;

[0067] Figure 14A A method corresponding to at least one embodiment of the present disclosure is provided Figure 10 Schematic architecture diagram of training the discriminant network using the training method shown in ;

[0068] Figure 14B A schematic flowchart of a process for training a discriminant network according to at least one embodiment of the present disclosure;

[0069] Figure 15 A schematic block diagram of an image processing device provided for at least one embodiment of the present disclosure; and

[0070] Figure 16 A schematic diagram of a storage medium provided in at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0071] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0072] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by persons of ordinary skill in the field to which this disclosure belongs. The words "first", "second" and similar terms used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0073] The present disclosure is described below through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of known functions and known components. When any component of an embodiment of the present disclosure appears in more than one figure, the component is represented by the same or similar reference numeral in each figure.

[0074] Image enhancement is a research hotspot in image processing. Due to various physical limitations during image acquisition (e.g., the small size of mobile phone camera image sensors and other software and hardware limitations) and interference from environmental noise, image quality can be significantly degraded. The goal of image enhancement is to improve the image's grayscale histogram and contrast through image enhancement techniques, thereby highlighting image details and enhancing the visual quality.

[0075] Image enhancement using deep neural networks is an emerging technology driven by the development of deep learning. For example, convolutional neural networks can process low-quality photos (input images) taken with mobile phones to produce high-quality output images, approaching the quality of photos taken with digital single-lens reflex cameras (DSLRs). For example, the Peak Signal-to-Noise Ratio (PSNR) is often used to characterize image quality, with higher PSNR values ​​indicating an image closer to that of a photo taken with a DSLR.

[0076] For example, Andrey Ignatov et al. proposed a method for image enhancement using a convolutional neural network. Please refer to the document, Andrey Ignatov, Nikolay Kobyshev, Kenneth Vanhoey, Radu Timofte, Luc VanGool, DSLR-Quality Photos on Mobile Devices with Deep ConvolutionalNetworks.arXiv:1704.02470v2[cs.CV], September 5, 2017. The entire text of the document is hereby incorporated by reference as part of this application. The method mainly uses convolutional layers, batch normalization layers and residual connections to construct a single-scale convolutional neural network, which can be used to process low-quality input images (for example, low contrast, underexposed or overexposed images, the entire image is too dark or too bright, etc.) into a higher-quality image. Using color loss, texture loss and content loss as loss functions in training can achieve better processing results.

[0077] Initially, convolutional neural networks (CNNs) were primarily used to recognize two-dimensional shapes, demonstrating their high invariance to translation, scaling, tilting, and other forms of image deformation. CNNs primarily simplify neural network models and reduce the number of weights through local receptive fields and weight sharing. With the advancement of deep learning technology, CNNs have expanded their application beyond image recognition and are now also used in face recognition, text recognition, animal classification, and image processing.

[0078] Figure 1 A schematic diagram of a convolutional neural network is shown. For example, the convolutional neural network can be used for image processing, using images as input and output and replacing scalar weights with convolution kernels. Figure 1 Only a convolutional neural network with a three-layer structure is shown in FIG, and the embodiments of the present disclosure are not limited to this. Figure 1 As shown, the convolutional neural network includes an input layer 101, a hidden layer 102, and an output layer 103. The input layer 101 has 4 inputs, the hidden layer 102 has 3 outputs, and the output layer 103 has 2 outputs. Finally, the convolutional neural network outputs 2 images.

[0079] For example, the four inputs of the input layer 101 may be four images, or four feature images of one image. The three outputs of the hidden layer 102 may be feature images of the image input through the input layer 101 .

[0080] For example, Figure 1As shown, the convolutional layer has weights and bias Weight Represents the convolution kernel, bias is a scalar added to the output of the convolutional layer, where k is the label representing the input layer 101, i and j are the labels of the units of the input layer 101 and the units of the hidden layer 102, respectively. For example, the first convolutional layer 201 includes a first set of convolution kernels ( Figure 1 in ) and the first set of biases ( Figure 1 in The second convolutional layer 202 includes a second set of convolution kernels ( Figure 1 in ) and the second set of biases ( Figure 1 in ). Typically, each convolutional layer includes dozens or hundreds of convolution kernels. If the convolutional neural network is a deep convolutional neural network, it may include at least five convolutional layers.

[0081] For example, Figure 1 As shown, the convolutional neural network also includes a first activation layer 203 and a second activation layer 204. The first activation layer 203 is located after the first convolutional layer 201, and the second activation layer 204 is located after the second convolutional layer 202. The activation layer (for example, the first activation layer 203 and the second activation layer 204) includes an activation function, which is used to introduce nonlinear factors into the convolutional neural network so that the convolutional neural network can better solve more complex problems. The activation function can include a rectified linear unit (ReLU) function, a sigmoid function, or a hyperbolic tangent function (tanh function). The ReLU function is a non-saturated nonlinear function, while the Sigmoid function and the tanh function are saturated nonlinear functions. For example, the activation layer can be a separate layer of the convolutional neural network, or the activation layer can also be included in the convolutional layer (for example, the first convolutional layer 201 can include the first activation layer 203, and the second convolutional layer 202 can include the second activation layer 204).

[0082] For example, in the first convolutional layer 201, first, several convolution kernels in the first set of convolution kernels are applied to each input. and several biases in the first set of biases To obtain the output of the first convolution layer 201; then, the output of the first convolution layer 201 can be processed by the first activation layer 203 to obtain the output of the first activation layer 203. In the second convolution layer 202, first, several convolution kernels in the second set of convolution kernels are applied to the output of the first activation layer 203. and several biases in the second set of biases Then, the output of the second convolution layer 202 can be processed by the second activation layer 204 to obtain the output of the second activation layer 204. For example, the output of the first convolution layer 201 can be the convolution kernel applied to its input. Then with bias As a result of the addition, the output of the second convolutional layer 202 can be the convolution kernel applied to the output of the first activation layer 203. Then with bias The result of addition.

[0083] Before using a convolutional neural network for image processing, it must be trained. After training, the convolutional neural network's convolution kernels and biases remain unchanged during image processing. During training, each convolution kernel and bias are adjusted using multiple sets of input / output example images and an optimization algorithm to obtain an optimized convolutional neural network model.

[0084] Figure 2A A schematic diagram of the structure of a convolutional neural network is shown. Figure 2B A schematic diagram of the working process of a convolutional neural network is shown. Figure 2A and 2B As shown in the figure, after the input image is input into the convolutional neural network through the input layer, it goes through several processing steps (such as Figure 2A The main components of a convolutional neural network may include multiple convolutional layers, multiple downsampling layers, and fully connected layers. In the present disclosure, it should be understood that each of these layers, such as multiple convolutional layers, multiple downsampling layers, and fully connected layers, refers to a corresponding processing operation, i.e., convolution processing, downsampling processing, fully connected processing, etc. The described neural networks also refer to corresponding processing operations. The example normalization layer or layer normalization layer to be described below is similar to this and will not be repeated here. For example, a complete convolutional neural network can be composed of the superposition of these three layers. For example, Figure 2A Only three levels of a convolutional neural network are shown, namely the first level, the second level, and the third level. For example, each level may include a convolution module and a downsampling layer. For example, each convolution module may include a convolution layer. Thus, the processing process of each level may include: convolution processing and downsampling (sub-sampling / down-sampling) processing of the input image. For example, according to actual needs, each convolution module may also include an instance normalization layer, so that the processing process of each level may also include normalization processing.

[0085] For example, the instance normalization layer is used to perform instance normalization on the feature images output by the convolutional layer, so that the grayscale values ​​of the pixels in the feature images vary within a predetermined range, thereby simplifying the image generation process and improving the image enhancement effect. For example, the predetermined range can be [-1, 1]. The instance normalization layer performs instance normalization on each feature image based on its own mean and variance. For example, the instance normalization layer can also be used to perform instance normalization on a single image.

[0086] For example, assuming that the size of the mini-batch gradient decent is T, the number of feature images output by a convolutional layer is C, and each feature image is a matrix with H rows and W columns, then the model representation of the feature image is (T, C, W, H). Therefore, the instance normalization formula of the instance normalization layer can be expressed as follows:

[0087]

[0088] Among them, x tijk y is the value of the t-th feature block (patch), the i-th feature image, the j-th row, and the k-th column in the feature image set output by the convolutional layer. tijk Represents x processed by the instance normalization layer tijk ε1 is a very small integer to avoid the denominator being 0.

[0089] The convolution layer is the core layer of the convolutional neural network. In the convolution layer of the convolutional neural network, a neuron is only connected to some neurons in the adjacent layers. The convolution layer can apply several convolution kernels (also called filters) to the input image to extract multiple types of features of the input image. Each convolution kernel can extract one type of feature. The convolution kernel is generally initialized in the form of a random decimal matrix. During the training process of the convolutional neural network, the convolution kernel will be learned to obtain reasonable weights. The result obtained after applying a convolution kernel to the input image is called a feature image (feature map), and the number of feature images is equal to the number of convolution kernels. The feature image output by a convolution layer of one level can be input to the convolution layer of the next adjacent level and processed again to obtain a new feature image. For example, Figure 2A As shown, the convolution layer of the first level can output a first feature image, which is input to the convolution layer of the second level and processed again to obtain a second feature image.

[0090] For example, Figure 2B As shown in the figure, the convolution layer can use different convolution kernels to convolve the data of a local receptive field of the input image. The convolution result is input into the activation layer, which calculates according to the corresponding activation function to obtain the feature information of the input image.

[0091] For example, Figure 2A and 2B As shown, the downsampling layer is placed between adjacent convolutional layers and is a form of downsampling. On the one hand, it can reduce the size of the input image, simplifying computational complexity and, to a certain extent, minimizing overfitting. On the other hand, it can also perform feature compression to extract key features from the input image. The downsampling layer reduces the size of the feature image without changing the number of feature images. For example, a 12×12 input image is downsampled by a 6×6 convolution kernel, resulting in a 2×2 output image. This means that 36 pixels in the input image are merged into one pixel in the output image. The final downsampling layer or convolutional layer can be connected to one or more fully connected layers, which connect all extracted features. The output of a fully connected layer is a one-dimensional matrix, or vector.

[0092] Figure 3 shows a schematic diagram of the structure of another convolutional neural network. For example, see Figure 3 In the example shown, the output of the last convolutional layer (i.e., the t-th convolutional layer) is input to the flattening layer for flattening. The flattening layer can convert the feature image (2D image) into a vector (1D). The flattening operation can be performed as follows:

[0093] v k =f k / j,k％j

[0094] Here, v is a vector containing k elements, and f is a matrix with i rows and j columns.

[0095] The output of the flattening layer (i.e., a 1D vector) is then input into a fully connected layer (FCN). A fully connected layer can have the same structure as a convolutional neural network, but the difference is that it uses different scalar values ​​instead of convolution kernels.

[0096] For example, the output of the last convolutional layer can also be input to the average layer (AVG). The average layer is used to average the output, that is, to represent the output image using the mean of the feature image. Therefore, a 2D feature image is converted into a scalar. For example, if the convolutional neural network includes an average layer, it may not include a flattening layer.

[0097] For example, according to actual needs, the averaging layer or the fully connected layer can be connected to the classifier, and the classifier can perform classification based on the extracted features. The output of the classifier can be used as the final output of the convolutional neural network, that is, the category label representing the image category.

[0098] For example, the classifier may be a support vector machine (SVM) classifier, a softmax classifier, a k-nearest neighbor (KNN) classifier, etc. Figure 3 As shown, in one example, the convolutional neural network includes a softmax classifier, which is a generator of a logical function that can compress a K-dimensional vector z containing arbitrary real numbers into a K-dimensional vector σ(z). The formula of the softmax classifier is as follows:

[0099]

[0100] Among them, Z j Represents the jth element in the K-dimensional vector z, σ(z) represents the predicted probability of each class label, σ(z) is a real number in the range (0,1), and the sum of the K-dimensional vector σ(z) is 1. According to the above formula, each class label in the K-dimensional vector z is assigned a certain prediction probability, and the class label with the largest prediction probability is selected as the label or category of the input image.

[0101] At least one embodiment of the present disclosure provides an image processing method. The image processing method includes: obtaining an input image; and processing the input image using a generative network to generate an output image; wherein the generative network includes a first subnetwork and at least one second subnetwork, and processing the input image using the generative network to generate the output image includes: processing the input image using the first subnetwork to obtain multiple first feature maps; performing branching processing and weight sharing processing on the multiple first feature maps using the at least one second subnetwork to obtain multiple second feature maps; and processing the multiple second feature maps to obtain the output image.

[0102] Some embodiments of the present disclosure also provide an image processing device, a neural network training method, and a storage medium corresponding to the above-mentioned image processing method.

[0103] The image processing method provided by at least one embodiment of the present disclosure combines branch processing and weight sharing processing to perform image enhancement processing, which can not only reduce the number of parameters but also facilitate the calculation of gradients during back propagation, thereby improving the processing speed and convergence speed while outputting high-quality images.

[0104] Some embodiments and examples of the present disclosure are described in detail below with reference to the accompanying drawings.

[0105] Figure 4 This is a flowchart of an image processing method provided by at least one embodiment of the present disclosure. Figure 4As shown, the image processing method includes steps S100 to S200.

[0106] Step S100: Acquire an input image.

[0107] For example, in step S100, the input image may include a photograph captured by a smartphone camera, a tablet camera, a personal computer camera, a digital camera lens, a surveillance camera, or a webcam, and may include images of people, animals, plants, or scenery, etc., although embodiments of the present disclosure are not limited thereto. For example, the input image may be a low-quality image, where the quality of the input image is lower than that of a photograph captured, for example, by a digital single-lens reflex camera. For example, the input image may be an RGB image comprising three channels, and embodiments of the present disclosure include but are not limited to this.

[0108] Step S200: Processing the input image using a generative network to generate an output image.

[0109] For example, in step S200, the generation network may perform image enhancement processing on the input image, so that the output image is a high-quality image. For example, the quality of the output image is close to that of a photo taken by a digital single-lens reflex camera.

[0110] Figure 5 For a corresponding Figure 4 The exemplary flow chart of step S200 shown in FIG. Figure 6A A method corresponding to at least one embodiment of the present disclosure is provided Figure 4 Schematic architecture block diagram of the generative network of the image processing method shown in, Figure 6B Another embodiment of the present disclosure provides a method corresponding to Figure 4 Schematic architectural block diagram of the generative network of the image processing method shown in .

[0111] Below, first combine Figure 6A The generative network shown is Figure 5 Step S200 shown is described in detail.

[0112] For example, Figure 5 As shown, the input image is processed using a generative network to generate an output image, that is, step S200, including step S210, step S220 and step S225.

[0113] Step S210: Use the first sub-network to process the input image to obtain multiple first feature maps.

[0114] For example, in some examples, such as Figure 6A As shown, the generated network may include a first sub-network N1. Figure 6AAs shown, the first sub-network N1 may include a convolution module CN, that is, the processing of the first sub-network N1 includes standard convolution processing, so that step S210 may include using the first sub-network N1 to perform standard convolution processing on the input image IN to obtain multiple first feature maps F1. It should be noted that in the embodiment of the present disclosure, the convolution module CN may include a convolution layer and an instance normalization layer, so that the standard convolution processing may include convolution processing and instance normalization processing. The following is similar to this and will not be repeated.

[0115] Step S220: Use at least one second sub-network to perform branching processing and weight sharing processing on the multiple first feature maps to obtain multiple second feature maps.

[0116] For example, in some examples, such as Figure 6A As shown, the generated network may include at least one second sub-network N2. Figure 6A As shown, in some embodiments, the generated network may include two second sub-networks N2, namely a first second sub-network N2 and a second second sub-network N2 (it should be noted that in Figure 6A The second sub-network N2 close to the first sub-network N1 is the first second sub-network), the first second sub-network N2 is connected to the first sub-network N1, and the first second sub-network N2 is connected to the second second sub-network N2. Thus, two second sub-networks N2 can be used to process multiple first feature maps F1. For example, Figure 6A As shown, multiple first feature maps F1 can be used as input of the first second sub-network N2, and the output of the first second sub-network N2 can be used as input of the second second sub-network N2. The output of the second second sub-network N2 is multiple second feature maps F2.

[0117] It should be noted that, in this disclosure, "connecting" two subnetworks may mean that the output of the earlier of the two subnetworks, in the direction of signal transmission, is used as the input of the later of the two subnetworks. For example, "connecting the first subnetwork N2 to the first subnetwork N1" may mean that the output of the first subnetwork N1 is used as the input of the first subnetwork N2.

[0118] For example, in some examples, such as Figure 6A As shown, each second sub-network N2 may include a first branch network N21, a second branch network N22, and a third branch network N23. Thus, the branch processing of each second sub-network N2 may include: converting the input of each second sub-network (such as Figure 6AThe first branch input B1 is processed by the first branch network N21 to obtain the first branch output O1, the second branch input B2 is processed by the second branch network N22 to obtain the second branch output O2, and the third branch input B3 is processed by the third branch network N23 to obtain the third branch output O3.

[0119] For example, in some embodiments of the present disclosure, the number of feature maps included in the input of each corresponding branch network may be the same, for example, the number of feature maps included in the first branch input B1, the number of feature maps included in the second branch input B2, and the number of feature maps included in the third branch input B3 are all the same. It should be noted that the embodiments of the present disclosure are not limited to this. For example, in some other embodiments of the present disclosure, the number of feature maps included in the input of each corresponding branch network may be different or not completely the same, for example, the number of feature maps included in the second branch input B2 and the number of feature maps included in the third branch input B3 are the same, but different from the number of feature maps included in the first branch input B1. It should be understood that corresponding branch networks refer to branch networks at the same level. For example, the first branch network N21, the second branch network N22, and the third branch network N23 form a group of corresponding branch networks. The fourth branch network N31, the fifth branch network N32, and the sixth branch network N33, which will be described below, form a group of corresponding branch networks. Furthermore, the seventh branch network N11, the eighth branch network N12, and the ninth branch network N13, which will be described below, also form a group of corresponding branch networks. Therefore, the requirements for the number of feature maps included in the inputs of the fourth branch network N31, the fifth branch network N32, and the sixth branch network N33, as well as the requirements for the number of feature maps included in the inputs of the seventh branch network N11, the eighth branch network N12, and the ninth branch network N13, can all refer to the requirements for the number of feature maps included in the inputs of the first branch network N21, the second branch network N22, and the third branch network N23, and will not be repeated here.

[0120] For example, the input of each second sub-network N2 may include 3n feature maps, where n is a positive integer. Thus, the 1st to nth feature maps may be divided into the first branch input B1, the n+1th to 2nth feature maps may be divided into the second branch input B2, and the 2n+1th to 3nth feature maps may be divided into the third branch input B3; alternatively, the 1st, 4th, 7th, ..., 3n-2th feature maps may be divided into the first branch input B1, the 2nd, 5th, 8th, ..., 3n-1th feature maps may be divided into the second branch input B2, and the 3rd, 6th, 9th, ..., 3nth feature maps may be divided into the third branch input B3. It should be noted that the embodiments of the present disclosure do not limit the specific division method.

[0121] For example, Figure 6A As shown, the first branch network N21, the second branch network N22, and the third branch network N23 can each include a convolution module CN, thereby performing standard convolution processing on the first branch input B1, the second branch input B2, and the third branch input B3, respectively, to obtain the corresponding first branch output O1, the second branch output O2, and the third branch output O3. For example, in some examples, the number of standard convolution processes performed on the first branch network N21, the second branch network N22, and the third branch network N23 can be the same; of course, the parameters of the standard convolution processes for the first branch network N21, the second branch network N22, and the third branch network N23 can be different. It should be noted that the embodiments of the present disclosure are not limited to this.

[0122] For example, in some examples, such as Figure 6A As shown, each second sub-network N2 may further include a first backbone network N20, so that the weight sharing process of each second sub-network N2 may include: connecting the first branch output O1, the second branch output O2 and the third branch output O3 to obtain a first intermediate output M1 (as shown in FIG. Figure 6A and processing the first intermediate output M1 using the first backbone network N20 to obtain the output of each second sub-network.

[0123] For example, taking the matrix of H rows and W columns of the feature maps included in the first branch output O1, the second branch output O2 and the third branch output O3 as an example, the number of feature maps included in the first branch output O1 is C1, the number of feature maps included in the second branch output O2 is C2, and the number of feature maps included in the third branch output O3 is C3, then the models of the first branch output O1, the second branch output O2 and the third branch output O3 are (C1, H, W), (C2, H, W) and (C3, H, W) respectively. Thus, the first branch output O1, the second branch output O2 and the third branch output O3 are connected, and the model of the first intermediate output M1 obtained is (C1+C2+C3, H, W). The number of feature maps included in the first intermediate output M1 is C1+C2+C3, and the present disclosure does not limit the arrangement order of each feature map in the model of the first intermediate output M1. It should be noted that the present disclosure includes but is not limited to this.

[0124] For example, Figure 6A As shown, the first backbone network N20 may include a convolution module CN and a downsampling layer DS, so that the first intermediate output M1 can be subjected to standard convolution processing and downsampling processing. It should be noted that the embodiments of the present disclosure do not limit the order of the convolution module CN and the downsampling layer DS in the first backbone network N20 (i.e., the order of the standard convolution processing and the downsampling processing).

[0125] Downsampling is used to reduce the size of the feature map, thereby reducing the amount of data in the feature map. For example, downsampling can be performed through a downsampling layer, but is not limited thereto. For example, the downsampling layer can implement downsampling using downsampling methods such as max pooling, average pooling, strided convolution, decimation (e.g., selecting fixed pixels), and demuxout (splitting the input image into multiple smaller images).

[0126] It should be noted that when the number of at least one second sub-network N2 is greater than or equal to 2, the downsampling processing methods and parameters in the first backbone network N20 of different second sub-networks N2 may be the same or different, and the embodiments of the present disclosure do not limit this.

[0127] It should be noted that Figure 6AThe number of second subnetworks N2 in the example is 2, and the embodiments of the present disclosure do not impose any specific limitation on the number of second subnetworks N2. For example, the number of second subnetworks N2 may also be 1, 3, etc. For example, the at least one second subnetwork may include a first second subnetwork, which is connected to the first subnetwork N1, and the multiple first feature maps F1 serve as inputs to the first second subnetwork. For another example, the at least one second subnetwork may include other second subnetworks in addition to the first second subnetwork, and each of the other second subnetworks uses the output of the previous second subnetwork connected thereto as input to the second subnetwork, and the output of the last second subnetwork is the multiple second feature maps F2.

[0128] Step S225: Process the multiple second feature maps to obtain an output image.

[0129] For example, specifically, Figure 5 As shown, step S225 may include steps S230 to S250.

[0130] Step S230: Process the multiple second feature maps to obtain multiple third feature maps.

[0131] For example, in some examples, such as Figure 6A As shown in , the generation network can also include a dense sub-network DenseNet. For example, Figure 6A As shown, in step S250, a dense sub-network DenseNet can be used to process the multiple second feature maps F2 to obtain multiple third feature maps F3.

[0132] Figure 7 A schematic diagram of a dense sub-network structure provided by at least one embodiment of the present disclosure. Figure 7 As shown, the dense sub-network DenseNet includes multiple dense modules DenseBlock, for example, the number of dense modules DenseBlock is N, where N≥2. It should be noted that, Figure 7 The number of dense modules DenseBlock in the dense sub-network DenseNet shown as N=4 is exemplary and should not be regarded as a limitation to the present disclosure.

[0133] For example, in some examples, combined with Figure 6A and Figure 7As shown, multiple second feature maps F2 can be used as inputs of the first dense module in N dense modules DenseBlock (for example, the first dense module is connected to the last second sub-network N2 in at least one second sub-network N2), and the multiple second feature maps F2 are also connected to the outputs of the i-1 dense modules before the i-th dense module in the N dense modules DenseBlock as inputs of the i-th dense module. The multiple second feature maps are also connected to the output of each dense module as multiple third feature maps F3. i is an integer, i≥2 and i≤N. It should be noted that the present disclosure includes but is not limited to this. For example, in other examples, only the output of each dense module can be connected as multiple third feature maps F3. For example, the connection operation here can refer to the connection operation in the aforementioned second sub-network, which will not be repeated here. For example, in some examples, such as Figure 7 As shown, each dense module DenseBlock can include a bottleneck layer B and a convolution layer Cv. Therefore, the processing of each dense module DenseBlock includes dimensionality reduction and convolution. For example, the bottleneck layer B can use a 1×1 convolution kernel to reduce the dimensionality of the data, reducing the number of feature maps, thereby reducing the number of parameters in the subsequent convolution processing, reducing the amount of calculation, and thus improving the processing speed.

[0134] For example, the dense sub-network DenseNet has the advantages of significantly reducing parameters, reducing computational complexity, effectively solving the gradient vanishing problem, supporting feature reuse and strengthening feature propagation, and having very good anti-overfitting performance.

[0135] Step S240: using the third sub-network to process the plurality of third feature maps to obtain a plurality of fourth feature maps.

[0136] For example, in some examples, such as Figure 6A As shown, the generated network may further include a third sub-network N3. Figure 6AAs shown, the third sub-network N3 may include a second backbone network N30, a fourth branch network N31, a fifth branch network N32 and a sixth branch network N33, so that the processing of the third sub-network N3 may include: using the second backbone network N30 to process multiple third feature maps F3 to obtain multiple fifth feature maps F5; dividing the multiple fifth feature maps F5 into a fourth branch input B4, a fifth branch input B5 and a sixth branch input B6; and, using the fourth branch network N31 to process the fourth branch input B4 to obtain a fourth feature map F4 corresponding to the fourth branch network N31, using the fifth branch network N32 to process the fifth branch input B5 to obtain a fourth feature map F4 corresponding to the fifth branch network N32, and using the sixth branch network N33 to process the sixth branch input B6 to obtain a fourth feature map F4 corresponding to the sixth branch network N33.

[0137] For example, the plurality of fourth feature maps F4 obtained in step S240 include a fourth feature map corresponding to the fourth branch network N31 , a fourth feature map corresponding to the fifth branch network N32 , and a fourth feature map corresponding to the sixth branch network N33 .

[0138] For example, in some examples, such as Figure 6A As shown, the second backbone network N30 may include an upsampling layer US, so that the input multiple third feature maps F3 can be upsampled to obtain multiple fifth feature maps F5.

[0139] Upsampling is used to increase the size of the feature map, thereby increasing the amount of data in the feature map. For example, upsampling can be performed by an upsampling layer, but is not limited thereto. For example, the upsampling layer can implement upsampling using upsampling methods such as strided transposed convolution and interpolation algorithms. Interpolation algorithms can include, for example, interpolation, bilinear interpolation, and bicubic interprolation.

[0140] It should be noted that interpolation algorithms can be used not only for upsampling but also for downsampling. For example, when using an interpolation algorithm for upsampling, the original pixel values ​​and interpolated values ​​can be retained, thereby increasing the size of the feature map; for example, when using an interpolation algorithm for downsampling, only the interpolated values ​​can be retained (the original pixel values ​​are removed), thereby reducing the size of the feature map.

[0141] Figure 8A A schematic diagram of an upsampling layer provided in at least one embodiment of the present disclosure is provided. Figure 8B A schematic diagram of another upsampling layer provided in at least one embodiment of the present disclosure.

[0142] For example, in some examples, such as Figure 8A As shown in Figure 1, the upsampling layer uses pixel interpolation to achieve upsampling. At this time, the upsampling layer can also be called a composite layer. The composite layer uses a 2×2 upsampling factor, so that the four input feature images (i.e., Figure 8A INPUT 4n, INPUT 4n+1, INPUT 4n+2, INPUT 4n+3) in are combined to obtain an output feature image with a fixed pixel order (i.e., Figure 8A OUTPUT n) in the .

[0143] For example, in some examples, for a two-dimensional feature image, the upsampling layer obtains a first number of input feature images and interleaves the pixel values ​​of these input feature images to produce the same first number of output feature images. Compared to the input feature images, the number of output feature images does not change, but the size of each output feature image increases by a corresponding multiple. Thus, the composite layer adds more data information through different permutations and combinations, and these combinations can give all possible upsampling combinations. Finally, a selection can be made from the upsampling combinations through the activation layer.

[0144] For example, in Figure 8B In the example shown, the upsampling layer uses the pixel value interleaving and rearrangement method to achieve upsampling. In this case, the upsampling layer can also be called a composite layer. The composite layer also uses a 2×2 upsampling factor, that is, every 4 input feature images (i.e., Figure 8B INPUT 4n, INPUT 4n+1, INPUT 4n+2, INPUT 4n+3) in the image are grouped together, and their pixel values ​​are interleaved to generate four output feature images (i.e., Figure 8B OUTPUT 4n, OUTPUT 4n+1, OUTPUT 4n+2, OUTPUT 4n+3 in FIG). The number of input feature images and the number of output feature images obtained after the composite layer processing are the same, and the size of each output feature image is increased to 4 times that of the input feature image, that is, the output feature image has 4 times the number of pixels as the input feature image.

[0145] For example, in some examples, such as Figure 6AAs shown, the fourth branch network N31, the fifth branch network N32, and the sixth branch network N33 can each include a convolution module CN, thereby performing standard convolution processing on the fourth branch input B4, the fifth branch input B5, and the sixth branch input B6, respectively. For example, in some examples, the number of standard convolution processing times of the fourth branch network N31, the fifth branch network N32, and the sixth branch network N33 can be the same; of course, the parameters of the standard convolution processing of the fourth branch network N31, the fifth branch network N32, and the sixth branch network N33 can be different. It should be noted that the embodiments of the present disclosure are not limited to this.

[0146] For example, in some examples, such as Figure 6A As shown, the fourth branch network N31, the fifth branch network N32, and the sixth branch network N33 can all include an upsampling layer US, so that the processing of the fourth branch network N31, the fifth branch network N32, and the sixth branch network N33 can also include upsampling processing. For example, in some examples, the number of upsampling processes of the fourth branch network N31, the fifth branch network N32, and the sixth branch network N33 can be the same; of course, the parameters of the upsampling processes of the fourth branch network N31, the fifth branch network N32, and the sixth branch network N33 can be different. It should be noted that the embodiments of the present disclosure are not limited to this.

[0147] It should be noted that the upsampling processing method in the second backbone network N30 may be the same as or different from the upsampling processing methods in the fourth branch network N31 , the fifth branch network N32 and the sixth branch network N33 , and the embodiments of the present disclosure do not limit this.

[0148] For example, in some examples, the number of fourth feature maps F4 corresponding to the fourth branch network N31 is 1, the number of fourth feature maps F4 corresponding to the fifth branch network N32 is 1, and the number of fourth feature maps F4 corresponding to the sixth branch network N33 is 1, that is, the multiple feature maps F4 include 3 feature maps.

[0149] Step S250: synthesizing the plurality of fourth feature maps to obtain an output image.

[0150] For example, in some examples, such as Figure 6A As shown in , the generation network can also include a synthesis module Merg. For example, Figure 6A As shown, in step S250, a synthesis module Merg may be used to process the plurality of fourth feature maps F4 to obtain an output image OUT.

[0151] For example, in some examples, the synthesis module Merg may include a first conversion matrix, and the first conversion module is used to convert the plurality of fourth feature maps F4 into the output image OUT. For example, in some examples, specifically, using the synthesis module Merg to process the plurality of fourth feature maps F4 to obtain the output image OUT may include: using the first conversion matrix, converting data information of the fourth feature map F4 corresponding to the fourth branch network N31, data information of the fourth feature map F4 corresponding to the fifth branch network N32, and data information of the fourth feature map F4 corresponding to the sixth branch network N33 into data information of the first color channel, data information of the second color channel, and data information of the third color channel of the output image OUT, to obtain the output image OUT.

[0152] For example, in some examples, the first color channel, the second color channel, and the third color channel may be red (R), green (G), and blue (B) channels, respectively, so that the output image OUT is an image in RGB format. It should be noted that the embodiments of the present disclosure include but are not limited to this.

[0153] For example, in some examples, the first conversion matrix can be used to convert an image in a YUV format into an image in an RGB format. For example, the conversion formula of the first conversion matrix can be expressed as follows:

[0154]

[0155] Among them, Y, U, and V respectively represent the brightness information (i.e., the data information of the first brightness channel), the first chrominance information (i.e., the data information of the first color difference channel), and the second chrominance information (i.e., the data information of the second color difference channel) of the YUV format image, and R, G, and B respectively represent the red information (i.e., the data information of the first color channel), green information (i.e., the data information of the second color channel), and blue information (i.e., the data information of the third color channel) of the converted RGB format image.

[0156] It should be noted that when using Figure 6A Before the generative network shown executes the image processing method provided by the embodiment of the present disclosure, the generative network may be trained. For example, during the training process, the parameters of the first transformation matrix are fixed. For example, Figure 6A After the generation network shown is trained, the data information of the fourth feature map F4 output by the fourth branch network N31, the data information of the fourth feature map F4 output by the fifth branch network N32, and the data information of the fourth feature map F4 output by the sixth branch network N33 respectively correspond to the data information of the first brightness channel, the data information of the first color difference channel, and the data information of the second color difference channel, so that after conversion through the first conversion matrix, the output image OUT in RGB format can be obtained.

[0157] For example, the output image OUT retains the content of the input image IN, but is a high-quality image. For example, the quality of the output image OUT may be close to the quality of a photo taken with a digital single-lens reflex camera.

[0158] For example, in some examples, the number of the fourth feature map F4 output by the fourth branch network N31, the fourth feature map F4 output by the fifth branch network N32, and the fourth feature map F4 output by the sixth branch network N33 are all 1, that is, the multiple feature maps F4 include 3 feature maps (corresponding to the first brightness channel, the first color difference channel, and the second color difference channel, respectively), and the first conversion matrix can convert the 3 feature maps into an output image in RGB format.

[0159] For example, since the fourth branch network N31, the fifth branch network N32 and the sixth branch network N33 all include standard convolution processing (standard convolution processing includes convolution processing and instance normalization processing), the range of grayscale values ​​of pixels of multiple fourth feature maps F4 can be, for example, [-1, 1]. Therefore, the processing of the synthesis module Merg can also include: converting the grayscale values ​​of the pixels of the output image OUT to, for example, within the range of [0, 255].

[0160] In the YUV format, Y represents brightness, and U and V represent chrominance. U and V are the two components that make up color. In the YUV color space, the first brightness channel (i.e., the Y channel) and the first color difference channel (i.e., the U channel) and the second color difference channel (i.e., the V channel) are separate. For example, YUV formats include YUV444, YUV420, and YUV422. The main difference between YUV444, YUV420, and YUV422 formats lies in the sampling and storage methods of the U and V channel data.

[0161] For example, the YUV444 format indicates that in each row of pixels, both chrominance information (ie, first chrominance information U and second chrominance information V) are complete, that is, both chrominance information are stored based on complete sampling.

[0162] Assume that the four pixels in an image are represented as follows:

[0163] [Y0U0V0][Y1U1V1][Y2U2V2][Y3U3V3]

[0164] During image processing, the data flow for storing or processing the four pixels is:

[0165] Y0U0V0Y1U1V1Y2U2V2Y3U3V3.

[0166] The mapped pixels are represented as follows:

[0167] [Y0U0V0][Y1U1V1][Y2U2V2][Y3U3V3]

[0168] That is, the mapped pixel points are the original pixel points.

[0169] For example, the YUV420 format indicates that each row of pixels contains only one type of chromaticity information (first chromaticity information U or second chromaticity information V), and the first chromaticity information U or the second chromaticity information V is sampled and stored at a 1 / 2 frequency. During image processing, adjacent rows process different chromaticity information.

[0170] Assume that two rows of 8 pixels in an image are represented as:

[0171] [Y0U0V0][Y1U1V1][Y2U2V2][Y3U3V3]

[0172] [Y4U4V4][Y5U5V5][Y6U6V6][Y7U7V7]

[0173] During image processing, the data flow for storing or processing the 8 pixels is:

[0174] Y0U0Y1Y2U2Y3

[0175] Y4V4Y5Y6V6Y7

[0176] In the first row of pixels, there is only first chromaticity information U; in the second row of pixels, there is only second chromaticity information V.

[0177] The mapped pixel points are expressed as:

[0178] [Y0U0V4][Y1U0V4][Y2U2V6][Y3U2V6]

[0179] [Y4U0V4][Y5U0V4][Y6U2V7][Y7U2V6]

[0180] In summary, four adjacent pixels in each row only take up six bytes when stored or processed. Compared to the YUV444 sampling format (which requires 12 bytes for four pixels), the YUV420 format reduces the amount of pixel data processed and stored. Although the mapped pixels are slightly different from the original pixels, these differences do not cause noticeable changes in the human eye.

[0181] For example, in some examples, Figure 6A In the illustrated generation network, the plurality of feature maps F4 may have an image format of YUV444. It should be noted that the embodiments of the present disclosure include but are not limited to the above.

[0182] Figure 6B The generative network shown is similar to Figure 6A The difference between the generated networks shown is mainly in the first sub-network N1 and the third sub-network N3. Figure 6B Other constructions of the generative network shown are similar to Figure 6A The generative networks shown are essentially the same and will not be described again where they are repeated.

[0183] The following combination Figure 6B The generative network shown is similar to Figure 6A The differences in the generative networks shown, and the corresponding Figure 5 The steps in the process shown are explained in detail.

[0184] For example, in some examples, the input image has a first color channel, a second color channel, and a third color channel. For example, the first color channel, the second color channel, and the third color channel can be red (R), green (G), and blue (B) channels, respectively. The embodiments of the present disclosure include but are not limited to this. For example, accordingly, in Figure 6B In the generated network shown, the first sub-network N1 may include a conversion module Tran, a seventh branch network N11, an eighth branch network N12, a ninth branch network N13 and a third trunk network N10, so step S210 may include the following steps S211 to S214.

[0185] Step S211: using the conversion module Tran to convert the data information of the first color channel, the second color channel and the third color channel of the input image IN into the data information of the first luminance channel, the first color difference channel and the second color difference channel of the intermediate input image MIN.

[0186] For example, in some examples, the conversion module Tran may include a second conversion matrix, which is used to convert the input image IN into the intermediate input image MIN. For example, in some examples, the second conversion matrix may be used to convert an image in RGB format into an image in YUV format. For example, the conversion formula of the second conversion matrix may be expressed as follows:

[0187]

[0188] Among them, R, G and B respectively represent the red information (i.e., the data information of the first color channel), green information (i.e., the data information of the second color channel) and blue information (i.e., the data information of the third color channel) of the RGB format image, and Y, U, and V respectively represent the brightness information (i.e., the data information of the first brightness channel), the first chrominance information (i.e., the data information of the first color difference channel) and the second chrominance information (i.e., the data information of the second color difference channel) of the converted YUV format image.

[0189] For example, in some examples, the input image IN has an RGB format, and the intermediate input image MIN has a YUV420 format, for example, thereby reducing the size of the U channel and the V channel, thereby reducing the number of convolution kernels in the generation network. It should be noted that this embodiment includes but is not limited to this.

[0190] Step S212: Use the seventh branch network to process the data information of the first luminance channel of the intermediate input image to obtain the seventh branch output, use the eighth branch network to process the data information of the first color difference channel of the intermediate input image to obtain the eighth branch output, and use the ninth branch network to process the data information of the second color difference channel of the intermediate input image to obtain the ninth branch output.

[0191] For example, in some examples, such as Figure 6B As shown, the data information of the first luminance channel, the first color difference channel, and the second color difference channel of the intermediate input image MIN are respectively used as the seventh branch input B7, the eighth branch input B8, and the ninth branch input B9, and are respectively processed by the seventh branch network N11, the eighth branch network N12, and the ninth branch network N13 to obtain the seventh branch output O7, the eighth branch output O8, and the ninth branch output O9.

[0192] For example, Figure 6B As shown, the seventh branch network N11 can include a convolution module CN and a downsampling layer DS, so that the seventh branch input B7 can be subjected to standard convolution processing and downsampling processing; the eighth branch network N12 and the ninth branch network N13 can both include a standard downsampling layer SDS, so that the eighth branch input B8 and the ninth branch input B9 can be subjected to standard downsampling processing, respectively.

[0193] For example, the standard downsampling layer may use an interpolation algorithm such as interpolation, bilinear interpolation, or bicubic interpolation to perform standard downsampling processing. For example, when using an interpolation algorithm to perform standard downsampling processing, only the interpolated value may be retained (the original pixel value may be removed), thereby reducing the size of the feature map.

[0194] For example, in some examples, the standard downsampling processing methods in the eighth branch network N12 and the ninth branch network N13 may be the same, but their parameters may be different. It should be noted that the embodiments of the present disclosure include but are not limited to the above.

[0195] For example, compared with the seventh branch network N11, the eighth branch network N12 is equivalent to omitting the convolution module for processing the highest resolution of the U channel, and the ninth branch network N13 is equivalent to omitting the convolution module for processing the highest resolution of the V channel, thereby improving the processing speed.

[0196] Step S213: Connect the seventh branch output, the eighth branch output, and the ninth branch output to obtain a second intermediate output.

[0197] For example, in some examples, such as Figure 6B As shown, the seventh branch output O7, the eighth branch output O8 and the ninth branch output O9 can be connected with reference to the connection method in the aforementioned second sub-network to obtain the second intermediate output M2. The specific details are not repeated here.

[0198] Step S214: Use the third backbone network to process the second intermediate output to obtain multiple first feature maps.

[0199] For example, in some examples, such as Figure 6B As shown, the third backbone network N10 can be used to process the second intermediate output M2 to obtain multiple first feature maps F1. Figure 6B As shown, the third backbone network N10 may include a convolution module CN, so that the standard convolution process can be performed on the input second intermediate output M2 to obtain multiple first feature maps F1.

[0200] For example, in Figure 6B In the generative network shown in FIG, at least one second sub-network N2 can be used to perform step S220, that is, to perform branching processing and weight sharing processing on multiple first feature maps F1 to obtain multiple second feature maps F2. For example, the specific details can be referred to the aforementioned method based on Figure 6A The corresponding description of the generation network execution step S220 shown is not repeated here. It should be noted that, Figure 6B In the generated network shown, the number of the second sub-network N2 being 1 is exemplary and should not be regarded as a limitation to the present disclosure.

[0201] For example, in Figure 6B In the generation network shown in FIG, a dense sub-network DenseNet can be used to perform step S230, that is, to process the plurality of second feature maps F2 to obtain a plurality of third feature maps F3. For example, the specific details can be referred to the aforementioned method based on Figure 6A The corresponding description of the generation network execution step S230 is shown and will not be repeated here.

[0202] For example, in Figure 6B In the generation network shown in FIG, the third sub-network N3 can be used to perform step S240, that is, the third sub-network N3 is used to process the plurality of third feature maps F3 to obtain a plurality of fourth feature maps F4. Figure 6A The generative network shown is similar to Figure 6BIn the generation network shown, the third sub-network N3 may also include a second backbone network N30, a fourth branch network N31, a fifth branch network N32 and a sixth branch network N33. Thus, the processing of the third sub-network N3 may also include: using the second backbone network N30 to process multiple third feature maps F3 to obtain multiple fifth feature maps F5; dividing the multiple fifth feature maps F5 into a fourth branch input B4, a fifth branch input B5 and a sixth branch input B6; and, using the fourth branch network N31 to process the fourth branch input B4 to obtain a fourth feature map F4 corresponding to the fourth branch network N31, using the fifth branch network N32 to process the fifth branch input B5 to obtain a fourth feature map F4 corresponding to the fifth branch network N32, and using the sixth branch network N33 to process the sixth branch input B6 to obtain a fourth feature map F4 corresponding to the sixth branch network N33.

[0203] For example, with Figure 6A The generative network shown is similar to Figure 6B In the generation network shown, the second backbone network N30 may also include an upsampling layer US, so that the input multiple third feature maps F3 can be upsampled to obtain multiple fifth feature maps F5.

[0204] For example, with Figure 6A The generative network shown is similar to Figure 6B In the generative network shown in FIG, the fourth branch network N31 may also include a convolution module and an upsampling layer for performing standard convolution processing and upsampling processing. Figure 6B The difference between the generative network shown is that in Figure 6B In the generation network shown, the fifth branch network N32 and the sixth branch network N33 may both include a convolution module CN and a standard upsampling layer SUS for performing standard convolution processing and standard upsampling processing.

[0205] For example, the standard upsampling layer may use an interpolation algorithm such as interpolation, bilinear interpolation, or bicubic interpolation to perform standard upsampling processing. For example, when using an interpolation algorithm to perform standard upsampling processing, the original pixel values ​​and the interpolated values ​​may be retained, thereby increasing the size of the feature map.

[0206] For example, compared with the fourth branch network N31, the fifth branch network N32 is equivalent to omitting the convolution module for processing the highest resolution of the U channel, and the sixth branch network N33 is equivalent to omitting the convolution module for processing the highest resolution of the V channel, thereby improving the processing speed. This is similar to the aforementioned seventh branch network N11, eighth branch network N12, and ninth branch network N13. It should be noted that the standard upsampling layer SUS in N32 of the fifth branch network generally corresponds to the standard downsampling layer SDS in the eighth branch network N12, and the standard upsampling layer SUS in N33 of the sixth branch network generally corresponds to the standard downsampling layer SDS in the ninth branch network N13. It should be noted that the embodiments of the present disclosure include but are not limited to this.

[0207] For example, in Figure 6B In the generation network shown in FIG. 1 , the synthesis module Merg can be used to perform step S250, that is, the synthesis module Merg is used to process the plurality of fourth feature maps F4 to obtain the output image OUT. For example, the specific details can be referred to the aforementioned method based on Figure 6A The corresponding description of the generation network execution step S250 is shown and will not be repeated here.

[0208] It should be noted that, in the embodiments of the present disclosure, Figure 6A The generative network shown and Figure 6B The generative networks shown are exemplary and non-limiting. It should also be noted that before training, the generative network may not have any image enhancement capabilities, or it may have image enhancement capabilities but the image enhancement effect is poor. After training the generative network to be trained, the resulting generative network has image enhancement capabilities and can generate high-quality images.

[0209] Figure 9A is a schematic diagram of an input image, Figure 9B Based on Figure 6A The generative network shown is Figure 9A Schematic diagram of the output image obtained by processing the input image shown, Figure 9C Based on Figure 6B The generative network shown is Figure 9A Schematic diagram of the output image obtained by processing the input image shown in . Figure 9A Compared to the input image shown, Figure 9B and Figure 9CThe output images shown retain the content of the input images, but improve the image contrast and improve the problem of the input images being too dark. As a result, compared to the input images, the quality of the output images can be close to the quality of photos taken by a digital single-lens reflex camera, that is, the output images are high-quality images. Therefore, the image processing method provided by the embodiments of the present disclosure achieves the effect of image enhancement processing.

[0210] The image processing method provided by the embodiments of the present disclosure can combine branch processing and weight sharing processing to perform image enhancement processing, which can not only reduce the number of parameters but also facilitate the calculation of gradients during back propagation. Therefore, while outputting high-quality images, it can also improve the processing speed and convergence speed.

[0211] At least one embodiment of the present disclosure further provides a method for training a neural network. Figure 10 A flowchart of a neural network training method provided in at least one embodiment of the present disclosure. Figure 10 As shown, the training method includes:

[0212] Step S300: training the discriminant network based on the generative network to be trained;

[0213] Step S400: Based on the discriminant network, the generative network to be trained is trained; and

[0214] The above training process is performed alternately to obtain the generation network in the image processing method provided by any of the above embodiments of the present disclosure.

[0215] For example, in some examples, the construction of the generative network to be trained can be similar to Figure 6A The generative network shown or Figure 6B The generative network shown is the same, and embodiments of the present disclosure include but are not limited to this. For example, after being trained by this training method, the generative network to be trained can execute the image processing method provided by any of the above embodiments of the present disclosure, that is, the generative network obtained by using this training method can execute the image processing method provided by any of the above embodiments of the present disclosure.

[0216] Figure 11A A method corresponding to at least one embodiment of the present disclosure is provided Figure 10 The schematic architecture diagram of the training method shown in the figure is used to train the generative network to be trained. Figure 11B A schematic flowchart of a process for training a generative network to be trained is provided for at least one embodiment of the present disclosure.

[0217] For example, combined with Figure 11A and Figure 11B As shown, based on the discriminant network, the generative network to be trained is trained, that is, step S300, including steps S310 to S330.

[0218] Step S310: Processing a first training input image using the generative network to be trained to generate a first training output image.

[0219] For example, similar to the input image in the aforementioned step S100, the first training input image may also include a photograph captured by a smartphone camera, a tablet camera, a personal computer camera, a digital camera lens, a surveillance camera, or a webcam, and may include images of people, animals, plants, or landscapes, etc., and the embodiments of the present disclosure are not limited to this. For example, the first training input image is a low-quality image, for example, the quality of the first training input image is lower than the quality of a photograph taken by a digital single-lens reflex camera. For example, in some examples, the first training input image may be an image in RGB format, and the embodiments of the present disclosure include but are not limited to this.

[0220] For example, the generative network G to be trained can have Figure 6A The generative network shown or Figure 6B For example, the initial parameters of the generative network G to be trained can be random numbers, such as random numbers that conform to a Gaussian distribution. It should be noted that the embodiments of the present disclosure are not limited to this.

[0221] For example, the specific process of step S310 can refer to the relevant description of the aforementioned step S200, that is, the first training input image corresponds to the input image, the first training output image corresponds to the output image, and the process of generating the first training output image according to the first training input image can refer to the aforementioned process of generating the output image according to the input image, which will not be repeated here.

[0222] Step S320: Based on the first training output image, a system loss value of the generator network to be trained is calculated using a system loss function.

[0223] For example, in some examples, such as Figure 11A As shown, the system loss function may include generating a network adversarial loss function, and accordingly, the system loss value may include generating a network adversarial loss value. Figure 11A As shown, during the training process of the generative network G to be trained, the discriminant network D can be used to process the first training output image, and the generative network adversarial loss value is calculated by the generative network adversarial loss function according to the output of the discriminant network D.

[0224] Figure 12 A schematic diagram of a discriminant network structure provided by at least one embodiment of the present disclosure. Figure 12As shown, the discriminant network D includes multiple convolutional modules CM, multiple downsampling layers DS, and a fully connected layer FCN. For example, the structures and functions of the convolutional modules CM, downsampling layers DS, and fully connected layers FCN in the discriminant network D can refer to the aforementioned descriptions related to the convolutional modules, downsampling layers, and fully connected layers, respectively, and the embodiments of the present disclosure are not limited thereto.

[0225] For example, Figure 12 As shown, in the discriminant network D, multiple convolution modules CM are connected in sequence, and there is a downsampling layer DS between some adjacent convolution modules CM, for example, Figure 12 As shown, the discriminant network D includes six convolutional modules CM connected in sequence, with a downsampling layer between the second and third convolutional modules, and a downsampling layer between the fourth and fifth convolutional modules. A fully connected layer FCN is connected to the last convolutional module CM. For example, each convolutional module CM can include a convolutional layer and an instance normalization layer; for example, at least some convolutional modules CM can also omit the instance normalization layer.

[0226] For example, Figure 12 As shown, the discriminant network D also includes an activation layer, which is connected to the fully connected layer FCN. Figure 12 As shown, the activation function of the activation layer can be a Sigmoid function, so that the output of the activation layer (i.e., the output of the discriminant network D) is a value within the range of [0, 1]. For example, the discriminant network D can determine whether the quality of the first training output image is close to that of a high-quality image (e.g., a photograph taken by a digital single-lens reflex camera). Taking the first training output image as the input of the discriminant network D as an example, the discriminant network D processes the first training output image to obtain the discriminant network D output. The value output by the discriminant network D indicates the degree of closeness between the quality of the first training output image and the quality of, for example, a photograph taken by a digital single-lens reflex camera. For example, a larger value output by the discriminant network D, such as one approaching 1, indicates that the discriminant network D determines that the quality of the first training output image is closer to that of a photograph taken by a digital single-lens reflex camera, i.e., the quality of the first training output image is higher. For example, a smaller value output by the discriminant network D, such as one approaching 0, indicates that the discriminant network D determines that the quality of the first training output image is less close to that of a photograph taken by a digital single-lens reflex camera, i.e., the quality of the first training output image is lower.

[0227] It should be noted that Figure 12 The discriminant network shown is schematic. For example, in some examples, Figure 12 The discriminative network shown may include more or fewer convolutional modules or downsampling layers. For example, in some examples, Figure 12The discriminant network shown may also include other modules or layer structures, such as a flattening module before the fully connected layer. For example, in some examples, Figure 12 Some modules or layer structures in the discriminant network shown can be replaced by other modules or layer structures, such as replacing the fully connected layer with a convolutional layer that performs an average operation (AVG) (refer to Figure 3 and the above-mentioned related descriptions), and for example, replacing the activation layer with a two-class softmax module. Further, the embodiments of the present disclosure do not limit the structure of the discriminant network, including but not limited to Figure 12 The discriminant network structure shown.

[0228] For example, in some examples, the generative network adversarial loss function can be expressed as:

[0229]

[0230] Among them, L G represents the generation network adversarial loss function, z1 represents the first training input image, P z1 (z1) represents a set of first training input images (for example, a batch of multiple first training input images), G(z1) represents the first training output image, and D(G(z1)) represents the output of the discriminant network D for the first training output image, that is, the output obtained by the discriminant network D processing the first training output image. =( ...

[0231] It should be noted that the generative network adversarial loss function expressed in the above formula is exemplary, and the embodiments of the present disclosure include but are not limited to this.

[0232] The training objective of the generative network G to be trained is to minimize the system loss value. Therefore, during the training process of the generative network G to be trained, minimizing the system loss value includes reducing the generative network adversarial loss value. For example, during the training process of the generative network G to be trained, the label of the first training output image is set to 1, that is, it is necessary to make the discriminant network D identify and determine that the quality of the first training output image is consistent with the quality of a photo taken by a digital single-lens reflex camera, for example. For example, during the training process of the generative network G to be trained, the parameters of the generative network G to be trained are continuously modified so that the output of the discriminant network D corresponding to the first training output image generated by the generative network G to be trained after the parameter modification continuously approaches 1, thereby continuously reducing the generative network adversarial loss value.

[0233] For example, in some examples, such as Figure 11AAs shown, the system loss function may also include a content loss function, and accordingly, the system loss value may include a content loss value. Figure 11A As shown, during the training process of the generation network G to be trained, the analysis network A can be used to process the first training output image, and the content loss value is calculated by the content loss function according to the output of the analysis network A.

[0234] Figure 13 This is a schematic diagram of a structure of an analysis network provided by at least one embodiment of the present disclosure. Figure 13 As shown, the analysis network A includes a plurality of convolution modules CM connected in sequence and a plurality of downsampling layers DS interleaved between adjacent convolution modules CM. For example, each convolution module CM includes a convolution layer, and each convolution layer includes a plurality of convolution kernels, which can be used to extract content features and style features of the input image of the analysis network A. For example, referring to Figure 11A , Figure 13 The input of the analysis network A shown may include a first training input image and a first training output image. For example, each convolution module CM may include a convolution layer and an instance normalization layer; for example, at least some convolution modules CM may also omit the instance normalization layer.

[0235] For example, the analysis network A can adopt a deep neural network capable of classifying images such as Figure 13 As shown, the input is processed by several convolution modules CM and downsampling layers DS to extract features. The output of each convolution module CM is a feature image of its input. The downsampling layer DS can reduce the size of the feature image and pass it to the convolution module of the next level. Multiple convolution modules CM can output multiple feature images, which can represent different levels of features of the input (for example, texture, edge, object, etc.). After being processed by several convolution modules CM and downsampling layers DS, the feature image is input to the flattening layer, which converts the feature image into a vector and then passes it to the fully connected layer and the classifier. The classifier layer can include a softmax classifier, which can output the probability that the input belongs to each category identifier, where the identifier with the largest probability will be used as the final output of the analysis network A. Thus, the analysis network A realizes image classification.

[0236] For example, the analysis network A can adopt a trained convolutional neural network model. Thus, during the training process of the generation network G to be trained, there is no need to correct the parameters of the analysis network A (for example, convolution kernel, etc.). For example, the analysis network A can adopt neural network models such as AlexNet, GoogleNet, VGG, Deep Residual Learning, etc. to extract the content features and style features of the input. The VGG network is a type of deep convolutional neural network, which was developed by the Visual Geometry Group of the University of Oxford and has been widely used in the field of visual recognition. For example, the VGG network can include 19 layers, and some of the layers can be standardized.

[0237] It should be noted that, in some examples, during the training process of the generative network G to be trained, only the part of the analysis network A used to extract the features of its input is needed, for example, Figure 13 The multiple convolution modules CM and multiple downsampling layers DS are shown in the dotted box. For example, in the analysis network A provided by the embodiment of the present disclosure, Figure 13 As shown, at least two convolution modules CM are used to extract style features, and at least one convolution module CM is used to extract content features. Figure 13 The analysis network shown is schematic. The embodiments of the present disclosure do not limit the structure of the analysis network, the specific details of extracting style features and content features (for example, the number and level of the first convolutional modules used to extract style features and content features, etc.). It should also be noted that in some examples, during the training process of the generative network G to be trained, only the part of the analysis network A described above that is used to extract the content features of its input is required.

[0238] For example, Figure 11A As shown, during the training process of the generative network G to be trained, the analysis network A is used to receive a first training input image and a first training output image, and respectively generate and output a first content feature map of the first training input image and a second content feature map of the first training output image. For example, the content feature represents the distribution of objects in the image throughout the entire image, for example, the content feature includes image content information.

[0239] For example, during the training process of the generator network G to be trained, the analysis network A can be used to extract the first content feature map of the first training input image and the second content feature map of the first training output image, and the content loss value of the generator network G to be trained can be calculated based on the first content feature map and the second content feature map by the content loss function. Figure 13 For the analysis network A shown, the single-layer content loss function can be expressed as:

[0240]

[0241] Where S1 is a constant, represents the value at the jth position in the first content feature map of the first training input image extracted by the i-th convolution kernel in the m-th convolution module in the analysis network A, Represents the value of the j-th position in the second content feature map of the first training output image extracted by the i-th convolution kernel in the m-th convolution module in the analysis network A.

[0242] For example, in Figure 13 In the analysis network A shown, the content features of the input image (for example, the input image here includes the first training input image and the first training output image) can be extracted by at least one convolution module CM, and the content loss function can be expressed as:

[0243]

[0244] Among them, L content represents the content loss function, C m represents the single-layer content loss function of the mth convolution module in at least one convolution module for extracting content features, w 1m Indicates C m The weight of .

[0245] For example, during the training process of the generative network G to be trained, minimizing the system loss value includes reducing the content loss value. For example, when using the generative network G to be trained for image enhancement processing, it is necessary to maintain the output and input of the generative network G to be trained with the same content features, that is, the first training output image retains the content of the first training input image. For example, during the training process of the generative network G to be trained, the parameters of the generative network G to be trained are continuously modified so that the content features of the first training output image generated by the generative network G to be trained after the parameter modification continuously approach the content features of the first training input image, thereby continuously reducing the content loss value.

[0246] For example, in some examples, such as Figure 11A As shown, the system loss function may also include a color loss function, and accordingly, the system loss value may include a color loss value. Figure 11A As shown, during the training process of the generative network G to be trained, a color loss function can be established according to the first training output image and the second training input image to calculate the color loss value.

[0247] For example, in some examples, the color loss function can be expressed as:

[0248] L color=abs(gaussian(G(z1))-gaussian(I1))

[0249] Among them, L color represents the color loss function, G(z1) represents the first training output image, I1 represents the second training input image, gaussian() represents Gaussian blur operation, and abs() represents absolute value operation.

[0250] For example, the second training input image may be a high-quality image, i.e., the quality of the second training input image is higher than that of the first training input image. For example, the second training input image may be a photographic image captured by a digital single-lens reflex camera. For example, the second training input image may include images of people, animals, plants, or landscapes, etc., although the embodiments of the present disclosure are not limited thereto.

[0251] For example, in some examples, the quality of the first training output image is close to the quality of a photo taken by a digital SLR camera, which can be at least partially reflected in that the color distribution and brightness distribution of each local area of ​​the first training output image are close to those of the photo taken by the digital SLR camera.

[0252] For example, during the training process of the generative network G to be trained, minimizing the system loss value includes reducing the color loss value. For example, when performing image enhancement processing using the generative network G to be trained, it is necessary to make the color distribution and brightness distribution of each local area of ​​the first training output image close to that of, for example, a photograph taken by a digital single-lens reflex camera. For example, during the training process of the generative network G to be trained, the parameters of the generative network G to be trained are continuously modified so that the color distribution and brightness distribution of each local area of ​​the first training output image generated by the generative network G to be trained after the parameter modification are close to that of, for example, a photograph taken by a digital single-lens reflex camera, thereby continuously reducing the color loss value.

[0253] For example, in some examples, the first training output image has a first color channel, a second color channel, and a third color channel. For example, reference may be made to the description of the output image in the aforementioned image processing method, which will not be repeated here. Figure 11A As shown, the system loss function may also include a contrast loss function, and accordingly, the system loss value may include a contrast loss value. Figure 11A As shown, during the training process of the generative network G to be trained, a contrast loss function can be established according to the first training output image and the third training input image to calculate the contrast loss value.

[0254] For example, in some examples, the contrastive loss function can be expressed as:

[0255] L L1=0.299*abs(F G(z1) -F I2 )+0.587*abs(S G(z1) -S I2 )+0.299*abs(T G(z1) -T I2 )

[0256] Among them, L L1 represents the contrast loss function, G(z1) represents the first training output image, I2 represents the third training input image, F G(z1) 、S G(z1) and T G(z1) Respectively represent the data information of the first color channel, the second color channel and the third color channel of the first training output image, F I2 、S I2 and T I2 Respectively represent the data information of the first color channel, the second color channel and the third color channel of the third training input image, and abs() represents an absolute value operation.

[0257] For example, the third training input image can have the same scene as the first training input image, that is, the same content, and the quality of the third training input image can be higher than that of the first training input image. For example, the third training input image can be a photograph taken with a digital single-lens reflex camera. Because the third training input image can be equivalent to the target output image of the generative network G to be trained, adding a contrastive loss function to the system loss function can improve convergence and processing speed.

[0258] For example, during the training of the generative network G to be trained, minimizing the system loss value includes reducing the contrast loss value. For example, when performing image enhancement processing using the generative network G to be trained, it is necessary to make the first training output image close to the third training input image. For example, during the training of the generative network G to be trained, the parameters of the generative network G to be trained are continuously modified so that the first training output image generated by the generative network G to be trained after the parameter modification is close to the third training input image, thereby continuously reducing the contrast loss value.

[0259] For example, in an embodiment of the present disclosure, the system loss function of the generative network G to be trained can be expressed as:

[0260] L total =αL G +βL content +χL color +δL L1

[0261] Among them, L totalRepresents the system loss function, α, β, χ, and δ are the weights of the generated network adversarial loss function, content loss function, color loss function, and contrast loss function in the system loss function, respectively.

[0262] For example, in some examples, to prevent overfitting, the weight δ of the contrast loss function may be smaller than the weight β of the content loss function; for example, in some examples, the weight of the contrast loss function does not exceed 20%.

[0263] For example, in step S320, the system loss value can be calculated using the system loss function represented by the above formula, and then the subsequent step S330 is executed to correct the parameters of the generative network G to be trained, thereby implementing step S300.

[0264] Step S330: Modify the parameters of the generative network to be trained based on the system loss value.

[0265] For example, the training process of the generative network G to be trained may also include an optimization function ( Figure 11A (not shown), the optimization function can calculate the error value of the parameters of the generation network G according to the system loss value calculated by the system loss function, and modify the parameters of the generation network G to be trained according to the error value. For example, the optimization function can use a stochastic gradient descent (SGD) algorithm, a batch gradient descent (BGD) algorithm, etc. to calculate the error value of the parameters of the generation network G.

[0266] For example, the generative network G to be trained is Figure 6A Taking the generative network shown in the figure as an example, the parameters of the generative network G to be trained are modified, including: Figure 6A The parameters of the generative network shown in FIG. 1 are modified except for the parameters of the synthesis module Merg, that is, the parameters of the synthesis module Merg remain unchanged. For example, the generative network G to be trained is Figure 6B Taking the generative network shown in the figure as an example, the parameters of the generative network G to be trained are modified, including: Figure 6B The parameters of the generation network shown are modified except for the parameters of the conversion module Tran and the synthesis module Merg, that is, the parameters of the conversion module Tran and the synthesis module Merg remain unchanged.

[0267] For example, based on the discriminant network, the generative network to be trained is trained, that is, step S300 may also include: judging whether the training of the generative network G to be trained meets the predetermined conditions; if the predetermined conditions are not met, repeating the training process of the generative network G to be trained; if the predetermined conditions are met, stopping the training process of the generative network G to be trained at this stage, and obtaining the generative network G trained at this stage. It should be noted that the generative network G trained at this stage can be used as the generative network G to be trained at the next stage. For example, in one example, the predetermined condition is that the system loss value corresponding to two consecutive (or more) first training input images no longer decreases significantly. For example, in another example, the predetermined condition is that the number of training times or training cycles of the generative network G reaches a predetermined number. It should be noted that the embodiments of the present disclosure are not limited to this.

[0268] For example, Figure 11A As shown in the figure, during the training process of the generative network G to be trained, the discriminative network D and the analysis network A need to be trained together. It should be noted that during the training process of the generative network G, the parameters of the discriminative network D remain unchanged. It should be noted that when the analysis network A adopts a pre-trained convolutional neural network model, the parameters of the analysis network A can also remain unchanged.

[0269] It should be noted that in the present disclosure, for example, the generative network to be trained, the discriminative network, the analytical network, and the various layers or modules (such as convolution modules, upsampling layers, downsampling layers, etc.) included in these neural networks each correspond to a program / method for executing a corresponding processing process, such as implemented by corresponding software, firmware, hardware, etc.; and the above examples are only schematic illustrations of the training process of the generative network to be trained. Those skilled in the art should know that in the training phase, a large number of sample images are needed to train the neural network; at the same time, in the training process of each sample image, multiple iterations can be included to correct the parameters of the generative network to be trained. For another example, the training phase also includes fine-tuning the parameters of the generative network to be trained to obtain more optimized parameters.

[0270] Figure 14A A method corresponding to at least one embodiment of the present disclosure is provided Figure 10 The schematic architecture diagram of the training method shown in the figure is used to train the discriminant network. Figure 14B A schematic flowchart of a process for training a discriminant network is provided for at least one embodiment of the present disclosure.

[0271] For example, combined with Figure 14A and Figure 14B As shown, based on the generative network to be trained, the discriminative network is trained, that is, step S400, including steps S410 to S430, as shown below:

[0272] Step S410: Processing the fourth training input image using the generative network to be trained to generate a second training output image;

[0273] Step S420: Based on the second training output image and the fifth training input image, calculating the discriminant network adversarial loss value by using the discriminant network adversarial loss function;

[0274] Step S430: Modify the parameters of the discriminant network according to the discriminant network adversarial loss value.

[0275] For example, based on the generative network to be trained, the discriminant network is trained, that is, step S400 may also include: judging whether the training of the discriminant network D meets a predetermined condition; if the predetermined condition is not met, repeating the training process of the discriminant network D; if the predetermined condition is met, stopping the training process of the discriminant network D at this stage, and obtaining the discriminant network D trained at this stage. For example, in one example, the predetermined condition is that the adversarial loss value of the discriminant network corresponding to two consecutive (or more) fifth training input images and second training output images no longer decreases significantly. For example, in another example, the predetermined condition is that the number of training times or training cycles of the discriminant network D reaches a predetermined number. It should be noted that the embodiments of the present disclosure are not limited to this.

[0276] For example, Figure 14A As shown in FIG, during the training process of the discriminant network D, it is necessary to train the generative network G to be trained together. It should be noted that during the training process of the discriminant network D, the parameters of the generative network G to be trained remain unchanged.

[0277] It should be noted that the above examples are merely illustrative of the training process for the discriminant network. Those skilled in the art will appreciate that during the training phase, a large number of sample images are required to train the neural network. Furthermore, the training process for each sample image may include multiple iterations to refine the parameters of the discriminant network. For example, the training phase may also include fine-tuning the parameters of the discriminant network to obtain more optimized parameters.

[0278] For example, the initial parameters of the discriminant network D may be random numbers, for example, random numbers that conform to a Gaussian distribution, which is not limited in the embodiments of the present disclosure.

[0279] For example, the training process of the discriminant network D can also include the optimization function ( Figure 14A(not shown), the optimization function can calculate the error value of the parameters of the discriminant network D according to the discriminant network adversarial loss value calculated by the discriminant network adversarial loss function, and correct the parameters of the discriminant network D according to the error value. For example, the optimization function can use a stochastic gradient descent (SGD) algorithm, a batch gradient descent (BGD) algorithm, etc. to calculate the error value of the parameters of the discriminant network D.

[0280] For example, the fourth training input image may be the same as the first training input image, for example, the set of the fourth training input image and the set of the first training input image are the same image set. For example, the fourth training input image may also include photos captured by a smartphone camera, a tablet camera, a personal computer camera, a digital camera lens, a surveillance camera, or a webcam, and may include images of people, animals, plants, or landscapes, etc., and the embodiments of the present disclosure are not limited to this. For example, the fourth training input image is a low-quality image, for example, the quality of the fourth training input image is lower than the quality of a photo taken by a digital single-lens reflex camera, for example. For example, in some examples, the fourth training input image may be an image in RGB format, and the embodiments of the present disclosure include but are not limited to this.

[0281] For example, the fifth training input image is a high-quality image, i.e., the quality of the fifth training input image is higher than that of the fourth training input image. For example, the fifth training input image can be a photographic image captured by a digital single-lens reflex camera. For example, the fifth training input image can include images of people, animals, plants, or landscapes, etc., and the embodiments of the present disclosure are not limited in this regard. For example, the fifth training input image can be the same as the second training input image, for example, the set of the fifth training input image and the set of the second training input image are the same image set. Of course, the fifth training input image can also be different from the second training input image, and the embodiments of the present disclosure are not limited in this regard.

[0282] For example, the discriminant network D can be Figure 12 The discriminant network shown is, but not limited to,

[0283] For example, in some examples, the discriminative network adversarial loss function can be expressed as:

[0284]

[0285] Among them, L D represents the discriminant network adversarial loss function, x represents the fifth training input image, P data(x) represents a set of fifth training input images (for example, a batch of multiple fifth training input images), D(x) represents the output of the discriminant network D for the fifth training input image x, that is, the output obtained by the discriminant network D after processing the fifth training input image x, represents the expectation for the set of the fifth training input image, z2 represents the fourth training input image, P z2 (z2) represents a set of fourth training input images (for example, a batch of multiple fourth training input images), G(z2) represents the second training output image, and D(G(z2)) represents the output of the discriminant network D for the second training output image, that is, the output obtained by the discriminant network D after processing the second training output image. It represents the expectation of the set of training input images. Therefore, the batch gradient descent algorithm can be used to optimize the parameters of the discriminant network D.

[0286] It should be noted that the discriminant network adversarial loss function expressed in the above formula is exemplary, and the present disclosure includes but is not limited to this.

[0287] The training objective of the discriminant network D is to minimize the discriminant network's adversarial loss. For example, during the training of the discriminant network D, the label of the fifth training input image is set to 1, which means that the discriminant network D needs to identify the fifth training input image as a photograph, such as one taken with a digital single-lens reflex camera, and therefore a high-quality image. Simultaneously, the label of the second training output image is set to 0, which means that the discriminant network D needs to identify the second training output image as not a photograph, such as one taken with a digital single-lens reflex camera, and therefore a low-quality image.

[0288] For example, during the training process of the discriminant network D, the parameters of the discriminant network D are continuously corrected so that the discriminant network D after parameter correction can accurately identify the quality of the fifth training input image and the second training output image, that is, the output of the discriminant network D corresponding to the fifth training input image is continuously approached to 1, and the output of the discriminant network D corresponding to the second training output image is continuously approached to 0, thereby continuously reducing the adversarial loss value of the generation network.

[0289] For example, in an embodiment of the present disclosure, the training of the generative network G to be trained and the training of the discriminative network D are performed alternately and iteratively. For example, for an untrained generative network G and a discriminative network D, the discriminative network D is generally first trained in the first stage to improve the discrimination ability of the discriminative network D (i.e., to identify the quality of the input of the discriminative network D), thereby obtaining the discriminative network D that has undergone the first stage of training; then, based on the discriminative network D that has undergone the first stage of training, the generative network G (i.e., the generative network G to be trained) is trained in the first stage to improve the image enhancement processing capability of the generative network G (i.e., to make the output of the generative network G a high-quality image), thereby obtaining the generative network G that has undergone the first stage of training. Similar to the first stage training, in the second stage training, based on the generative network G trained in the first stage, the discriminative network D trained in the first stage is subjected to the second stage training to improve the discrimination ability of the discriminative network D, and obtain the discriminative network D trained in the second stage; then, based on the discriminative network D trained in the second stage, the generative network G trained in the first stage is subjected to the second stage training to improve the image enhancement processing ability of the generative network G, and obtain the generative network G trained in the second stage. And so on, the discriminative network D and the generative network G are then subjected to the third stage training, the fourth stage training, and so on, until the quality of the output of the generative network G obtained can be close to the quality of photos taken by a digital single-lens reflex camera, that is, the training output image is a high-quality image.

[0290] It should be noted that during the alternating training of the generative network G and the discriminative network D, the competition between the generative network G and the discriminative network D is reflected in the fact that the output of the generative network G (the high-resolution image generated by the generative network G) has different labels during their respective training processes (labeled 1 during the training of the generative network G and labeled 0 during the training of the discriminative network D). This is also reflected in the fact that the second part of the discriminative network's adversarial loss function (i.e., the part related to the high-resolution image generated by the generative network G) is opposite to the generative network's adversarial loss function in the system loss function. It should also be noted that, ideally, the images output by the trained generative network G are high-quality images (i.e., close to the quality of photos taken by a digital single-lens reflex camera, for example), and the output of the discriminative network D for both the fifth training input image and the second training output image generated by the generative network G is 0.5, indicating that the generative network G and the discriminative network D reach a Nash equilibrium after the adversarial game.

[0291] At least one embodiment of the present disclosure provides a neural network training method that combines generative adversarial network technology. The generative network trained by this training method can combine branch processing and weight sharing processing to perform image enhancement processing, which can not only reduce the number of parameters but also facilitate the calculation of gradients during back propagation. Therefore, while outputting high-quality images, it can also improve the processing speed and convergence speed.

[0292] At least one embodiment of the present disclosure further provides an image processing device. Figure 15 A schematic block diagram of an image processing device provided in one embodiment of the present disclosure.

[0293] For example, Figure 15 As shown, the image processing apparatus 500 includes a memory 510 and a processor 520. For example, the memory 510 is used to non-temporarily store computer-readable instructions, and the processor 520 is used to execute the computer-readable instructions. When the computer-readable instructions are executed by the processor 520, the image processing method provided by any embodiment of the present disclosure is executed.

[0294] For example, the memory 510 and the processor 520 can communicate with each other directly or indirectly. For example, the memory 510 and the processor 520 and other components can communicate through a network connection. The network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The network may include a local area network, the Internet, a telecommunications network, an Internet of Things based on the Internet and / or telecommunications network, and / or any combination of the above networks. For example, a wired network can communicate using twisted pair, coaxial cable or optical fiber transmission, and for example, a wireless network can communicate using 3G / 4G / 5G mobile communication networks, Bluetooth, Zigbee or WiFi. The present disclosure does not limit the type and function of the network.

[0295] For example, the processor 520 can control other components in the image processing device to perform desired functions. The processor 520 can be a device with data processing capabilities and / or program execution capabilities, such as a central processing unit (CPU), a tensor processing unit (TPU), or a graphics processing unit (GPU). The central processing unit (CPU) can be an X86 or ARM architecture. The GPU can be directly integrated into the motherboard separately, or built into the north bridge chip of the motherboard. The GPU can also be built into the central processing unit (CPU).

[0296] For example, the memory 510 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, a flash memory, etc.

[0297] For example, the memory 510 may store one or more computer instructions, and the processor 520 may execute the computer instructions to implement various functions. The computer-readable storage medium may also store various applications and various data, such as the first to fifth training input images, and various data used and / or generated by the application.

[0298] For example, when some computer instructions stored in the memory 510 are executed by the processor 520, one or more steps of the image processing method provided in any embodiment of the present disclosure may be executed, and / or one or more steps of the neural network training method provided in any embodiment of the present disclosure may be executed.

[0299] For example, for a detailed description of the processing process of the image processing method, please refer to the relevant description in the embodiment of the above-mentioned image processing method. For a detailed description of the processing process of the neural network training method, please refer to the relevant description in the embodiment of the above-mentioned neural network training method. The repeated parts will not be repeated here.

[0300] It should be noted that the image processing device provided in the embodiments of the present disclosure is exemplary rather than restrictive. According to actual application requirements, the image processing device may also include other conventional components or structures. For example, in order to realize the necessary functions of the image processing device, those skilled in the art may set other conventional components or structures according to the specific application scenarios, and the embodiments of the present disclosure do not limit this.

[0301] The technical effects of the image processing device provided by at least one embodiment of the present disclosure can be referred to the corresponding descriptions of the image processing method and the neural network training method in the above embodiments, and will not be repeated here.

[0302] At least one embodiment of the present disclosure further provides a storage medium. Figure 16 A schematic diagram of a storage medium provided by an embodiment of the present disclosure. Figure 16 As shown, the storage medium 600 non-temporarily stores computer-readable instructions 601. When the non-temporary computer-readable instructions 601 are executed by a computer (including a processor), the instructions of the image processing method provided by any embodiment of the present disclosure can be executed.

[0303] For example, one or more computer instructions may be stored on the storage medium 600. Some of the computer instructions stored on the storage medium 600 may be, for example, instructions for implementing one or more steps in the above-described image processing method. Other computer instructions stored on the storage medium may be, for example, instructions for implementing one or more steps in the above-described neural network training method.

[0304] For example, the storage medium may include a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a compact disc read-only memory (CD-ROM), a flash memory, or any combination of the above storage media, or other applicable storage media.

[0305] The technical effects of the storage medium provided by the embodiments of the present disclosure can be referred to the corresponding descriptions of the image processing method and the neural network training method in the above embodiments, and will not be repeated here.

[0306] Regarding this disclosure, the following points need to be explained:

[0307] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure, and other structures may refer to conventional designs.

[0308] (2) Unless there is any conflict, the features of the same embodiment and different embodiments of the present disclosure may be combined with each other.

[0309] The above are only specific embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. An image processing method, comprising: Get the input image; as well as The input image is processed using a generative network to generate an output image; wherein, The generation network includes a first sub-network and at least one second sub-network, Processing the input image using the generative network to generate the output image includes: Processing the input image using the first sub-network to obtain a plurality of first feature maps; Using the at least one second sub-network to perform branching processing and weight sharing processing on the plurality of first feature maps to obtain a plurality of second feature maps, wherein the output of the branching processing serves as the input of the weight sharing processing; and Processing the plurality of second feature maps to obtain the output image; Each of the second sub-networks includes a first branch network, a second branch network, and a third branch network, and the branch processing of each of the second sub-networks includes: dividing the input of each of the second sub-networks into a first branch input, a second branch input, and a third branch input; and Using the first branch network to process the first branch input to obtain a first branch output, using the second branch network to process the second branch input to obtain a second branch output, and using the third branch network to process the third branch input to obtain a third branch output; The at least one second sub-network includes a first second sub-network, the first second sub-network is connected to the first sub-network, and the plurality of first feature maps serve as inputs of the first second sub-network; Each of the second sub-networks further includes a first backbone network, and the weight sharing process of each of the second sub-networks includes: connecting the first branch output, the second branch output, and the third branch output to obtain a first intermediate output; and Processing the first intermediate output using the first backbone network to obtain an output of each of the second sub-networks; The input of each second sub-network includes 3n feature maps, where n is a positive integer, and the input of each second sub-network is divided into a first branch input, a second branch input, and a third branch input, including: Dividing the 3n feature maps into three feature map groups, respectively serving as the first branch input, the second branch input, and the third branch input, wherein each feature map group includes at least one feature map from the 3n feature maps; The processing of the first branch network is standard convolution processing, the processing of the second branch network is standard convolution processing, the processing of the third branch network is standard convolution processing, and the processing of the first trunk network is standard convolution processing and downsampling processing; The input image has a first color channel, a second color channel and a third color channel, The first sub-network includes a conversion module, a seventh branch network, an eighth branch network, a ninth branch network and a third trunk network. Processing the input image using the first sub-network to obtain the plurality of first feature maps includes: Using the conversion module, the data information of the first color channel, the second color channel, and the third color channel of the input image are converted into data information of the first luminance channel, the first color difference channel, and the second color difference channel of the intermediate input image; Using the seventh branch network to process data information of a first luminance channel of the intermediate input image to obtain a seventh branch output, using the eighth branch network to process data information of a first color difference channel of the intermediate input image to obtain an eighth branch output, and using the ninth branch network to process data information of a second color difference channel of the intermediate input image to obtain a ninth branch output; connecting the seventh branch output, the eighth branch output, and the ninth branch output to obtain a second intermediate output; and Using the third backbone network to process the second intermediate output to obtain the plurality of first feature maps, The processing of the seventh branch network is standard convolution processing and downsampling processing, the processing of the eighth branch network is standard downsampling processing, the processing of the ninth branch network is standard downsampling processing, and the processing of the third trunk network is standard convolution processing.

2. The image processing method according to claim 1, wherein: The generation network also includes a third sub-network, Processing the plurality of second feature maps to obtain the output image comprises: Processing the plurality of second feature maps to obtain a plurality of third feature maps; Processing the plurality of third feature maps using the third sub-network to obtain a plurality of fourth feature maps; and The plurality of fourth feature maps are synthesized to obtain an output image.

3. The image processing method according to claim 2, wherein: The third sub-network includes a second trunk network, a fourth branch network, a fifth branch network and a sixth branch network, Processing the plurality of third feature maps using the third sub-network to obtain the plurality of fourth feature maps includes: Using the second backbone network to process the plurality of third feature maps to obtain a plurality of fifth feature maps; dividing the plurality of fifth feature maps into a fourth branch input, a fifth branch input, and a sixth branch input; and Using the fourth branch network to process the fourth branch input to obtain a fourth feature map corresponding to the fourth branch network, using the fifth branch network to process the fifth branch input to obtain a fourth feature map corresponding to the fifth branch network, and using the sixth branch network to process the sixth branch input to obtain a fourth feature map corresponding to the sixth branch network; The plurality of fourth feature maps include a fourth feature map corresponding to the fourth branch network, a fourth feature map corresponding to the fifth branch network, and a fourth feature map corresponding to the sixth branch network.

4. The image processing method according to claim 3, wherein: The processing of the second trunk network includes upsampling processing, the processing of the fourth branch network includes standard convolution processing, the processing of the fifth branch network includes standard convolution processing, and the processing of the sixth branch network includes standard convolution processing.

5. The image processing method according to claim 4, wherein: The processing of the fourth branch network further includes upsampling processing, the processing of the fifth branch network further includes upsampling processing, and the processing of the sixth branch network further includes upsampling processing.

6. The image processing method according to claim 2, wherein: The processing of the first sub-network includes standard convolution processing, Processing the input image using the first sub-network to obtain the plurality of first feature maps includes: Performing standard convolution processing on the input image using the first sub-network to obtain the multiple first feature maps.

7. The image processing method according to claim 3, wherein: The processing of the fourth branch network includes standard convolution processing and upsampling processing, the processing of the fifth branch network includes standard convolution processing and standard upsampling processing, and the processing of the sixth branch network includes standard convolution processing and standard upsampling processing.

8. The image processing method according to claim 2, wherein: The generation network also includes a dense sub-network, and the dense sub-network includes N dense modules. Processing the plurality of second feature maps to obtain the plurality of third feature maps includes: Processing the plurality of second feature maps using the dense sub-network to obtain the plurality of third feature maps; The plurality of second feature maps are used as inputs of the first dense module among the N dense modules. The plurality of second feature maps are connected to the output of the i-1 dense modules before the i-th dense module among the N dense modules as the input of the i-th dense module, The plurality of second feature maps are connected to the output of each of the dense modules to form the plurality of third feature maps, N and i are integers, N≥2, i≥2 and i≤N.

9. The image processing method according to claim 8, wherein: The processing of each dense module includes dimensionality reduction and convolution.

10. The image processing method according to claim 3, wherein: The generation network also includes a synthesis module, Performing synthesis processing on the plurality of fourth feature maps to obtain the output image includes: The synthesis module is used to perform synthesis processing on the multiple fourth feature maps to obtain the output image.

11. The image processing method according to claim 10, wherein: The synthesis module includes a first conversion matrix, Using the synthesis module to perform synthesis processing on the plurality of fourth feature maps to obtain the output image includes: Using the first conversion matrix, data information of the fourth feature map corresponding to the fourth branch network, data information of the fourth feature map corresponding to the fifth branch network, and data information of the fourth feature map corresponding to the sixth branch network are converted into data information of the first color channel, data information of the second color channel, and data information of the third color channel of the output image to obtain the output image.

12. A method for training a neural network, comprising: Based on the generative network to be trained, the discriminative network is trained; Based on the discriminant network, training the generative network to be trained; as well as, The above training process is performed alternately to obtain the generating network in the image processing method according to any one of claims 1 to 11; wherein, Based on the discriminant network, training the generative network to be trained includes: Processing a first training input image using the generative network to be trained to generate a first training output image; Based on the first training output image, calculating a system loss value of the generator network to be trained using a system loss function; and The parameters of the generator network to be trained are modified based on the system loss value.

13. The training method according to claim 12, wherein: The system loss function includes a generated network adversarial loss function, and the system loss value includes a generated network adversarial loss value; the generated network adversarial loss function is expressed as: Among them, L G represents the generative network adversarial loss function, z1 represents the first training input image, P z1 (z1) represents the set of the first training input images, G(z1) represents the first training output image, D(G(z1)) represents the output of the discriminant network for the first training output image, It represents finding the expectation for the set of the first training input images to obtain the generative network adversarial loss value.

14. The training method according to claim 13, wherein: The system loss function also includes a content loss function, and the system loss value also includes a content loss value; Calculating a system loss value of the generator network to be trained based on the first training output image using a system loss function, including: extracting a first content feature map of the first training input image and a second content feature map of the first training output image using an analysis network, and calculating the content loss value of the generator network using the content loss function based on the first content feature map and the second content feature map. Wherein, the analysis network comprises at least one convolution module for extracting the first content feature map and the second content feature map; The content loss function is expressed as: Among them, L content Denotes the content loss function, C m represents the single-layer content loss function of the mth convolutional module in the at least one convolutional module, w 1m Indicates C m The weight of The single-layer content loss function is expressed as: Where S1 is a constant, represents the value of the j-th position in the first content feature map of the first training input image extracted by the i-th convolution kernel in the m-th convolution module, Represents the value of the j-th position in the second content feature map of the first training output image extracted by the i-th convolution kernel in the m-th convolution module.

15. The training method according to claim 14, wherein: The system loss function also includes a color loss function, and the system loss value also includes a color loss value; the color loss function is expressed as: L color =abs(gaussian(G(z1))-gaussian(I1)) Among them, L color represents the color loss function, G(z1) represents the first training output image, I1 represents the second training input image, gaussian() represents Gaussian blur operation, and abs() represents absolute value operation; The quality of the second training input image is higher than that of the first training input image.

16. The training method according to claim 15, wherein: The first training output image has a first color channel, a second color channel, and a third color channel; The system loss function also includes a contrast loss function, and the system loss value also includes a contrast loss value; the contrast loss function is expressed as: L L1 =0.299*abs(F G(z1) -F I2 )+0.587*abs(S G(z1) -S I2 )+0.299*abs(T G(z1) -T I2 ) Among them, L L1 represents the contrast loss function, G(z1) represents the first training output image, I2 represents the third training input image, F G(z1) 、S G(z1) and T G(z1) Respectively represent the data information of the first color channel, the second color channel and the third color channel of the first training output image, F I2 、S I2 and T I2 Respectively represent the data information of the first color channel, the second color channel and the third color channel of the third training input image, and abs() represents an absolute value operation; The third training input image has the same scene as the first training input image, and the quality of the third training input image is higher than that of the first training input image.

17. The training method according to any one of claims 12 to 16, wherein: Training the discriminant network based on the generative network to be trained includes: Processing a fourth training input image using the generative network to be trained to generate a second training output image; Calculating a discriminative network adversarial loss value using a discriminative network adversarial loss function based on the second training output image and the fifth training input image; and Modifying the parameters of the discriminant network according to the adversarial loss value of the discriminant network; The quality of the fifth training input image is higher than the quality of the fourth training input image.

18. The training method according to claim 17, wherein: The discriminant network adversarial loss function is expressed as: Among them, L D represents the discriminant network adversarial loss function, x represents the fifth training input image, P data (x) represents the set of the fifth training input images, D(x) represents the output of the discriminant network for the fifth training input image, represents the expectation for the set of the fifth training input images, z2 represents the fourth training input image, P z2 (z2) represents the set of the fourth training input images, G(z2) represents the second training output image, D(G(z2)) represents the output of the discriminant network for the second training output image, It represents finding the expectation for the set of the fourth training input images.

19. An image processing apparatus, comprising: a memory for non-transitory storage of computer-readable instructions; as well as A processor is configured to execute the computer-readable instructions, wherein the computer-readable instructions, when executed by the processor, execute the image processing method according to any one of claims 1 to 11.

20. A storage medium non-transitorily storing computer-readable instructions, which, when executed by a computer, executes the image processing method according to any one of claims 1 to 11.

Images