Image Processing Method and Apparatus, Device, Video Processing Method and Storage Medium

Abstract

An image processing method, apparatus, device, video processing method, and storage medium. The image processing method includes: receiving an input image; processing the input image using a convolutional neural network to obtain an output image, where the clarity of the output image is higher than that of the input image. Processing the input image using a convolutional neural network to obtain an output image includes: extracting features from the input image to obtain a plurality of first images; performing a splicing process on the input image and the plurality of first images to obtain a first image group; extracting features from the first image group to obtain a plurality of second images; fusing the plurality of second images and the plurality of first images to obtain a plurality of third images; performing a splicing process on the input image and the plurality of third images to obtain a second image group; extracting features from the second image group to obtain the output image. The convolutional neural network required by this image processing method has a simple structure, can improve the clarity of the picture, and is convenient to be deployed on the terminal.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to an image processing method and apparatus, a device, a video processing method, and a storage medium. Background Art

[0002] Currently, deep learning technologies based on artificial neural networks have made great progress in fields such as object classification, text processing, recommendation engines, image search, face recognition, age and speech recognition, human-computer dialogue, and sentiment computing. With the in-depth research on artificial neural network structures and the improvement of related algorithms, deep learning technologies have made breakthrough progress in the field of human-like data perception. Deep learning technologies can be used to describe image content, identify objects in complex environments in images, and process images as needed. Summary of the Invention

[0003] At least one embodiment of the present disclosure provides an image processing method applicable to a convolutional neural network. The method includes: receiving an input image; processing the input image using the convolutional neural network to obtain an output image, where the clarity of the output image is higher than that of the input image; processing the input image using the convolutional neural network to obtain the output image includes: extracting features from the input image to obtain a plurality of first images; performing a splicing process on the input image and the plurality of first images to obtain a first image group, where the first image group includes the input image and the plurality of first images; extracting features from the first image group to obtain a plurality of second images; fusing the plurality of second images and the plurality of first images to obtain a plurality of third images; performing a splicing process on the input image and the plurality of third images to obtain a second image group, where the second image group includes the input image and the plurality of third images; extracting features from the second image group to obtain the output image.

[0004] For example, in the method provided by an embodiment of the present disclosure, the number of convolutional kernels used to extract features from the input image in the convolutional neural network is N, 12 ≤ N ≤ 20 and N is an integer, the number of convolutional kernels used to extract features from the first image group in the convolutional neural network is M, 12 ≤ M ≤ 20 and M is an integer, and the number of convolutional kernels used to extract features from the second image group in the convolutional neural network is 3.

[0005] For example, in the method provided by an embodiment of the present disclosure, N = M = 16; the size of the convolution kernel for feature extraction of the input image, the size of the convolution kernel for feature extraction of the first image group, and the size of the convolution kernel for feature extraction of the second image group are all 3×3; the activation function for feature extraction in the convolutional neural network is: y = max(0, x), where x represents the input of the activation function and y represents the output of the activation function; the input image includes a red-channel input image, a green-channel input image, and a blue-channel input image, and the output image includes a red-channel output image, a green-channel output image, and a blue-channel output image.

[0006] For example, the method provided by an embodiment of the present disclosure further includes: training a second neural network to be trained based on a pre-trained first neural network to obtain a trained second neural network, thereby obtaining the convolutional neural network; wherein, the parameters of the first neural network are more than those of the second neural network, the pre-trained first neural network is configured to transform an original image with a first clarity input to the pre-trained first neural network into a new image with a second clarity, the second clarity is greater than the first clarity, the trained second neural network is the convolutional neural network, the network structure of the second neural network to be trained is the same as that of the convolutional neural network, and the parameters of the second neural network to be trained are different from those of the convolutional neural network.

[0007] For example, in the method provided by an embodiment of the present disclosure, training the second neural network to be trained based on the pre-trained first neural network to obtain the trained second neural network, thereby obtaining the convolutional neural network, includes: alternately training the discriminative network and the second neural network based on the pre-trained first neural network, the second neural network to be trained, and the discriminative network to obtain the trained second neural network, thereby obtaining the convolutional neural network.

[0008] For example, in the method provided in an embodiment of the present disclosure, training the discrimination network includes: inputting first sample data into the first neural network and the second neural network respectively to obtain first data output from the first neural network and second data output from the second neural network; setting the first data to have a true value label, and inputting the first data with the true value label into the discrimination network to obtain a first discrimination result, setting the second data to have a false value label, and inputting the second data with the false value label into the discrimination network to obtain a second discrimination result; calculating a first loss function based on the first discrimination result and the second discrimination result; adjusting the parameters of the discrimination network according to the first loss function to obtain an updated discrimination network.

[0009] For example, in the method provided in an embodiment of the present disclosure, training the second neural network includes: inputting second sample data into the first neural network and the second neural network respectively to obtain third data output from the first neural network and fourth data output from the second neural network; setting the fourth data to have a true value label, and inputting the fourth data with the true value label into the updated discrimination network to obtain a third discrimination result output from the discrimination network; calculating an error function based on the third data and the fourth data, calculating a discrimination function based on the third discrimination result, and calculating a second loss function based on the error function and the discrimination function; adjusting the parameters of the second neural network according to the second loss function to obtain an updated second neural network.

[0010] For example, in the method provided in an embodiment of the present disclosure, the second loss function is a weighted sum of the error function and the discrimination function.

[0011] For example, in the method provided in an embodiment of the present disclosure, the weight of the error function is 90 to 110, and the weight of the discrimination function is 0.5 to 2.

[0012] For example, in the method provided in an embodiment of the present disclosure, the first sample data and the second sample data are image data obtained based on multiple videos with the same bit rate.

[0013] For example, in the method provided by an embodiment of the present disclosure, the second neural network to be trained is trained based on the pre-trained first neural network to obtain the trained second neural network, thereby obtaining the convolutional neural network, including: inputting third sample data into the first neural network and the second neural network respectively to obtain fifth data output from the first neural network and sixth data output from the second neural network; calculating a third loss function based on the fifth data and the sixth data; adjusting the parameters of the second neural network according to the third loss function to obtain an updated second neural network.

[0014] For example, in the method provided by an embodiment of the present disclosure, the first neural network includes multiple levels of downsampling units and corresponding multiple levels of upsampling units. The output of each level of downsampling unit is used as the input of the next level of downsampling unit. The input of each level of upsampling unit includes the output of the downsampling unit corresponding to this level of upsampling unit and the output of the previous level of upsampling unit corresponding to this level of upsampling unit.

[0015] At least one embodiment of the present disclosure further provides a terminal device, including a processor, wherein the processor is configured to: obtain an input video bit rate and an input video, where the input video includes multiple input image frames; select a video processing method corresponding to the input video bit rate according to the input video bit rate to process at least one of the multiple input image frames to obtain at least one output image frame, where the clarity of the at least one output image frame is higher than the clarity of the at least one input image frame; wherein different input video bit rates correspond to different video processing methods.

[0016] For example, in a terminal device provided in an embodiment of the present disclosure, the video processing method includes: processing the at least one input image frame based on a trained neural network to obtain the at least one output image frame; wherein, processing the at least one input image frame based on the trained neural network to obtain the at least one output image frame includes: extracting features from the at least one input image frame to obtain a plurality of first output images; performing splicing processing on the at least one input image frame and the plurality of first output images to obtain a first output image group, wherein the first output image group includes the at least one input image frame and the plurality of first output images; extracting features from the first output image group to obtain a plurality of second output images; fusing the plurality of second output images and the plurality of first output images to obtain a plurality of third output images; performing splicing processing on the at least one input image frame and the plurality of third output images to obtain a second output image group, wherein the second output image group includes the at least one input image frame and the plurality of third output images; extracting features from the second output image group to obtain the at least one output image frame.

[0017] For example, in a terminal device provided in an embodiment of the present disclosure, the trained neural networks corresponding to different video processing methods are different.

[0018] For example, in a terminal device provided in an embodiment of the present disclosure, the trained neural networks corresponding to different video processing methods are respectively trained using different sample data sets, and the different sample data sets are respectively obtained based on different video sets. Each video set includes a plurality of videos, and the videos in the same video set have the same bit rate, while the videos in different video sets have different bit rates.

[0019] At least one embodiment of the present disclosure further provides a video processing method, including: obtaining an input video bit rate and an input video, wherein the input video includes a plurality of input image frames; selecting a video processing sub-method corresponding to the input video bit rate according to the input video bit rate to process at least one of the plurality of input image frames to obtain at least one output image frame, wherein the clarity of the at least one output image frame is higher than that of the at least one input image frame; wherein different input video bit rates correspond to different video processing sub-methods.

[0020] At least one embodiment of the present disclosure further provides an image processing apparatus, including: a processor; a memory including one or more computer program modules; wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include those for implementing the image processing method according to any embodiment of the present disclosure.

[0021] At least one embodiment of the present disclosure further provides a storage medium for storing non - transient computer - readable instructions, which can implement the image - processing method described in any embodiment of the present disclosure when executed by a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the accompanying drawings of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description only relate to some embodiments of the present disclosure and do not limit the present disclosure.

[0023] Figure 1 It is a schematic diagram of a convolutional neural network;

[0024] Figure 2 It is a schematic flowchart of an image - processing method provided by some embodiments of the present disclosure;

[0025] Figure 3A It is a schematic diagram of the convolutional neural network adopted by an image - processing method provided by some embodiments of the present disclosure;

[0026] Figure 3B For Figure 2 It is a schematic flowchart of step S20 in the image - processing method shown;

[0027] Figure 4 It is a schematic flowchart of another image - processing method provided by some embodiments of the present disclosure;

[0028] Figure 5A It is a schematic flowchart of training a discrimination network in an image - processing method provided by some embodiments of the present disclosure;

[0029] Figure 5B For Figure 5A It is a schematic diagram of the scheme for training the discrimination network shown;

[0030] Figure 5C It is a discrimination network;

[0031] Figure 6A It is a schematic flowchart of training a second neural network in an image - processing method provided by some embodiments of the present disclosure;

[0032] Figure 6B For Figure 6A It is a schematic diagram of the scheme for training the second neural network shown;

[0033] Figure 7A It is a neural network with a denoising function;

[0034] Figure 7BIt is a neural network with a defuzzification function;

[0035] Figure 8A It is a schematic flowchart for training a second neural network in another image processing method provided by some embodiments of the present disclosure;

[0036] Figure 8B It is Figure 8A a schematic diagram of a solution for training the second neural network shown in;

[0037] Figure 9A It is a schematic block diagram of a terminal device provided by some embodiments of the present disclosure;

[0038] Figure 9B It is a schematic block diagram of another terminal device provided by some embodiments of the present disclosure;

[0039] Figure 10A It is a data flow diagram of a terminal device provided by some embodiments of the present disclosure;

[0040] Figure 10B It is an operation flowchart of a terminal device provided by some embodiments of the present disclosure;

[0041] Figure 11A It is a schematic diagram of a video frame;

[0042] Figure 11B It is for Figure 11A the frame shown in being processed by the terminal device provided by the embodiments of the present disclosure, and the resulting effect diagram;

[0043] Figure 12 It is a schematic block diagram of an image processing device provided by some embodiments of the present disclosure;

[0044] Figure 13 It is a schematic block diagram of another image processing device provided by some embodiments of the present disclosure;

[0045] Figure 14 It is a schematic diagram of a storage medium provided by some embodiments of the present disclosure; and

[0046] Figure 15 It is a schematic flowchart of a video processing method provided by some embodiments of the present disclosure. Detailed implementation manners

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present disclosure. Apparently, the described embodiments are only a part rather than all of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the described embodiments of the present disclosure without creative efforts shall fall within the scope of protection of the present disclosure.

[0048] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure shall have the ordinary meanings as understood by those of ordinary skill in the art to which the present disclosure pertains. The terms "first", "second", and similar terms used in the present disclosure do not denote any order, quantity, or importance, but are only used to distinguish different components. Similarly, terms such as "a", "an", or "the" do not denote a limitation of quantity, but rather indicate the presence of at least one. The terms "comprising" or "including" and similar terms mean that the elements or items appearing before the term cover the elements or items listed after the term and their equivalents, without excluding other elements or items. The terms "connected" or "coupled" and similar terms are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms "upper", "lower", "left", "right", etc. are only used to represent relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0049] With the development of deep learning technology, deep learning technology has been increasingly widely applied in the field of image enhancement. Image enhancement technology based on deep learning can be used for image processing, such as image denoising, image restoration, image deblurring, image super-resolution enhancement, image rain / fog removal, etc.

[0050] A convolutional neural network (CNN) is a commonly used neural network structure that can be used for image processing. Figure 1 is a schematic diagram of a convolutional neural network. For example, this convolutional neural network can be used for image processing, which uses an image as both input and output and replaces scalar weights with convolutional kernels. Figure 1 Only a convolutional neural network with a three-layer structure is shown in, and the embodiments of the present disclosure are not limited thereto. As Figure 1 shown, the convolutional neural network includes an input layer a01, a hidden layer a02, and an output layer a03. The input layer a01 has 4 inputs, the hidden layer a02 has 3 outputs, and the output layer a03 has 2 outputs. This convolutional neural network finally outputs 2 images. For example, the hidden layer a02 is also called the intermediate layer and is mainly used for feature extraction. The neurons therein can adopt various forms to form a bias for the output result.

[0051] For example, the four inputs of the input layer a01 can be four images, or four features of one image. The three outputs of the hidden layer a02 can be feature images of the images input through the input layer a01.

[0052] For example, as Figure 1 shown, the hidden layer a02 includes a first convolutional layer b01 and a first activation layer b03. In this example, the convolutional layer has weights and biases The weights represent convolutional kernels, and the biases are scalars added to the output of the convolutional layer, where k is a label indicating the number of the input layer a01 or the hidden layer a02, and i and j are the labels of the respective units in the two connected layers. For example, the first convolutional layer b01 includes a first set of convolutional kernels ( Figure 1 in ) and a first set of biases ( Figure 1 in ). Similarly, the output layer a03 includes a second convolutional layer b02 and a second activation layer b04. The second convolutional layer b02 includes a second set of convolutional kernels ( Figure 1 in ) and a second set of biases ( Figure 1 in ). Generally, each convolutional layer includes dozens or hundreds of convolutional kernels. If the convolutional neural network is a deep convolutional neural network, it can include at least five convolutional layers.

[0053] For example, as Figure 1 shown, the first activation layer b03 is located after the first convolutional layer b01, and the second activation layer b04 is located after the second convolutional layer b02. The activation layer includes an activation function, which is used to introduce non-linearity into the convolutional neural network so that the convolutional neural network can better solve more complex problems. The activation function can include a rectified linear unit function (ReLU function), a sigmoid function (Sigmoid function), or a hyperbolic tangent function (tanh function), etc. The ReLU function is a non-saturating non-linear function, and the Sigmoid function and the tanh function are saturating non-linear functions. For example, the activation layer can be a separate layer of the convolutional neural network, or the activation layer can also be included in the convolutional layer.

[0054] For example, in the first convolutional layer b01, first, a number of convolutional kernels in the first set of convolutional kernels and a number of biases in the first set of biases to obtain the output of the first convolutional layer b01; then, the output of the first convolutional layer b01 can be processed by the first activation layer b03 to obtain the output of the first activation layer b03. In the second convolutional layer b02, first, a number of convolutional kernels in the second set of convolutional kernels are applied to the output of the first activation layer b03 and a number of biases in the second set of biases to obtain the output of the second convolutional layer b02; then, the output of the second convolutional layer b02 can be processed by the second activation layer b04 to obtain the output of the second activation layer b04. For example, the output of the first convolutional layer b01 can be the result of applying a convolutional kernel to its input and then adding it to a bias The output of the second convolutional layer b02 can be the result of applying a convolutional kernel to the output of the first activation layer b03 and then adding it to a bias added.

[0055] Before performing image processing using a convolutional neural network, the convolutional neural network needs to be trained. After training, the convolutional kernels and biases of the convolutional neural network remain unchanged during image processing. During the training process, each convolutional kernel and bias are adjusted through multiple sets of input / output example images and optimization algorithms to obtain an optimized convolutional neural network model.

[0056] In the field of video playback, due to the limitation of transmission bandwidth, the video quality played by the TVs used by users is poor. There are mainly the following two reasons for the poor video quality. One is that the resolution of video images is relatively low. The resolution of TV channels or Internet videos is usually standard definition (360p or 720p). When the user's TV screen has a 4K resolution (2160p), the video images need to be interpolated and enlarged, resulting in a large loss of detailed information. On the other hand, due to the requirements of video live transmission bandwidth and video storage space, the original video needs to be compressed at a low bit rate. The compressed video also loses a large amount of detailed information and introduces a lot of noise.

[0057] As the central control component of a smart TV, the terminal video processor can operate the basic functions of the TV and can also have functions such as voice control and program recommendation. With the continuous improvement of chip computing power, to address the problem of poor video quality, the terminal video processor can integrate the function of enhancing video images. However, the video image enhancement function of the terminal video processor still mainly uses traditional and simple algorithms at present, and the ability to improve video quality is limited. The amount of computation of deep learning algorithms (such as convolutional neural networks) is large, and it is difficult for the terminal video processor to support the operation of a general convolutional neural network.

[0058] Therefore, how to design a convolutional neural network with a simple structure and related training methods to improve the clarity of images, so that the convolutional neural network can not only have a good image enhancement function, but also be easily deployed on terminal devices, has become an urgent problem to be solved.

[0059] At least one embodiment of the present disclosure provides an image processing method, an apparatus, a terminal device, a video processing method, and a storage medium. The convolutional neural network required by the image processing method has a simple structure, can save device computing power, can remove the noise of low-quality images and videos, improve the picture clarity, realize real-time picture quality enhancement, and is easily deployed on terminal devices. The image processing method provided by at least one embodiment also has a good neural network training effect.

[0060] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that the same reference numerals in different drawings will be used to refer to the same elements that have been described.

[0061] At least one embodiment of the present disclosure provides an image processing method, which is applicable to a convolutional neural network. The image processing method includes: receiving an input image; using the convolutional neural network to process the input image to obtain an output image. The clarity of the output image is higher than that of the input image. Using the convolutional neural network to process the input image to obtain an output image includes: extracting features from the input image to obtain a plurality of first images; performing a splicing process on the input image and the plurality of first images to obtain a first image group, where the first image group includes the input image and the plurality of first images; extracting features from the first image group to obtain a plurality of second images; fusing the plurality of second images and the plurality of first images to obtain a plurality of third images; performing a splicing process on the input image and the plurality of third images to obtain a second image group, where the second image group includes the input image and the plurality of third images; extracting features from the second image group to obtain the output image.

[0062] It should be noted that in the embodiments of the present disclosure, "clarity" refers to, for example, the clarity of each detail texture and its boundary in the image. The higher the clarity, the better the visual effect of the human eye. The clarity of the output image is higher than that of the input image, which means that, for example, the input image is processed by using the image processing method provided by the embodiments of the present disclosure, such as performing denoising and / or deblurring processing, so that the obtained output image is clearer than the input image.

[0063] Figure 2 FIG. is a flowchart of an image processing method provided by some embodiments of the present disclosure. For example, the image processing method is applicable to a convolutional neural network, that is, the image processing method uses the convolutional neural network to implement the processing of images. For example, as Figure 2 shown, the image processing method includes the following operations.

[0064] Step S10: Receive an input image;

[0065] Step S20: Process the input image by using a convolutional neural network to obtain an output image, where the clarity of the output image is higher than that of the input image.

[0066] For example, in step S10, the input image may be an image to be processed, such as an image with low clarity. The image to be processed may be a video frame extracted from a video, a picture downloaded through a network or taken by a camera, or an image obtained through other means, and the embodiments of the present disclosure do not limit this. There are, for example, a lot of noises in the input image and the image quality is relatively blurred. Therefore, it is necessary to use the image processing method provided by the embodiments of the present disclosure to denoise and / or deblur, so as to improve the clarity and realize image quality enhancement. For example, when the input image is a color image, the input image may include a red (R) channel input image, a green (G) channel input image, and a blue (B) channel input image. Here, the input image is a color image, which includes three RGB channels.

[0067] For example, in step S20, a convolutional neural network may be used to implement image processing, such as denoising and / or deblurring the input image, so that the clarity of the obtained output image is higher than that of the input image. For example, when the input image includes a red (R) channel input image, a green (G) channel input image, and a blue (B) channel input image, the output image may include a red (R) channel output image, a green (G) channel output image, and a blue (B) channel output image. Here, the output image is a color image, which includes three RGB channels.

[0068] Figure 3A Schematic diagram of a convolutional neural network adopted by an image processing method provided by some embodiments of the present disclosure. For example, in the image processing method provided by the embodiments of the present disclosure, the input image is input into Figure 3A the convolutional neural network shown, and the convolutional neural network processes the input image to obtain an output image, thereby completing image processing, so that the clarity of the output image is higher than that of the input image.

[0069] For example, as Figure 3A shown, the convolutional neural network includes an input layer INP, a first convolutional layer C1, a first splicing layer P1, a second convolutional layer C2, a fusion layer Fu1, a second splicing layer P2, and an output layer OT arranged in sequence.

[0070] For example, the input layer INP is used to receive an input image. When the input image is a black-and-white image, the input image may include one channel. When the input image is an RGB color image, the input image may include three channels. That is, there is 1 input image, but it includes a red (R) channel input image, a green (G) channel input image, and a blue (B) channel input image.

[0071] The first convolutional layer C1 is used to perform a convolutional operation on the input image received by the input layer INP to achieve feature extraction. For example, the first convolutional layer C1 includes multiple convolutional kernels, multiple biases, and an activation function, whereby multiple feature images (which can also be referred to as feature maps) can be calculated. The activation function is used to perform a non-linear mapping on the result of the convolutional operation to assist in expressing complex features. For example, the activation function in the first convolutional layer C1 uses the rectified linear unit (ReLU). Using the ReLU function is easier to converge and has better prediction performance. The ReLU function can be expressed as: y = max(0, x), where x represents the input to the ReLU function and y represents the output of the ReLU function. Of course, the embodiments of the present disclosure are not limited thereto, and the activation function can also use the Sigmoid function, the hyperbolic tangent function (tanh), or any other applicable function, which can be determined according to actual needs.

[0072] The first splicing layer P1 is used to use the input image received by the input layer INP and the feature image output by the first convolutional layer C1 as the input to the second convolutional layer C2. That is, the first splicing layer P1 is used to splice the input image received by the input layer INP and the feature image output by the first convolutional layer C1, and use the feature image obtained after the splicing process as the input to the second convolutional layer C2. For example, the first splicing layer P1 can use the concat function. The concat function can be expressed as: concat(str1, str2,...) = (str1, str2,...), where str1 represents the first image participating in the splicing, str2 represents the second image participating in the splicing, and so on. The concat function does not change the content of each image itself and only returns a copy of the spliced multiple images. For the specific description of the concat function, reference can be made to the conventional design and will not be elaborated here. Thus, the information of the input image can be linked with the information of the feature image output by the first convolutional layer C1, so that the feature image input to the second convolutional layer C2 contains the information of the input image (that is, contains the original image information).

[0073] The second convolutional layer C2 is used to perform a convolutional operation on the feature image output by the first splicing layer P1 to achieve feature extraction. For example, the second convolutional layer C2 includes a plurality of convolutional kernels, a plurality of biases, and an activation function, whereby a plurality of feature images can be calculated. For example, the activation function in the second convolutional layer C2 can also use the ReLU function. Of course, the embodiments of the present disclosure are not limited thereto. The activation function in the second convolutional layer C2 can also use the Sigmoid function, the tanh function, or any other applicable function, which can be determined according to actual needs. The activation function in the first convolutional layer C1 and the activation function in the second convolutional layer C2 can be the same or different, which can be determined according to actual needs, and the embodiments of the present disclosure do not limit this.

[0074] The fusion layer Fu1 is used to fuse the feature image output by the second convolutional layer C2 and the feature image output by the first convolutional layer C1. The fusion layer Fu1 can fuse the features of different layers in the convolutional neural network. For example, these different layers can be two connected layers, that is, the output of one layer is used as the input of another layer; these different layers can also be two layers spaced apart, that is, the output of any one layer is not used as the input of another layer. For example, any applicable fusion algorithm can be used to achieve the fusion of feature images, and the embodiments of the present disclosure do not limit this.

[0075] The second splicing layer P2 is used to jointly use the input image received by the input layer INP and the feature image output by the fusion layer Fu1 as the input of the output layer OT. That is, the second splicing layer P2 is used to perform a splicing process on the input image received by the input layer INP and the feature image output by the fusion layer Fu1, and use the feature image obtained after the splicing process as the input of the output layer OT. For example, the second splicing layer P2 can use the concat function. The concat function does not change the content of each image itself and only returns a copy of the spliced multiple images. Thus, the information of the input image can be linked with the information of the feature image output by the fusion layer Fu1, so that the feature image input to the output layer OT contains the information of the input image (that is, contains the original image information).

[0076] The output layer OT is used to perform a convolution operation on the feature image output by the second splicing layer P2 to convert the feature image into a multi-channel or single-channel image that meets the requirements, and output the image obtained by the operation as the output image of the convolutional neural network. For example, the output layer OT can perform a convolution operation to achieve feature extraction and channel conversion. For example, the output layer OT includes a plurality of convolutional kernels, a plurality of biases, and an activation function, whereby a feature image can be calculated as the output image. For example, the activation function in the output layer OT can also use the ReLU function. Of course, the embodiments of the present disclosure are not limited thereto, and the activation function in the output layer OT can also use the Sigmoid function, the tanh function, or any other applicable function, which can be determined according to actual needs. For example, the output layer OT may further include a fully connected layer, and the fully connected layer is used to perform a non-linear combination on the extracted features to obtain an output.

[0077] It should be noted that in the embodiments of the present disclosure, the above-mentioned convolutional neural network may further include more types of operation layers. For example, it may further include a pooling layer, etc., and the embodiments of the present disclosure do not limit this. For example, the convolutional layer is not limited to two layers (not limited to the first convolutional layer C1 and the second convolutional layer C2), and a third convolutional layer, a fourth convolutional layer, and more convolutional layers may also be provided. Similarly, the splicing layer is not limited to two layers (not limited to the first splicing layer P1 and the second splicing layer P2), and a third splicing layer, a fourth splicing layer, and more splicing layers may also be provided. Similarly, the fusion layer is not limited to 1 layer, and more fusion layers may also be provided.

[0078] It should be noted that in the embodiments of the present disclosure, the first convolutional layer C1, the second convolutional layer C2, and the output layer OT can also be used to perform pixel padding before the convolution operation to make the size of the image input to this layer the same as the size of the image output by this layer. For a detailed description of pixel padding, reference can be made to the conventional design, and details are not described here.

[0079] In this convolutional neural network, by setting a plurality of splicing layers (such as the first splicing layer P1 and the second splicing layer P2), the feature image information in the convolutional neural network is linked with the information of the input image, so that the feature image in the subsequent processing process contains the information of the input image (that is, contains the original image information), thereby making the convolutional neural network have a good detail restoration effect. When this convolutional neural network is trained to have a denoising and / or deblurring function, its denoising and / or deblurring effect is very good, thereby effectively improving the clarity of the image and realizing image enhancement. The structure of this convolutional neural network is simple, the operation is easy to implement, it can effectively save computing power, improve the operation efficiency, is convenient to be deployed on terminal devices, and is suitable for real-time image enhancement on mobile devices.

[0080] Figure 3B For Figure 2Schematic flowchart of step S20 in the image processing method shown. For example, as Figure 3B shown, in some examples, Figure 2 step S20 shown may specifically include the following operations.

[0081] Step S21: Extract features from the input image to obtain multiple first images;

[0082] Step S22: Perform splicing processing on the input image and the multiple first images to obtain a first image group, where the first image group includes the input image and the multiple first images;

[0083] Step S23: Extract features from the first image group to obtain multiple second images;

[0084] Step S24: Fuse the multiple second images and the multiple first images to obtain multiple third images;

[0085] Step S25: Perform splicing processing on the input image and the multiple third images to obtain a second image group, where the second image group includes the input image and the multiple third images;

[0086] Step S26: Extract features from the second image group to obtain the output image.

[0087] For example, the above steps S21 - S26 can be implemented by Figure 3A the convolutional neural network shown below. An exemplary description of steps S21 - S26 will be given in conjunction with Figure 3A the convolutional neural network shown.

[0088] For example, in step S21, the first convolutional layer C1 is used to extract features from the input image to obtain multiple first images. For example, the input image can be an RGB color image, and the input image includes three channels. That is, there is 1 input image, but it includes a red (R) channel input image, a green (G) channel input image, and a blue (B) channel input image. The first image is a feature image (which can also be called a feature map) obtained after being processed by the first convolutional layer C1. For example, the number of convolutional kernels used to extract features from the input image in this convolutional neural network (that is, the convolutional kernels in the first convolutional layer C1) is N, where 12 ≤ N ≤ 20 and N is an integer. For example, in some examples, N = 16. That is, the first convolutional layer C1 has 16 convolutional kernels. At this time, the number of multiple first images calculated by the first convolutional layer C1 is also 16.

[0089] For example, in step S22, the first splicing layer P1 is used to splice the input image and multiple first images to obtain a first image group. Here, "splicing process" refers to splicing multiple images using, for example, the concat function. The concat function does not change the content of each image itself and only returns a copy after splicing multiple images. For example, the first image group includes the input image and multiple first images. For example, in some examples, when the input image has three channels and the first convolutional layer C1 has 16 convolutional kernels, the number of first images is 16. The three channels of the input image are spliced with 16 first images to obtain the first image group, and the first image group includes a total of 19 images, where 16 are the feature images output by the first convolutional layer C1 (i.e., the aforementioned first images), and 3 are the three channels of the input image.

[0090] For example, in step S23, the second convolutional layer C2 is used to extract features from the first image group to obtain multiple second images. For example, the second image is a feature image (which can also be called a feature map) obtained after being processed by the second convolutional layer C2. For example, the number of convolutional kernels used in this convolutional neural network to extract features from the first image group (i.e., the convolutional kernels in the second convolutional layer C2) is M, where 12 ≤ M ≤ 20 and M is an integer. For example, in some examples, M = 16, that is, the second convolutional layer C2 has 16 convolutional kernels. At this time, the number of multiple second images calculated by the second convolutional layer C2 is also 16.

[0091] It should be noted that M and N can be equal or not equal. That is, the number of convolutional kernels in the second convolutional layer C2 and the number of convolutional kernels in the first convolutional layer C1 can be equal or not equal, which can be determined according to actual needs, and the embodiments of the present disclosure do not limit this. For example, in some examples, the number of convolutional kernels in both the first convolutional layer C1 and the second convolutional layer C2 is 16, which can not only take into account the computational magnitude that the mobile device can bear but also have a good image processing effect.

[0092] For example, in step S24, in some examples, the fusion layer Fu1 can be used to process multiple second images to obtain multiple third images. That is, the fusion layer Fu1 is used to fuse multiple second images and multiple first images to obtain multiple third images. For example, in some examples, when the first convolutional layer C1 has 16 convolutional kernels and the second convolutional layer C2 also has 16 convolutional kernels, there are 16 first images and 16 second images, and the first images and the second images can form a one-to-one correspondence. At this time, the fusion process can be, for example, performing an addition operation on the corresponding pixel points in the corresponding first image and second image.

[0093] It should be noted that in the embodiments of the present disclosure, in step S24, it is not limited to using the fusion layer Fu1 to obtain the third image. One or more convolutional layers, one or more splicing layers, one or more pooling layers, or any other operation layer can also be used to process the second image to obtain the third image. The embodiments of the present disclosure do not limit this.

[0094] For example, in step S25, the second splicing layer P2 is used to splice the input image and multiple third images to obtain a second image group. Here, "splicing process" means splicing multiple images using, for example, the concat function. The concat function does not change the content of each image itself and only returns a copy of the spliced multiple images. For example, the second image group includes the input image and multiple third images. For example, in some examples, when the input image has three channels and both the first convolutional layer C1 and the second convolutional layer C2 have 16 convolutional kernels, the second image group includes a total of 19 images, of which 16 are feature images output by the fusion layer Fu1 and 3 are the three channels of the input image.

[0095] For example, in step S26, the output layer OT is used to extract features from the second image group to obtain an output image. For example, the output layer OT is used to convert the second image group into a multi-channel or single-channel image that meets the requirements. For example, the output image is a feature image obtained after being processed by the output layer OT. For example, the number of convolutional kernels (i.e., the convolutional kernels in the output layer OT) used to extract features from the second image group in this convolutional neural network is 3. At this time, the number of output images calculated by the output layer OT is also 1, but it includes images of three channels: the red (R) channel output image, the green (G) channel output image, and the blue (B) channel output image. It should be noted that when the input image has only one channel, correspondingly, the number of convolutional kernels in the output layer OT is also set to 1, so that the obtained output image also has one channel.

[0096] For example, in some examples, the sizes of the convolutional kernels in the first convolutional layer C1, the second convolutional layer C2, and the output layer OT are all 3×3. This size can meet the requirements of feature extraction and also take into account computing power resources. Of course, the embodiments of the present disclosure are not limited to this. The size of the convolutional kernel can also be any size such as 4×4, 5×5, etc., and the sizes of the convolutional kernels in the first convolutional layer C1, the second convolutional layer C2, and the output layer OT can be the same or different, which can be determined according to actual needs. The embodiments of the present disclosure do not limit this.

[0097] For example, in some examples, the input image has three channels, namely the red-channel input image, the green-channel input image, and the blue-channel input image. It can be set that both the first convolutional layer C1 and the second convolutional layer C2 include 16 convolutional kernels, and the output layer OT includes 3 convolutional kernels, and the size of each convolutional kernel is 3×3. In this example, it is set that the batchsize of the network is 1, and the stride of the convolutional kernel is 1, that is, the size of the feature image remains unchanged, which makes the operation and processing simpler. It should be noted that both the batchsize and the stride of the convolutional kernel are not limited to 1 and can also be other values, which can be determined according to actual needs.

[0098] As Figure 3A shown, after passing through the first convolutional layer C1, the image changes from 3 channels to 16 channels (①), and the width and height information remains unchanged. Here, multiple "channels" can refer to multiple feature images. Before the second convolutional layer C2 starts to calculate, the first splicing layer P1 will link the original image information after the calculation result of the first convolutional layer C1 (②) and pass it as a common input to the second convolutional layer C2. The fusion layer Fu1 fuses the output of the second convolutional layer C2 (③) and the output of the first convolutional layer C1 (④), and then links the original image information through the second splicing layer P2 (⑤) and passes it to the output layer OT. The output layer OT learns the information of the RGB three channels, and the output of the output layer OT is the RGB image information (⑥).

[0099] For example, the output result of each layer can be expressed as (B, H, W, F). Wherein, B represents the batchsize, H represents the image height, W represents the image width, and F represents the number of feature images. The output results of the above-mentioned respective stages are specifically: ①: 1*H*W*16, ② 1*H*W*19, ③ 1*H*W*16, ④ 1*H*W*16, ⑤ 1*H*W*19, ⑥ 1*H*W*3.

[0100] After the processing of the above steps S21-S26, the input image is processed by using the convolutional neural network and the output image is obtained. The clarity of the output image is higher than that of the input image, thereby realizing image enhancement.

[0101] Figure 4 The flowchart of another image processing method provided by some embodiments of the present disclosure. For example, as Figure 4 shown, except that it further includes step S30, the image processing method provided by this embodiment is basically the same as the Figure 2 image processing method shown. For similar steps S10-S20, reference can be made to the foregoing content and will not be elaborated here.

[0102] In this embodiment, before performing steps S10 - S20, the image processing method further includes the following operations.

[0103] Step S30: Train the second neural network to be trained based on the pre - trained first neural network to obtain the trained second neural network, thereby obtaining a convolutional neural network.

[0104] For example, the first neural network has more parameters than the second neural network. The pre - trained first neural network is configured to transform the original image with a first sharpness input into the pre - trained first neural network into a new image with a second sharpness, where the second sharpness is greater than the first sharpness. That is, the first neural network has the function of enhancing image sharpness.

[0105] For example, in some examples, step S30 may include: alternately training the discriminator network and the second neural network based on the pre - trained first neural network, the second neural network to be trained, and the discriminator network to obtain the trained second neural network, thereby obtaining a convolutional neural network.

[0106] For example, the trained second neural network is the convolutional neural network described above. The network structure of the second neural network to be trained is the same as that of the convolutional neural network. Here, the network structure may refer to the number of each layer of the network, the setting method, and the setting order, etc., and may also refer to the data transmission path and method in the network, etc. The parameters of the second neural network to be trained are different from those of the convolutional neural network. That is, by training the second neural network to be trained and optimizing its parameters, the convolutional neural network described above can be obtained. For example, the parameters of the second neural network and the convolutional neural network both include the weight parameters in each convolutional layer. For example, the larger the absolute value of the weight parameter, the greater the contribution of the neuron corresponding to the weight parameter to the output of the neural network, and thus the more important it is to the neural network.

[0107] For example, the pre - trained first neural network can be trained in the following way. First, obtain the ground - truth samples (i.e., input videos with higher sharpness, including multiple input image frames), process the ground - truth samples (such as adding noise, blurring, etc.) to obtain the corresponding output videos with lower sharpness (including multiple output image frames), and the multiple output image frames correspond one - to - one with the multiple input image frames. Then, use the corresponding input image frames and output image frames as a set of training pairs to train the untrained first neural network to obtain the trained first neural network. For example, the multiple input image frames and / or multiple output image frames obtained at this time can be used as training samples for subsequent training of the second neural network.

[0108] For example, in some examples, in step S30, the sample data is input into the first neural network and the second neural network. Based on the results output by the first neural network and the second neural network, the discrimination network is first trained, and then the second neural network is trained. Then, the sample data is input into the first neural network and the second neural network again. Based on the results output by the first neural network and the second neural network, the discrimination network is first trained, and then the second neural network is trained, and so on. For example, the sample data input in different training stages can be different. By alternately training the discrimination network and the second neural network, the discrimination network and the second neural network learn from each other adversarially, so as to produce better output results.

[0109] For example, the first neural network can be a relatively large neural network that has been trained. The second neural network can be established based on the structure of the convolutional neural network described above, but the parameters still need to be trained. For example, the first neural network is used to train the second neural network, and the first neural network has more parameters than the second neural network. For example, the parameters of the neural network include the weight parameters of each convolutional layer in the neural network. The larger the absolute value of the weight parameter, the greater the contribution of the neuron corresponding to the weight parameter to the output of the neural network, and thus the more important it is to the neural network. Generally, the more parameters a neural network has, the higher its complexity and the larger its "capacity", which means that the neural network can complete more complex learning tasks. Compared with the first neural network, the second neural network is simplified, with fewer parameters and a simpler network structure, so that the second neural network occupies less resources (such as computing resources, storage resources, etc.) during its operation, and thus can be applied to lightweight terminals. By using the above training method, the second neural network can learn the reasoning ability of the first neural network, so that the second neural network has strong reasoning ability while having a simple structure.

[0110] Figure 5A The flowchart shows the training process of the discrimination network in an image processing method provided by some embodiments of the present disclosure. Figure 5B is Figure 5A The schematic diagram of the training scheme for the discrimination network shown below. The training scheme for the discrimination network will be described exemplarily in combination with Figure 5A and Figure 5B The training scheme for the discrimination network will be described exemplarily.

[0111] For example, in some examples, as Figure 5A and Figure 5B shown, the training of the discrimination network includes the following operations.

[0112] Step S31: Input the first sample data into the first neural network NE1 and the second neural network NE2 respectively, to obtain the first data output from the first neural network NE1 and the second data output from the second neural network NE2;

[0113] Step S32: Set the first data with a true value label, and input the first data with the true value label into the discriminative network Disc to obtain a first discrimination result. Set the second data with a false value label, and input the second data with the false value label into the discriminative network Disc to obtain a second discrimination result;

[0114] Step S33: Calculate a first loss function based on the first discrimination result and the second discrimination result;

[0115] Step S34: Adjust the parameters of the discriminative network Disc according to the first loss function to obtain an updated discriminative network Disc.

[0116] For example, in Step S31, the first sample data may be image data obtained based on a video, and both the first data and the second data are image data. Of course, the first sample data may also be image data obtained by other means, and the embodiments of the present disclosure do not limit this. For example, in some examples, the first sample data is image data obtained based on multiple videos with the same bit rate. Thus, the convolutional neural network trained using this first sample data can have better processing capabilities and effects on the image data of videos with this bit rate, making the convolutional neural network highly targeted at the bit rate. For example, in some examples, the original video can be compressed in video quality at different bit rates, and Gaussian noise and quantum noise can be randomly added to form a low-quality video. Video frames are extracted from the low-quality video, and thus the first sample data can be obtained, and this first sample data is the video frame of the low-quality video.

[0117] For example, the first neural network NE1 can be a pre-trained, large neural network with the function of enhancing clarity. For example, the first neural network NE1 includes multiple levels of downsampling units and corresponding multiple levels of upsampling units. The output of each level of downsampling unit is used as the input of the next level of downsampling unit. The input of each level of upsampling unit includes the output of the downsampling unit corresponding to this level of upsampling unit and the output of the previous level of upsampling unit corresponding to this level of upsampling unit. This neural network can effectively capture the feature information in the image and enhance the clarity of the image by using multiple downsampling units to perform multiple feature extractions on the image and using multiple upsampling units to perform multiple upsamplings on the image. Each feature image output by each downsampling unit is input into the corresponding upsampling unit. For example, the first neural network NE1 can adopt Figure 7AThe network structure shown will be described later, and no further elaboration will be provided here. It should be noted that the first neural network NE1 can be any pre-trained neural network or any combination of neural networks, which can be determined according to actual needs, and the embodiments of the present disclosure do not limit this. For example, the first neural network NE1 is used to enhance the clarity of sample data so that the trained convolutional neural network also has the function of enhancing clarity. For example, enhancing clarity can be specifically realized as denoising and / or deblurring to achieve image enhancement.

[0118] For example, the second neural network NE2 is established based on the network structure of the convolutional neural network described above. That is, the second neural network NE2 has the same network structure as the convolutional neural network described above, but its parameters still need to be trained and corrected.

[0119] For example, in step S32, the first data is set to have a true value label, and the second data is set to have a false value label. For example, in some examples, the true value label can be represented as [1], and the false value label can be represented as [0]. The first data with the true value label is input into the discriminative network Disc to obtain a first discrimination result, and the second data with the false value label is input into the discriminative network Disc to obtain a second discrimination result.

[0120] For example, the discriminative network Disc can adopt a convolutional neural network model in a stack mode. Figure 5C It is a discriminative network, and this discriminative network adopts a convolutional neural network model in a stack mode, for example. As Figure 5C shown, the discriminative network Disc includes multiple operation layers, and each operation layer is composed of a convolutional layer and an activation layer (using the ReLU function), and finally the result is output through the fully connected layer FC. For example, in this example, the convolutional kernel sizes of the convolutional layers in the 4 operation layers are all 3×3, and the numbers of feature images output by the 4 operation layers are 32, 64, 128, and 192 respectively. The result finally output by the discriminative network Disc is a binary classification probability value. That is, the discriminative network Disc discriminates and outputs any value between 0 and 1. The closer this value is to 1, the greater the probability that the input is judged to be true, and vice versa, the greater the probability that the input is judged to be false.

[0121] It should be noted that the embodiments of the present disclosure are not limited to this. The discriminative network Disc can also adopt any type of discriminative network, and the relevant description can refer to the conventional design, which will not be elaborated here.

[0122] For example, as Figure 5A and Figure 5BAs shown, in step S33, a first loss function is calculated based on the first discrimination result and the second discrimination result, and this first loss function is the loss function of the discrimination network Disc. For example, in some examples, the first loss function can adopt a cross-entropy loss function, and the cross-entropy loss function represents the difference between the true sample label and the predicted probability. For example, the formula for the cross-entropy loss function is:

[0123]

[0124] where N is the number of data (that is, the total number of the first data and the second data), y i is the label corresponding to each data (that is, 0 or 1), and p i is the predicted value of the discrimination network for each data.

[0125] It should be noted that in the embodiments of the present disclosure, the loss function of the discrimination network Disc can adopt any type of function, not limited to the cross-entropy loss function, which can be determined according to actual needs, and the embodiments of the present disclosure do not limit this.

[0126] For example, in step S34, the parameters of the discrimination network Disc are adjusted according to the first loss function to obtain an updated discrimination network Disc, and this updated discrimination network Disc has better discrimination ability.

[0127] Through the above steps S31 - S34, one training for the discrimination network Disc can be completed.

[0128] Figure 6A This is a schematic flowchart of training the second neural network in an image processing method provided by some embodiments of the present disclosure, Figure 6B is Figure 6A a schematic diagram of the training scheme for the second neural network shown.

[0129] Next, in combination with Figure 6A and Figure 6B an exemplary description of the training scheme for the second neural network will be given.

[0130] For example, in some examples, as Figure 6A and Figure 6B shown, training the second neural network includes the following operations.

[0131] Step S35: Input the second sample data into the first neural network NE1 and the second neural network NE2 respectively, and obtain the third data output from the first neural network NE1 and the fourth data output from the second neural network NE2;

[0132] Step S36: Set the fourth data to have a true label, and input the fourth data with the true label into the updated discriminative network Disc to obtain a third discrimination result output from the discriminative network Disc;

[0133] Step S37: Calculate an error function based on the third data and the fourth data, calculate a discrimination function based on the third discrimination result, and calculate a second loss function based on the error function and the discrimination function;

[0134] Step S38: Adjust the parameters of the second neural network NE2 according to the second loss function to obtain an updated second neural network NE2.

[0135] The above steps S35 - S38 can be executed after steps S31 - S36 are completed.

[0136] For example, in step S35, the second sample data may be image data obtained based on a video, and both the third data and the fourth data are image data. Of course, the second sample data may also be image data obtained by other means, and the embodiments of the present disclosure do not limit this. For example, in some examples, the second sample data is image data obtained based on multiple videos with the same bit rate. Thus, the convolutional neural network trained using this second sample data can have better processing capabilities and effects on the image data of videos with this bit rate, making the convolutional neural network highly targeted at the bit rate. For example, in some examples, the original video can be subjected to video quality compression at different bit rates, and Gaussian noise and quantum noise can be randomly added to form a low - quality video, and video frames can be extracted from the low - quality video, thereby obtaining the second sample data, which is the video frame of the low - quality video. For example, the first sample data and the second sample data may be the same or different.

[0137] For example, in step S36, set the fourth data to have a true label (e.g., represented as [1]), and input the fourth data with the true label into the discriminative network Disc to obtain a third discrimination result output from the discriminative network Disc. For example, the value range of the third discrimination result is 0 - 1. It should be noted that the discriminative network Disc at this time is the discriminative network updated after the above steps S31 - S34.

[0138] For example, in step S37, an error function is calculated based on the third data and the fourth data. For example, in some examples, the error function can adopt the mean absolute error (L1 loss). For example, the calculation formula of the mean absolute error is as follows: L1loss = mean(|X - Y|), where X is the fourth data and Y is the third data. It should be noted that the error function is not limited to the mean absolute error (L1 loss), and can also be any other applicable error function, which can be determined according to actual needs, and the embodiments of the present disclosure do not limit this. The discrimination function D2 is calculated based on the third discrimination result. For example, any applicable method can be used to calculate the discrimination function D2, and the embodiments of the present disclosure do not limit this.

[0139] After calculating the error function (such as the mean absolute error) and the discrimination function D2, the second loss function is calculated based on the error function (such as the mean absolute error) and the discrimination function D2. The second loss function is the loss function of the second neural network NE2. For example, the second loss function is the weighted sum of the error function (such as the mean absolute error) and the discrimination function D2, and can be expressed as: NE1 loss = W1 * L1loss + W2 * D2, where NE1 loss represents the second loss function, W1 represents the weight of the error function, and W2 represents the weight of the discrimination function D2. For example, the weight W1 of the error function is 90 to 110 (for example, 100), and the weight of the discrimination function D2 is 0.5 to 2 (for example, 1).

[0140] For example, in step S38, the parameters of the second neural network NE2 are adjusted according to the second loss function to obtain the updated second neural network NE2.

[0141] Through the above steps S35 - S38, one training for the second neural network NE2 can be completed.

[0142] For example, through the above steps S31 - S38, an adversarial training for the discriminative network Disc and the second neural network NE2 can be completed alternately. It should be noted that each alternate training is carried out on the basis of the update of the previous training. That is, when training the second neural network NE2, the discriminative network Disc updated based on the previous training is used for training, and when training the discriminative network Disc, the second neural network NE2 updated based on the previous training is used for training. According to requirements, the adversarial training for the discriminative network Disc and the second neural network NE2 can be carried out alternately once or multiple times. Through optimization and iteration, the image processing ability of the trained second neural network NE2 can meet the requirements. For example, in some examples, about 20 million alternate trainings are carried out in total, and the second neural network NE2 that meets the requirements can be obtained. For example, the parameters of the trained second neural network NE2 (such as the weight parameters in the convolutional layer) have been optimized and corrected, and the trained second neural network NE2 is the convolutional neural network described above.

[0143] For example, in some examples, in an adversarial training for the discriminative network Disc and the second neural network NE2, the first sample data and the second sample data used can be the same. That is, the same sample data can be used to complete a training for the discriminative network Disc and a training for the second neural network NE2. For example, in the case of carrying out multiple adversarial trainings, for the same adversarial training, the first sample data and the second sample data used can be the same. For example, when an adversarial training is completed, in the subsequent second adversarial training for the discriminative network Disc and the second neural network NE2, the first sample data used is different from the first sample data used in the previous adversarial training, and the second sample data used is different from the second sample data used in the previous adversarial training. In this way, the training efficiency can be improved, the training method can be simplified, and the data utilization rate can be increased.

[0144] In the image processing method provided by the embodiments of the present disclosure, by adopting the above training method, a convolutional neural network that meets the requirements can be quickly trained, and the training effect is good. The trained convolutional neural network can have strong image processing ability and good image processing effect. When the first neural network NE1 adopted has denoising and / or deblurring functions, the trained convolutional neural network also has denoising and / or deblurring functions, and the denoising and / or deblurring effects are very good.

[0145] It should be noted that in the embodiments of the present disclosure, the image processing method may further include more steps, and the execution order of each step can be adjusted according to actual requirements. The embodiments of the present disclosure do not limit this.

[0146] For example, the first neural network NE1 can adopt any trained neural network. For example, it can adopt a larger neural network with a denoising function or a larger neural network with a deblurring function. Or, it can also adopt a combination of a larger neural network with a denoising function and a larger neural network with a deblurring function.

[0147] Figure 7A It is a neural network with a denoising function. As Figure 7A shown, this neural network includes multiple levels of downsampling units and corresponding multiple levels of upsampling units, and the multiple levels of downsampling units correspond to the multiple levels of upsampling units one by one. For example, in Figure 7A , the unit on the left is the downsampling unit, and the unit on the right is the upsampling unit. The output of each level of downsampling unit serves as the input of the next level of downsampling unit, and the input of each level of upsampling unit includes the output of the downsampling unit corresponding to this level of upsampling unit and the output of the previous level of upsampling unit of this level of upsampling unit. That is to say, the output of the downsampling unit is provided not only to the adjacent next-level downsampling unit, but also to the upsampling unit corresponding to this downsampling unit.

[0148] For example, each downsampling unit includes operation layers such as a Conv2d convolutional layer, a Relu activation function, and a Conv2d downsampling by 2 times, and each upsampling unit includes operation layers such as a Conv2d convolutional layer, a Relu activation function, and a Conv2d upsampling by 2 times. This neural network simultaneously inputs three consecutive frames of images, and the output is the result of the middle frame of the input. Assume that the size of the input frame sequence is (H, W, C). If the input is an RGB image, then C = 3; if the input is a grayscale image, then C = 1. When three frames of images are input simultaneously and merged in the C channel, the input size of the network model is (H, W, C * 3).

[0149] This neural network has a U-shaped symmetric structure. The left side is mainly composed of downsampling units, and the right side is mainly composed of upsampling units. In this neural network, multiple downsampling units and multiple upsampling units correspond to each other. The feature image output by each downsampling unit on the left is input into the corresponding upsampling unit on the right, so that the feature maps obtained at each level can be effectively used in subsequent calculations.

[0150] In this neural network, the batch size of the network is set to 1 (B = 1), and the number of feature layers in each layer of the network is F. The Conv2d convolutional layer selects a convolutional kernel of (3, 3), and the stride is 1, that is, the size of the feature map remains unchanged, so the output size is (B, H, W, F). The Conv2d downsampling by 2 times selects a convolutional kernel of (3, 3), and at the same time the stride is 2, that is, downsampling by two times, and the output size is (B, H / / 2, W / / 2, F). The Conv2d upsampling by 2 times selects a convolutional kernel of (4, 4), and the stride is 2, that is, upsampling by 2 times, and the output size is (B, H, W, F). The Conv2d feature to RGB map (parameter sharing) selects a (3, 3) convolutional kernel, and the stride is 1. The main purpose is to change the number of feature layers from F to 3 to obtain the output RGB map.

[0151] It should be noted that the parameters of this convolutional kernel are shared, that is, the parameters of all Conv2d convolutional kernels in the figure are the same. During the training process, five images with different scales, namely F1, F2, F3, F4, and F5, are output respectively. For example, the sizes of these five images with different scales are: F1(H, W, 3), F2(H / / 2, W / / 2, 3), F3(H / / 4, W / / 4, 3), F4(H / / 8, W / / 8, 3), F5(H / / 16, W / / 16, 3).

[0152] During the training process of this neural network, the loss functions are calculated between F1, F2, F3, F4, and F5 output at different scales and the ground truth respectively. For example, GT1 is the ground truth. In order to obtain the ground truth maps of other scales, GT1 is downsampled by BICUBIC to obtain GT2(H / / 2, W / / 2, 3), GT3(H / / 4, W / / 4, 3), GT4(H / / 8, W / / 8, 3), and GT5(H / / 16, W / / 16, 3) respectively. When using this neural network after training, only the Conv2d convolutional layer with parameter sharing is used when finally outputting F1, and F2, F3, F4, and F5 are no longer output, that is, this convolutional layer is not used at F2, F3, F4, and F5.

[0153] Figure 7B It is a neural network with a deblurring function. As Figure shown, this neural network includes multiple functional layer groups with different numbers of convolutional kernels. Each functional layer group includes one or more Conv convolutional layers, DConv convolutional layers, and re-blocking layers, etc. Moreover, in this network, skip connections are established between different functional layer groups. By setting the skip connections, the problem of gradient disappearance in the case of a relatively deep network layer can be solved, and at the same time, it helps the backpropagation of gradients and speeds up the training process.

[0154] For the description of this neural network, reference can be made to the neural network applying skip connections and re-blocking layers in the conventional design, which will not be elaborated here.

[0155] It should be noted that in the embodiments of the present disclosure, the trained convolutional neural network can process the input image to improve the clarity of the image, but is not limited to only having denoising and / or deblurring functions. The convolutional neural network can also have any other functions, and only the first neural network NE1 with the corresponding function needs to be adopted and the second neural network NE2 is trained using the first neural network NE1.

[0156] ​ FIG. is a schematic flowchart of training the second neural network in another image processing method provided by some embodiments of the present disclosure. ​ For ​ FIG. is a schematic diagram of the training scheme for training the second neural network. For example, in this example, ​ and ​ shown in the training method can be used to train the convolutional neural network. Different from the training method shown in ​ FIG., in this example, the discriminative network is no longer adopted, and only the first neural network NE1 is used to train the second neural network NE2 to obtain the required convolutional neural network.

[0157] Next, in combination with ​ and ​ an exemplary description of the training scheme of the second neural network will be given.

[0158] For example, in some examples, as ​ and ​ shown, training the second neural network includes the following operations.

[0159] Step S41: Input the third sample data into the first neural network NE1 and the second neural network NE2 respectively, and obtain the fifth data output from the first neural network NE1 and the sixth data output from the second neural network NE2;

[0160] Step S42: Calculate the third loss function based on the fifth data and the sixth data;

[0161] Step S43: Adjust the parameters of the second neural network NE2 according to the third loss function to obtain the updated second neural network NE2.

[0162] For example, first, select a set of video datasets with good clarity, add a certain amount of blur and noise, and then compress them to a bit rate of 1M to reduce the video quality so that the processed videos meet the standard definition video standards in actual video websites. Extract video frames from the processed video datasets, and the obtained video frames form sample data. For example, the video datasets with good clarity can use the AIM dataset, which contains 240 videos of 1280*720, and each video has 100 frames.

[0163] For example, in this example, the first neural network NE1 includes two larger neural networks NE1a and NE1b arranged in sequence. NE1a and NE1b are, for example, ​ the neural network shown and ​ the neural network shown. The setting order of these two neural networks is not restricted.

[0164] For example, in step S41, the third sample data is input into the first neural network NE1. One neural network NE1a in the first neural network NE1 processes the third sample data and then inputs the processing result into another neural network NE1b in the first neural network NE1. The other neural network NE1b processes the received image and outputs the processing result as the output of the first neural network NE1. Thus, the fifth data output by the first neural network NE1 has been denoised and deblurred, and the first neural network NE1 has the functions of both denoising and deblurring. The fifth data is used as the ground truth image. The third sample data is input into the second neural network NE2, and the sixth data output by the second neural network NE2 is used as the false value image.

[0165] For example, in steps S42 and S43, the third loss function is calculated based on the fifth data and the sixth data, and backpropagation is performed using the third loss function to adjust the parameters of the second neural network NE2. Thus, the updated second neural network NE2 is obtained, that is, the required convolutional neural network is obtained.

[0166] For example, in this example, the third loss function can adopt the weighted sum of the mean absolute error (L1 loss) and the Sobel error (Sobel loss). The calculation formula of the mean absolute error is as follows: L1 loss = mean(|X - Y|), where X is the sixth data and Y is the fifth data. The calculation formula of the Sobel error is: Sobel loss = mean(|sobel_edge(gray(X)) - sobel_edge(gray(Y))|), where X is the sixth data, Y is the fifth data, and sobel_edge() represents the Sobel edge. After converting X and Y to the grayscale domain, the Sobel edge is extracted, and then the mean is calculated after taking the difference. For example, in some examples, the weight of the mean absolute error (L1 loss) can be set to 0.5 to 1.5 (e.g., 1), and the weight of the Sobel error (Sobel loss) can be set to 1.5 to 2.5 (e.g., 2), so that better training results can be obtained.

[0167] For example, the Sobel error (also known as the Sobel operator) is one of the most important operators in pixel image edge detection. This operator contains two groups of 3*3 matrices, namely horizontal and vertical. Convolving it with the image can respectively obtain the approximate values of the brightness differences in the horizontal and vertical directions. For example, using Gx and Gy to represent the approximate values of the gray-scale partial derivatives in the horizontal and vertical directions respectively, the calculation formula of the Sobel operator is as follows:

[0168]

[0169] In the above Sobel operator, A represents the image. For each pixel point, gradients in the x and y directions can be obtained, and the estimated value of the gradient can be calculated through the following formula: For example, a threshold Gmax can be defined. If G is smaller than Gmax, it can be considered that this point is a boundary value, then this point is retained and set to white, otherwise this point is set to black. Thus, the image gradient information is obtained. During the training process, both the fifth data and the sixth data are converted to the grayscale domain, the Sobel gradients are calculated respectively, and the mean difference of the gradient map is calculated as the loss value for backpropagation.

[0170] In this example, since the first neural network NE1 has both denoising and deblurring functions, the trained convolutional neural network can simultaneously learn the denoising and deblurring functions, can well restore the clarity of the video frame, and at the same time maintain the image information. By adopting the above training method, it is possible to quickly train a convolutional neural network that meets the requirements with only less sample data.

[0171] At least one embodiment of the present disclosure further provides a terminal device, which can process an input video based on a convolutional neural network to improve the picture clarity, achieve real-time picture quality enhancement, with good processing effect and high processing efficiency.

[0172] ​ It is a schematic block diagram of a terminal device provided by some embodiments of the present disclosure. For example, as ​ shown, the terminal device 100 includes a processor 110.

[0173] For example, the processor 110 is configured to: obtain an input video bit rate and an input video, where the input video includes a plurality of input image frames; select a video processing method corresponding to the input video bit rate according to the input video bit rate to process at least one input image frame among the plurality of input image frames to obtain at least one output image frame. For example, the clarity of at least one output image frame is higher than that of at least one input image frame, and different input video bit rates correspond to different video processing methods.

[0174] For example, the above video processing method includes: processing at least one input image frame based on a trained neural network to obtain at least one output image frame. For example, ​ the shown convolutional neural network can be used to implement the processing of the input image frame. This neural network has a denoising and / or deblurring function, can effectively improve the clarity of the image, has a good detail repair effect, and can achieve image enhancement. For example, the image processing method provided by any embodiment of the present disclosure can be used to implement the processing of the input image frame.

[0175] For example, processing at least one input image frame based on a trained neural network to obtain at least one output image frame may include the following operations:

[0176] Performing feature extraction on at least one input image frame to obtain a plurality of first output images;

[0177] Performing splicing processing on at least one input image frame and the plurality of first output images to obtain a first output image group, where the first output image group includes at least one input image frame and the plurality of first output images;

[0178] Performing feature extraction on the first output image group to obtain a plurality of second output images;

[0179] Fusing the plurality of second output images and the plurality of first output images to obtain a plurality of third output images;

[0180] Performing splicing processing on at least one input image frame and the plurality of third output images to obtain a second output image group, where the second output image group includes at least one input image frame and the plurality of third output images;

[0181] Feature extraction is performed on the second output image group to obtain at least one output image frame.

[0182] For example, the steps of processing the input image frame described above are basically the same as ​ the steps shown, and the relevant descriptions can be referred to the foregoing content, which will not be elaborated here.

[0183] For example, the trained neural networks corresponding to different video processing methods are different. That is to say, different input video bitrates correspond to different video processing methods, different video processing methods correspond to different neural networks, and input videos with different input video bitrates are processed using different neural networks. The neural network has bitrate pertinence. By distinguishing the bitrates, the neural network can have a better processing effect on videos corresponding to the corresponding bitrates. Moreover, input videos with different bitrates are processed using different video processing methods (or different neural networks), so that the output videos obtained after processing the input videos with different bitrates have relatively consistent clarity, and the processing effect is not affected by the bitrate, thereby improving the performance stability and consistency of the terminal device 100.

[0184] For example, the trained neural networks corresponding to different video processing methods are respectively trained using different sample data sets, and different sample data sets are respectively obtained based on different video sets. Each video set includes multiple videos. The videos in the same video set have the same bitrate, and the videos in different video sets have different bitrates. That is to say, for the same neural network, the sample data set used for its training comes from videos with the same bitrate; for different neural networks, the sample data sets used for their training come from videos with different bitrates. The bitrate of a video refers to the number of data bits transmitted per unit time during data transmission, and the unit is usually kbps, that is, kilobits per second. The higher the bitrate, the smaller the compression ratio of the video, the smaller the loss of picture quality, the smaller the noise of the image, and the closer it is to the original video. The lower the bitrate, the greater the noise of the image. Correspondingly, the denoising intensity of the neural network corresponding to the low bitrate is large, and the denoising intensity of the neural network corresponding to the high bitrate is small. It should be noted that the "high" and "low" in the above description are both relative, that is, the high bitrate and the low bitrate are relative to each other.

[0185] ​ A schematic block diagram of another terminal device provided by some embodiments of the present disclosure. For example, as ​ shown, the terminal device 200 is implemented as a terminal video processor 200, for example, and includes a hardware development board 210. A software development package 211 is deployed on the hardware development board 210. The hardware development board 210 includes a central processing unit 212 and a neural network processor 213.

[0186] For example, the software development kit 211 can be implemented as a collection of programs and related files with a general interface or a custom interface, that is, a Software Development Kit (SDK). For example, the software development kit 211 can be deployed in the on-chip memory (ROM) of the hardware development board 210 and read from the ROM during runtime.

[0187] For example, the software development kit 211 includes multiple neural network modules UN, and the multiple neural network modules UN are respectively obtained based on multiple convolutional neural networks, and the multiple neural network modules UN correspond one-to-one with the multiple convolutional neural networks. For example, the convolutional neural network can be ​ the convolutional neural network shown in the figure. This convolutional neural network has a denoising and / or deblurring function, can effectively improve the clarity of the image, has a good detail restoration effect, and can achieve image enhancement.

[0188] For example, the multiple neural network modules UN are obtained by performing parameter quantization on the multiple convolutional neural networks. For example, the data type can be converted from 32-bit floating-point number (float32) to 8-bit integer (int8), so as to achieve parameter quantization, effectively save computing power, and enable the terminal video processor 200 to support the operation of the neural network module UN. For example, a convolutional neural network can be trained at the precision of float32 (for example, trained using ​ the training method shown in the figure), and then the trained convolutional neural network is subjected to parameter quantization, and the data type is converted to int8, so as to obtain the neural network module UN. The obtained neural network module UN has the same function as the convolutional neural network. Although there is a slight difference in the processing effect between the neural network module UN and the convolutional neural network due to parameter quantization, it is difficult for the human eye to perceive, and this quality loss can be ignored.

[0189] Through parameter quantization, the amount of computation and data can be effectively reduced, so that the neural network module UN is suitable for running on the terminal video processor 200, and computing power and memory can be saved. For example, in the embodiments of the present disclosure, when the parameter compression is about 300 times, the output of the neural network module UN can still preferably maintain the image quality. Therefore, the neural network module UN can be deployed on the terminal video processor 200, providing the possibility for realizing real-time image enhancement.

[0190] For example, different neural network modules UN are used to process video data with different bit rates. For example, multiple ​The convolutional neural networks shown have the same structure but different parameters. For example, different convolutional neural networks are respectively trained using different sample data sets, and different sample data sets are respectively obtained based on different video sets. Each video set includes multiple videos. The videos in the same video set have the same bit rate, and the videos in different video sets have different bit rates. That is, for the same convolutional neural network, the sample data set used for its training comes from videos with the same bit rate; for different convolutional neural networks, the sample data sets used for their training come from videos with different bit rates. The bit rate of a video refers to the number of data bits transmitted per unit time during data transmission, and the unit is usually kbps, that is, thousands of bits per second. The higher the bit rate, the smaller the compression ratio of the video, the smaller the loss of picture quality, the smaller the noise of the image, and the closer it is to the original video. The lower the bit rate, the greater the noise of the image. Accordingly, the denoising intensity of the neural network corresponding to the low bit rate is large, and the denoising intensity of the neural network corresponding to the high bit rate is small. It should be noted that the "high" and "low" in the above description are relative, that is, the high bit rate and the low bit rate are compared with each other.

[0191] By differentiating the bit rates, the multiple convolutional neural networks obtained by training have bit rate pertinence and have better processing effects on videos corresponding to the bit rates. For example, several convolutional neural networks corresponding to several common video bit rates can be trained. By quantifying the parameters of these convolutional neural networks, the multiple neural network modules UN obtained also have bit rate pertinence. Different neural network modules UN are used to process video data with different bit rates to achieve better processing effects.

[0192] For example, the central processing unit 212 is configured to: call the software development kit 211 and select one neural network module UN from multiple neural network modules UN according to the bit rate of the input video, and control the neural network processor 213 to process the input video based on the selected neural network module UN to improve the clarity of the input video.

[0193] For example, multiple neural network modules UN are used to process video data with different bit rates, and each neural network module UN is used to process video data corresponding to the bit rate. According to the bit rate of the input video, the corresponding neural network module UN is selected, and the input video is processed using this neural network module UN, thereby improving the clarity of the input video and realizing real-time picture quality enhancement.

[0194] For example, the hardware development board 210 may be an ARM development board. Correspondingly, the central processing unit 212 may be a central processing unit (CPU) with an ARM architecture. For example, the neural network processor 213 may be a neural-network processing unit (NPU) applicable to the ARM development board. It adopts an architecture of data-driven parallel computing and is suitable for processing massive multimedia data such as videos and images. It should be noted that in the embodiments of the present disclosure, the types and hardware structures of the hardware development board 210, the central processing unit 212, and the neural network processor 213 are not limited, and any applicable hardware can be used, which can be determined according to actual needs, as long as the corresponding functions can be achieved. The embodiments of the present disclosure do not limit this. For example, the hardware development board 210 may use the Android system or the Linux system for hardware integration, or may also use the operating system in a smart TV or a set-top box for integration. The embodiments of the present disclosure do not limit this.

[0195] For example, the terminal video processor 200 may be implemented as a component in a television terminal (such as a smart TV), or may be implemented as a set-top box, or may also be implemented as a component in a video playback device or in any other arbitrary form. The embodiments of the present disclosure do not limit this.

[0196] It should be noted that in the embodiments of the present disclosure, the terminal device 200 may also include more components, structures, and modules, and is not limited to ​ and ​ the situations shown, so as to achieve more comprehensive functions. The embodiments of the present disclosure do not limit this. For example, in some examples, a selection module may also be deployed in the terminal device 200. The selection module is used to select the corresponding neural network module UN according to the bit rate of the input video, so as to process the input video using the selected neural network module UN. For example, in some other examples, a video decoding module and a video encoding module may also be deployed in the terminal device 200. The video decoding module is used to perform decoding operations, and the video encoding module is used to perform encoding operations.

[0197] For example, at least one neural network module UN among multiple neural network modules UN is used to perform the following processing: receiving an input image; processing the input image to obtain an output image. For example, the clarity of the output image is higher than that of the input image. For example, processing the input image to obtain an output image includes: extracting features from the input image to obtain multiple first images; performing a splicing process on the input image and the multiple first images to obtain a first image group, where the first image group includes the input image and the multiple first images; extracting features from the first image group to obtain multiple second images; fusing the multiple second images and the multiple first images to obtain multiple third images; performing a splicing process on the input image and the multiple third images to obtain a second image group, where the second image group includes the input image and the multiple third images; extracting features from the second image group to obtain the output image. For example, the neural network module UN is used to implement the image processing method provided by the embodiments of the present disclosure, and the relevant description can be referred to the foregoing content, which will not be elaborated here.

[0198] ​ It is a data flow diagram of a terminal device provided by some embodiments of the present disclosure. ​ It is an operation flowchart of a terminal device provided by some embodiments of the present disclosure. For example, in some examples, as ​ and ​ shown, after the software development kit 211 is started, multiple neural network modules UN can be imported first, and then the video frames and the bit rate information of the video are read. Since the model is bound to the video bit rate, when the bit rate of the video is determined, the neural network module UN corresponding to the bit rate is selected for processing, and finally the result is output.

[0199] Specifically, the following method can be used to process the input video.

[0200] First, initialize the model and configuration parameters. After the initialization is completed, the input video (such as a video file or a video stream) is decoded to obtain video frames. For example, when the input video is a video file, the terminal device (such as the terminal video processor 200) can implement the image quality enhancement of the offline video. When the input video is a video stream, the terminal video processor 200 can implement the real-time image quality enhancement of the live video.

[0201] Next, read the video frames and start processing the video frames. Select the neural network module UN corresponding to the bit rate from multiple neural network modules UN according to the bit rate information of the input video. Then, use the selected neural network module UN for model inference, that is, perform image processing (such as denoising and / or deblurring), so as to obtain the processed video frames. The clarity of the processed video frames is improved. Then, the processed video frames are encoded and then output to the display, and the display shows the processed video.

[0202] ​ It is a schematic diagram of a video frame extracted from a video with a 1M bitrate and standard definition 360p resolution. ​ It is the effect diagram after processing the ​ shown picture by using the terminal device provided in the embodiments of the present disclosure. By comparing ​ and ​ it can be seen that after being processed by the terminal device (such as the terminal video processor 200), the picture clarity is improved, and image quality enhancement is achieved. The terminal video processor 200 has good image quality enhancement ability and can achieve real-time processing.

[0203] By using the neural network module UN, the terminal device (such as the terminal video processor 200) provided in the embodiments of the present disclosure can remove the noise of low-quality images and videos, improve the picture clarity, and achieve image quality enhancement. Moreover, since the structure of the neural network module UN deployed on the terminal video processor 200 is simple, it can save device computing power, improve processing efficiency, be supported by the hardware capabilities of the terminal device, meet the processing speed requirements of video streams, and achieve real-time image quality enhancement.

[0204] At least one embodiment of the present disclosure further provides an image processing device. The convolutional neural network adopted by the image processing device has a simple structure, can save device computing power, can remove the noise of low-quality images and videos, improve the picture clarity, achieve real-time image quality enhancement, and is convenient to be applied to terminal devices. The image processing device provided by at least one embodiment also has good neural network training effects.

[0205] ​ It is a schematic block diagram of an image processing device provided by some embodiments of the present disclosure. As ​ shown, the image processing device 300 includes a processor 310 and a memory 320. The memory 320 is used to store non-transitory computer-readable instructions (such as one or more computer program modules). The processor 310 is used to run the non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are run by the processor 310, one or more steps in the above-mentioned image processing method can be executed. The memory 320 and the processor 310 can be interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0206] For example, the processor 310 can be a central processing unit (CPU), a digital signal processor (DSP), or other forms of processing units with data processing capabilities and / or program execution capabilities, such as a field-programmable gate array (FPGA), etc.; for example, the central processing unit (CPU) can be of the X86 or ARM architecture, etc. The processor 310 can be a general-purpose processor or a dedicated processor, and can control other components in the image processing apparatus 300 to perform desired functions.

[0207] For example, the memory 320 can include any combination of one or more computer program products, and the computer program products can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory can include, for example, random access memory (RAM) and / or cache memory, etc. Non-volatile memory can include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules can be stored on the computer-readable storage medium, and the processor 310 can run one or more computer program modules to implement various functions of the image processing apparatus 300. Various application programs and various data, as well as various data used and / or generated by the application programs, etc., can also be stored in the computer-readable storage medium.

[0208] It should be noted that in the embodiments of the present disclosure, the specific functions and technical effects of the image processing apparatus 300 can refer to the description of the image processing method in the foregoing text, and will not be elaborated herein.

[0209] ​ FIG. is a schematic block diagram of another image processing apparatus provided for some embodiments of the present disclosure. The image processing apparatus 400 is, for example, suitable for implementing the image processing method provided by the embodiments of the present disclosure. The image processing apparatus 400 can be a user terminal, etc. It should be noted that ​ The illustrated image processing apparatus 400 is only an example, and it will not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0210] Such as ​As shown, the image processing apparatus 400 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 410, which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 420 or a program loaded from a storage device 480 into a random access memory (RAM) 430. In the RAM 430, various programs and data required for the operation of the image processing apparatus 400 are also stored. The processing device 410, the ROM 420, and the RAM 430 are connected to each other via a bus 440. An input / output (I / O) interface 450 is also connected to the bus 440.

[0211] Generally, the following devices may be connected to the I / O interface 450: an input device 460 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 470 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 480 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 490. The communication device 490 may allow the image processing apparatus 400 to communicate with other electronic devices wirelessly or wiredly to exchange data. Although ​ the image processing apparatus 400 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices, and the image processing apparatus 400 may alternatively implement or have more or fewer devices.

[0212] For example, the image processing method provided by an embodiment of the present disclosure may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program codes for executing the above-mentioned image processing method. In such an embodiment, the computer program may be downloaded and installed from a network through the communication device 490, or installed from the storage device 480, or installed from the ROM 420. When the computer program is executed by the processing device 410, the image processing method provided by an embodiment of the present disclosure may be executed.

[0213] At least one embodiment of the present disclosure further provides a storage medium for storing non-temporary computer-readable instructions, which can implement the image processing method described in any embodiment of the present disclosure when executed by a computer. Using this storage medium, image processing can be performed through a convolutional neural network. The structure of the convolutional neural network is simple, which can save device computing power, can remove the noise of low-quality images and videos, improve the picture clarity, achieve real-time picture quality enhancement, and is convenient to be applied to terminal devices. The storage medium provided by at least one embodiment also has a good neural network training effect.

[0214] ​Schematic diagram of a storage medium provided by some embodiments of the present disclosure. As ​ shown, the storage medium 500 is used to store non-transitory computer-readable instructions 510. For example, when the non-transitory computer-readable instructions 510 are executed by a computer, one or more steps of the image processing method described above can be executed.

[0215] For example, the storage medium 500 can be applied to the above-mentioned image processing device 300. For example, the storage medium 500 can be ​ the memory 320 in the image processing device 300 shown. For example, the relevant description of the storage medium 500 can refer to ​ the corresponding description of the memory 320 in the image processing device 300 shown, which will not be elaborated here.

[0216] At least one embodiment of the present disclosure further provides a video processing method. The video processing method can process an input video based on a convolutional neural network to improve the picture clarity, achieve real-time picture quality enhancement, with good processing effect and high processing efficiency.

[0217] ​ Schematic flowchart of a video processing method provided by some embodiments of the present disclosure. For example, as ​ shown, the video processing method can include the following operations.

[0218] Step S61: Obtain the input video bit rate and the input video, where the input video includes a plurality of input image frames;

[0219] Step S62: Select a video processing sub-method corresponding to the input video bit rate according to the input video bit rate to process at least one input image frame among the plurality of input image frames, and obtain at least one output image frame, where the clarity of at least one output image frame is higher than that of at least one input image frame.

[0220] For example, different input video bit rates correspond to different video processing sub-methods. Here, the "video processing sub-method" can be the image processing method described in any embodiment of the present disclosure, or the method for processing a video with a certain bit rate in the terminal device provided in any embodiment of the present disclosure. The relevant description can refer to the foregoing content and will not be elaborated here.

[0221] The following points need to be noted:

[0222] (1) The drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure, and other structures can refer to the general design.

[0223] (2) Without conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0224] As described above, it is only the specific implementation manner of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims

1. An image processing method, applicable to a convolutional neural network, wherein, The method includes: Receiving an input image; Processing the input image by using the convolutional neural network to obtain an output image, wherein the clarity of the output image is higher than that of the input image; Processing the input image by using the convolutional neural network to obtain the output image includes: Performing feature extraction on the input image to obtain a plurality of first images; Performing splicing processing on the input image and the plurality of first images to obtain a first image group, wherein the first image group includes the input image and the plurality of first images; Performing feature extraction on the first image group to obtain a plurality of second images; Fusing the plurality of second images and the plurality of first images to obtain a plurality of third images; Performing splicing processing on the input image and the plurality of third images to obtain a second image group, wherein the second image group includes the input image and the plurality of third images; Performing feature extraction on the second image group to obtain the output image, wherein, the method further includes: Training a second neural network to be trained based on a pre-trained first neural network to obtain a trained second neural network, thereby obtaining the convolutional neural network; wherein, the number of parameters of the first neural network is more than that of the second neural network, the pre-trained first neural network is configured to transform an original image with a first clarity input to the pre-trained first neural network into a new image with a second clarity, the second clarity is greater than the first clarity, the trained second neural network is the convolutional neural network, the network structure of the second neural network to be trained is the same as that of the convolutional neural network, and the parameters of the second neural network to be trained are different from those of the convolutional neural network.

2. The method according to claim 1, wherein, The number of convolutional kernels for feature extraction of the input image in the convolutional neural network is N, 12 ≤ N ≤ 20 and N is an integer, The number of convolutional kernels for feature extraction of the first image group in the convolutional neural network is M, 12 ≤ M ≤ 20 and M is an integer, The number of convolutional kernels for feature extraction of the second image group in the convolutional neural network is 3.

3. The method according to claim 2, wherein N = M = 16; The sizes of the convolutional kernels for feature extraction of the input image, the convolutional kernels for feature extraction of the first image group, and the convolutional kernels for feature extraction of the second image group are all 3×3; The activation function for feature extraction in the convolutional neural network is: y = max(0, x), where x represents the input of the activation function and y represents the output of the activation function; The input image includes a red-channel input image, a green-channel input image, and a blue-channel input image, and the output image includes a red-channel output image, a green-channel output image, and a blue-channel output image.

4. The method according to claim 1, wherein, Training the second neural network to be trained based on the pre-trained first neural network to obtain the trained second neural network, thereby obtaining the convolutional neural network, including: Based on the pre-trained first neural network, the second neural network to be trained, and the discriminative network, alternately training the discriminative network and the second neural network to obtain the trained second neural network, thereby obtaining the convolutional neural network.

5. The method according to claim 4, wherein Training the discriminative network includes: Inputting first sample data into the first neural network and the second neural network respectively to obtain first data output from the first neural network and second data output from the second neural network; Setting the first data to have a true label, inputting the first data with the true label into the discriminative network to obtain a first discrimination result, setting the second data to have a false label, and inputting the second data with the false label into the discriminative network to obtain a second discrimination result; Calculating a first loss function based on the first discrimination result and the second discrimination result; Adjusting the parameters of the discriminative network according to the first loss function to obtain an updated discriminative network.

6. The method according to claim 5, wherein, Training the second neural network includes: Inputting second sample data into the first neural network and the second neural network respectively to obtain third data output from the first neural network and fourth data output from the second neural network; Setting the fourth data to have a true label, and inputting the fourth data with the true label into the updated discriminative network to obtain a third discrimination result output from the discriminative network; Calculating an error function based on the third data and the fourth data, calculating a discrimination function based on the third discrimination result, and calculating a second loss function based on the error function and the discrimination function; Adjusting the parameters of the second neural network according to the second loss function to obtain an updated second neural network.

7. The method according to claim 6, wherein, The second loss function is a weighted sum of the error function and the discrimination function.

8. The method according to claim 7, wherein, The weight of the error function is 90 - 110, and the weight of the discrimination function is 0.5 - 2.

9. The method according to claim 6, wherein The first sample data and the second sample data are image data obtained based on multiple videos with the same bit rate.

10. The method according to claim 1, wherein Training the second neural network to be trained based on the pre-trained first neural network to obtain the trained second neural network, thereby obtaining the convolutional neural network, including: Inputting third sample data into the first neural network and the second neural network respectively to obtain fifth data output from the first neural network and sixth data output from the second neural network; Calculating a third loss function based on the fifth data and the sixth data; Adjusting the parameters of the second neural network according to the third loss function to obtain an updated second neural network.

11. The method according to claim 1, wherein, The first neural network includes multiple levels of downsampling units and corresponding multiple levels of upsampling units. The output of each level of downsampling unit serves as the input of the next level of downsampling unit. The input of each level of upsampling unit includes the output of the downsampling unit corresponding to this level of upsampling unit and the output of the previous level of upsampling unit of this level of upsampling unit.

12. A terminal device, including a processor, Among them, The processor is configured to: Obtain an input video bit rate and an input video, where the input video includes multiple input image frames; Select a video processing method corresponding to the input video bit rate according to the input video bit rate to process at least one input image frame among the multiple input image frames, so as to obtain at least one output image frame, where the clarity of the at least one output image frame is higher than the clarity of the at least one input image frame; Wherein, different input video bit rates correspond to different video processing methods, Wherein, the video processing method includes: Process the at least one input image frame based on a trained neural network to obtain the at least one output image frame; Wherein, processing the at least one input image frame based on the trained neural network to obtain the at least one output image frame includes: Extract features from the at least one input image frame to obtain multiple first output images; Perform splicing processing on the at least one input image frame and the multiple first output images to obtain a first output image group, where the first output image group includes the at least one input image frame and the multiple first output images; Extract features from the first output image group to obtain multiple second output images; Fuse the multiple second output images and the multiple first output images to obtain multiple third output images; Perform splicing processing on the at least one input image frame and the multiple third output images to obtain a second output image group, where the second output image group includes the at least one input image frame and the multiple third output images; Extract features from the second output image group to obtain the at least one output image frame, Wherein, the video processing method further includes: Train a second neural network to be trained based on a pre-trained first neural network to obtain a trained second neural network, thereby obtaining the trained neural network; Wherein, the number of parameters of the first neural network is more than the number of parameters of the second neural network. The pre-trained first neural network is configured to transform an original image with a first clarity input to the pre-trained first neural network into a new image with a second clarity, and the second clarity is greater than the first clarity. The trained second neural network is the trained neural network. The network structure of the second neural network to be trained is the same as the network structure of the trained neural network, and the parameters of the second neural network to be trained are different from the parameters of the trained neural network.

13. The terminal device according to claim 12, wherein, The trained neural networks corresponding to different video processing methods are different.

14. The terminal device according to claim 13, wherein, The trained neural networks corresponding to different video processing methods are respectively trained using different sample data sets, and the different sample data sets are respectively obtained based on different video sets. Each video set includes multiple videos. The videos in the same video set have the same bit rate, and the videos in different video sets have different bit rates.

15. A video processing method, comprising: Obtaining an input video bit rate and an input video, where the input video includes a plurality of input image frames; Selecting a video processing sub-method corresponding to the input video bit rate according to the input video bit rate to process at least one of the plurality of input image frames, so as to obtain at least one output image frame, where the clarity of the at least one output image frame is higher than the clarity of the at least one input image frame; wherein different input video bit rates correspond to different video processing sub-methods, wherein the video processing sub-method includes: Processing the at least one input image frame based on a trained neural network to obtain the at least one output image frame; wherein processing the at least one input image frame based on the trained neural network to obtain the at least one output image frame includes: Performing feature extraction on the at least one input image frame to obtain a plurality of first output images; Performing splicing processing on the at least one input image frame and the plurality of first output images to obtain a first output image group, where the first output image group includes the at least one input image frame and the plurality of first output images; Performing feature extraction on the first output image group to obtain a plurality of second output images; Fusing the plurality of second output images and the plurality of first output images to obtain a plurality of third output images; Performing splicing processing on the at least one input image frame and the plurality of third output images to obtain a second output image group, where the second output image group includes the at least one input image frame and the plurality of third output images; Performing feature extraction on the second output image group to obtain the at least one output image frame, wherein the video processing sub-method further includes: Training a second neural network to be trained based on a pre-trained first neural network to obtain a trained second neural network, thereby obtaining the trained neural network; wherein the first neural network has more parameters than the second neural network. The pre-trained first neural network is configured to transform a raw image with a first clarity input to the pre-trained first neural network into a new image with a second clarity, and the second clarity is greater than the first clarity. The trained second neural network is the trained neural network. The network structure of the second neural network to be trained is the same as the network structure of the trained neural network, and the parameters of the second neural network to be trained are different from the parameters of the trained neural network.

16. An image processing apparatus, comprising: A processor; A memory, including one or more computer program modules; Wherein, the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include a method for implementing the image processing method according to any one of claims 1-11.

17. A storage medium for storing non-transitory computer-readable instructions that, when executed by a computer, can implement the image processing method according to any one of claims 1-11.

Images