Video playing, image processing and model training method, device and electronic equipment

Abstract

Embodiments of the present application provide a video playing method, an image processing method and a model training method, a device and electronic equipment. The image processing method comprises: acquiring a to-be-processed image; inputting the to-be-processed image into an image processing model to perform image processing according to a super-resolution algorithm. The image processing model comprises a first convolutional network and a second convolutional network connected to each other, convolutional parameters of the first convolutional network are trained to represent image parameters of the super-resolution algorithm, and convolutional parameters of the second convolutional network are preset to represent operator parameters of the super-resolution algorithm. The scheme of the embodiments of the present application improves the efficiency of image super-resolution processing, and at the same time, uses a simple convolutional network structure to ensure low cost of the image processing model.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer technology, and in particular, to a video playing, image processing and model training method and device and electronic equipment. BACKGROUND

[0002] Image super-resolution, also referred to as image super-resolution, is a technology of generating a high-resolution image from a low-resolution image or image sequence, which is widely applied in video playing scenarios such as live broadcast, on-demand, real-time communication, player and the like requiring high-quality video.

[0003] In the increasing demand of the video field, especially in a weak network environment, the code rate of video compression coding is very low, resulting in poor video quality and serious distortion, and thus a low-cost image super-resolution solution is needed. SUMMARY

[0004] Therefore, embodiments of the present application provide a video playing, image processing and model training method and device and electronic equipment to at least partially solve the above problems.

[0005] According to a first aspect of embodiments of the present application, an image processing method is provided, comprising: acquiring a to-be-processed image; inputting the to-be-processed image into an image processing model to perform image processing according to a super-resolution algorithm, wherein the image processing model comprises a first convolutional network and a second convolutional network connected to each other, convolutional parameters of the first convolutional network are used to represent image parameters of the super-resolution algorithm via training, and convolutional parameters of the second convolutional network are used to represent operator parameters of the super-resolution algorithm via pre-setting.

[0006] According to a second aspect of embodiments of the present application, a model training method is provided, comprising: acquiring training data; training a neural network based on the training data to obtain an image processing model, the image processing model being used to perform image processing based on a super-resolution algorithm, the neural network comprising a first convolutional network and a second convolutional network connected to each other, convolutional parameters of the first convolutional network being used to represent image parameters in the super-resolution algorithm via adjustment, and convolutional parameters of the second convolutional network being used to represent operator parameters in the super-resolution algorithm via pre-setting.

[0007] According to a third aspect of the embodiments of the present application, an image processing apparatus is provided, comprising: an obtaining module, configured to obtain an image to be processed; and a processing module, configured to input the image to be processed into an image processing model to perform image processing according to a super-resolution algorithm, wherein the image processing model comprises a first convolutional network and a second convolutional network connected with each other, convolutional parameters of the first convolutional network are trained to represent image parameters of the super-resolution algorithm, and convolutional parameters of the second convolutional network are preset to represent operator parameters of the super-resolution algorithm.

[0008] According to a fourth aspect of the embodiments of the present application, a model training apparatus is provided, comprising: an obtaining module, configured to obtain training data; and a training module, configured to train a neural network based on the training data to obtain an image processing model, wherein the image processing model is used to perform image processing based on a super-resolution algorithm, the neural network comprises a first convolutional network and a second convolutional network connected with each other, convolutional parameters of the first convolutional network are adjusted to represent image parameters in the super-resolution algorithm, and convolutional parameters of the second convolutional network are preset to represent operator parameters in the super-resolution algorithm.

[0009] According to a fifth aspect of the embodiments of the present application, a video playing method is provided, comprising: obtaining video data; performing super-resolution processing on the video data according to an image processing method, wherein the image processing method is the method according to the first aspect; and playing the video data after super-resolution processing.

[0010] According to a sixth aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the method according to the first aspect or the second aspect.

[0011] According to a seventh aspect of the embodiments of the present application, a computer storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the method according to the first aspect or the second aspect.

[0012] In the scheme of the embodiments of the present application, the convolutional parameters of the first convolutional network represent the image parameters through training, and the convolutional parameters of the second convolutional network represent the operator parameters through presetting, the super-resolution algorithm is efficiently and reliably implemented through the convolutional networks, the efficiency of image super-resolution processing is improved, and the low cost of the image processing model is ensured by using a simple convolutional network structure. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art based on these drawings.

[0014] Figure 1A A schematic block diagram of a video playing system for an example;

[0015] Figure 1B A step flow chart of an image processing method according to an embodiment of the present application;

[0016] Figure 2A A schematic diagram of a convolutional network structure according to an embodiment of the present application;

[0017] Figure 2B A schematic diagram of a convolutional network structure according to an embodiment of the present application;

[0018] Figure 3 A schematic diagram of a convolutional network structure according to another embodiment of the present application;

[0019] Figure 4A A schematic diagram of a convolutional network structure according to another embodiment of the present application;

[0020] Figure 4B A schematic diagram of a convolutional network structure according to another embodiment of the present application;

[0021] Figure 4C A schematic diagram of a convolutional network structure according to another embodiment of the present application;

[0022] Figure 5 A schematic diagram of a convolutional network structure according to another embodiment of the present application;

[0023] Figure 6 A structural block diagram of an image processing device according to another embodiment of the present application;

[0024] Figure 7A A step flow chart of a model training method according to another embodiment of the present application;

[0025] Figure 7B A structural block diagram of a model training device according to another embodiment of the present application;

[0026] Figure 8 A structural schematic diagram of an electronic device according to another embodiment of the present application. DETAILED DESCRIPTION

[0027] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by a person of ordinary skill in the art should belong to the scope of protection of the present application.

[0028] The specific implementation of the embodiments of the present application will be further described in conjunction with the drawings of the embodiments of the present application.

[0029] Figure 1A A schematic block diagram of an exemplary video playing system. Figure 1A The video playing system includes a terminal device 110, a video playing server 120, and a model training server 130. The model training server 130 and the video playing server 120 can communicate with the terminal device 110 through a network such as the Internet.

[0030] The video playing server 120 can obtain video data such as real-time video data from a video data resource 121 in response to a user's on-demand or live operation through the terminal device 110. Then, a communication module 122 of the video playing server 120 can perform communication with a communication module 111 of the terminal device 110, and transmit the video data to the terminal device 110 for playing through a video playing module 113.

[0031] Specifically, the video data can be stored in a predetermined encoding format in the video data resource 121, and the video playing server 120 transmits the video data in the video data resource 121 to the terminal device 110. Alternatively, the video data can be stored in an uncoded manner in the video data resource 121, and the video playing server 120 encodes the video data before transmitting to the terminal device 110 through the communication module 122. Correspondingly, the terminal device 110 decodes the encoded video data for playing.

[0032] It should be understood that, on the one hand, the video data is encoded at the video playing server 120 side and decoded at the terminal device 110 side, which is beneficial to the video transmission efficiency, for example, saving communication resources such as transmission bandwidth. On the other hand, the encoding processing of the video data results in a certain resolution loss of the video data.

[0033] Thus, the terminal device 110 can perform super-resolution processing on the video data received from the video playback server 120 to obtain a video image with a higher resolution. For example, the image processing module 112 performs real-time super-resolution processing on the video data, specifically, the image processing module 112 performs super-resolution processing on the decoded video data, and then the video playback module 113 plays the video data after the super-resolution processing.

[0034] Specifically, the image processing module 112 can invoke the image processing model 114 to perform super-resolution processing on the video data. The image processing model 114 can be trained by the model training module 131 of the model training server 130 and deployed to the terminal device 110 via the model deployment module 132.

[0035] It should be understood that the image processing model 114 can be periodically updated and deployed by the model deployment module 132 or in response to a predetermined update instruction.

[0036] Figure 1B A flowchart of the steps of the image processing method according to an embodiment of the present application. The scheme of the present embodiment can be applied to any appropriate electronic device with data processing capability, including but not limited to servers, mobile terminals (such as mobile phones, PADs, etc.), and PC computers, etc. In one example, the electronic device can execute the method of the present embodiment based on a machine learning software framework in cooperation with a hardware configuration such as a GPU. Figure 1B The image processing method comprises:

[0037] S110: Obtain a to-be-processed image.

[0038] It should be understood that the to-be-processed image can be a video frame in the video data in the above embodiments, or can be non-video image data. Specifically, when the to-be-processed image is a video frame, a decoded video frame can be obtained, for example, a group of video frames decoded based on a key frame can be obtained.

[0039] It should also be understood that the to-be-processed image can be a multi-channel image including three channels of RBG, or can be a single-channel image, for example, a grayscale image. The to-be-processed image can also be a feature map of a multi-channel image or a single-channel image, i.e., a feature map output after processing a multi-channel image or a single-channel image via a convolution layer or a convolution group.

[0040] S120: Input the to-be-processed image into an image processing model to perform image processing according to a super-resolution algorithm.

[0041] The image processing model comprises a first convolutional neural network (CNN) and a second convolutional neural network connected to each other, convolutional parameters of the first convolutional neural network are trained to represent image parameters of the super-resolution algorithm, and convolutional parameters of the second convolutional neural network are preset to represent operator parameters of the super-resolution algorithm.

[0042] It should be understood that the image processing model can be implemented by a convolutional network, the first convolutional network is a part of the image processing model whose parameters need to be adjusted, and the convolutional parameters of the second convolutional network can be preset to implement various operator parameters, such as summation operation, subtraction operation, multiplication operation, division operation, merging operation, normalization operation, etc. The convolutional parameters of the second convolutional network can be various operators defined in a software framework of the convolutional network, so as to improve the configuration efficiency of the neural network, in other words, the convolutional parameters of the second convolutional network can be a conventional convolutional network.

[0043] It should also be understood that the super-resolution algorithm includes but is not limited to image wavelet processing, image edge processing, image sharpening processing, image bilateral filtering processing, etc. The image parameters include but are not limited to image sharpening parameters, channel pixel parameters, and other parameters directly related to images and reflecting image indicators.

[0044] In the scheme of the embodiment of the present application, the convolutional parameters of the first convolutional network are trained to represent image parameters, and the convolutional parameters of the second convolutional network are preset to represent operator parameters, so that the super-resolution algorithm is efficiently and reliably implemented by the convolutional network, the efficiency of image super-resolution processing is improved, and the low cost of the image processing model is ensured by using a simple convolutional network structure. Especially for images in video data, the resolution is lost after encoding processing, and the embodiment of the present application can provide efficient super-resolution processing at a lower cost and computational overhead.

[0045] It should be understood that the image processing method provided by the embodiment of the present application is suitable for various types of neural network structures, and the super-resolution algorithm is implemented by using a neural network, which improves the efficiency of image processing compared with traditional super-resolution algorithms.

[0046] In some examples, the super-resolution algorithm includes a bilateral filter algorithm, and the image parameters include respective pixel differences between respective pixel values and a center pixel value, and a standard deviation of the respective pixel differences. The operator parameters are used to determine respective pixel values processed based on the bilateral filter algorithm according to the respective pixel differences and the standard deviation. The first convolutional network includes a first convolutional group preconfigured to characterize the respective pixel differences and linearly superimposed with parameters obtained via training, and a second convolutional group preconfigured to characterize the standard deviation and linearly superimposed with parameters obtained via training. This embodiment realizes the image parameters of the bilateral filter algorithm through the convolutional groups, reduces the cost of image processing, and further improves the effect of the bilateral filter algorithm by linearly superimposing the parameters obtained via training. It should be understood that the second convolutional network can include operators such as an addition operator (add), a multiplication operator (mul), a division operator (div), a merging operator (concat), a normalization operator (softmax), and the like.

[0047] As shown in Figure 2A The neural network of the image processing model includes a plurality of convolutional groups 201-209. The bilateral filter algorithm used for image processing satisfies the following formula:

[0048]

[0049]

[0050] wherein aij is a weight of a pixel difference; Ixy is a pixel value filtered based on the bilateral filter algorithm, i is a position (e.g., coordinate) of a first dimension (e.g., X-axis) in a channel map or a feature map of an image, and j is a position (e.g., coordinate) of a second dimension (e.g., Y-axis) in the channel map or the feature map; and σ is a standard deviation of respective pixel values. The summation operation and the multiplication operation in the weighted processing of formula (2) are examples of operator parameters.

[0051] It should be understood that the single-channel map fI is a respective pixel difference between respective pixel values and a center pixel value in a single-channel map, wherein I can be 0, 1, 2, and 3. In one example, when I is 0, fI represents a respective pixel difference between respective pixel values and a center pixel value in a grayscale map. In another example, Rxy, Gxy, and Bxy represent channel pixel values of channel R (red), G (green), and B (blue), respectively, and f1, f2, and f3 are respective pixel differences between respective pixel values and a center pixel value in the corresponding channel map. The neural network structure is described below only as an example of processing a single-channel map. The single-channel map can be any of the above RGB channel maps or a grayscale map.

[0052] In the present example, the first convolutional network and the second convolutional network are described and illustrated in terms of a plurality of convolutional groups. The convolutional groups 201-209 can be implemented as one or more convolutions and an activation function connected thereto. The convolutional parameters of each convolutional group include weight parameters and [output channel number m; input channel number n; convolution kernel size k; convolution kernel size k], bias B [output channel number], and padding parameters, where the padding parameters can be preset to 0 or other values. In addition, the output channel number m and the input channel number n corresponding to each channel map can be any value as long as the output channel number m of a channel map is the same as the input channel number n of the next channel map. In the present example, m can take a value of 1 or 8, and n can take a value of 1 or 8. The convolution kernel size k corresponding to each channel map can also be predefined as any value, and in the present example, the convolution kernel size is 1, 3, or 5. However, it should not be understood that the input channel number n, the output channel number m, and the convolution kernel size k are limited.

[0053] The convolutional group 201 indicates obtaining a single channel map and obtaining the standard deviation σ of each pixel value in the single channel map. Specifically, the single channel map can be a single channel map of the image to be processed (e.g., obtained from the input layer (input)), or a feature map obtained after the single channel map is processed by the convolutional group (e.g., obtained from the sum (add)). In the present example, the convolutional parameters of the convolutional group 201 are W [1; 8; 3; 3], B [1], in other words, an 8-channel feature map is processed by a convolution kernel size of 3 x 3 to obtain a single channel feature map.

[0054] The convolutional group 202 indicates obtaining each pixel value of the single channel map. In the present example, the convolutional parameters of the convolutional group 202 are W [1; 8; 3; 3]. In other words, an 8-channel feature map is processed by a convolution kernel size of 3 x 3 to output a single channel feature map.

[0055] The convolutional group 203 indicates calculating aij. Specifically, aij is calculated by the above formula (1). Specifically, i and j are not simultaneously 0. In the present example, the convolutional parameters of the convolutional group 203 are W [8; 1; 3; 3]. In other words, a single channel feature map is processed by a convolution kernel size of 3 x 3 to output an 8-channel feature map.

[0056] The convolutional group 204 indicates calculating fi. Specifically, fi is calculated by the above formula (1). I . Specifically, i and j are not simultaneously 0. In the present example, the convolutional parameters of the convolutional group 204 are W [8; 1; 3; 3]. In other words, a single channel feature map is processed by a convolution kernel size of 3 x 3 to obtain an 8-channel feature map.

[0057] The convolution group 205 indicates that ∑fi is calculated by summation. In this example, the convolution parameter of the convolution group 205 is W[l;8;1;1]. In other words, the 8-channel feature map is processed by a convolution kernel size of 1x1 to obtain a single-channel feature map, i.e., the convolution kernel size of 1x1 performs inter-channel fusion of 8 channels.

[0058] The convolution group 206 indicates that ∑aij is calculated by summation. In this example, the convolution parameter of the convolution group 206 is W[l;8;1;1], B[l]. In other words, the 8-channel feature map is processed by a convolution kernel size of 1x1 to obtain a single-channel feature map.

[0059] The convolution group 207 indicates that the calculation results of the convolution group 205 and the convolution group 206 are applied to the above formula (2) to obtain Ixy.

[0060] The convolution group 208 indicates that the output of the convolution group 207 is divided into 8 channels. In this example, the convolution parameter of the convolution group 208 is W[8;1;3;3], B[8]. In other words, the single-channel feature map is processed by a convolution kernel size of 3x3 to obtain an 8-channel feature map.

[0061] The convolution group 209 indicates that the output results of the 8 channels are merged with other feature maps, which can be achieved by an add operator or a concat operator.

[0062] In addition, in this example, the convolution group 202 and the convolution group 204 form a first convolution network. Other convolution groups form a second convolution network. In other words, the plurality of convolution groups forming the first convolution network are not necessarily directly connected to each other.

[0063] Alternatively, Figure 2B A convolution network structure according to an embodiment of the present application is shown.

[0064] The convolution group 2001 indicates that a single-channel image is obtained, and a standard deviation σ of each pixel value in the single-channel image is obtained. Specifically, the single-channel image can be a single-channel image of a to-be-processed image (e.g., obtained from an input layer), or a feature map obtained by processing a single-channel image via a convolution group (e.g., obtained from an add).

[0065] The convolution group 2002 indicates that the pixel values of the single-channel image are obtained. In this example, the convolution parameter of the convolution group 2002 is W[l;8;3;3]. In other words, the 8-channel feature map is processed by a convolution kernel size of 3x3 to output a single-channel feature map.

[0066] The convolution group 2003 indicates that aij / ∑aij is calculated. Specifically, aij is calculated by the above formula (1), and the softmax operator obtains aij, and then normalization operation is performed to obtain aij / ∑aij. In this example, the convolution parameter of the convolution group 2003 is W[9;1;3;3]. The output channel number 9 indicates that 9 aij will be calculated subsequently.

[0067] The convolution group 2004 indicates that fI is calculated. Specifically, fI is calculated by the above formula (1). Specifically, i and j are not simultaneously 0. In this example, the convolution parameter of the convolution group 2004 is W[9;1;3;3]. In other words, the single-channel feature map is processed by convolution with a convolution kernel size of 3x3 to obtain a 9-channel feature map.

[0068] The convolution group 2005 indicates that fI x aij / ∑aij is obtained by multiplication operation.

[0069] The convolution group 2006 indicates that ∑fI x aij / ∑aij is obtained by summation calculation.

[0070] In addition, the convolution parameter of the first convolution network represents the image parameter through the base image parameter and the adjustment parameter. The image parameter is a function of the base image parameter and the adjustment parameter, and the adjustment parameter is input via a parameter adjustment interface of the image processing model. For example, the parameter adjustment interface can be configured in a human-computer interaction interface through which a user can set the adjustment parameter through a device running the image processing model. It should be understood that the parameter adjustment interface can be implemented as a resolution setting in the human-computer interaction interface (for example, a corresponding relationship between the resolution and the adjustment parameter is established) to adjust the video quality or the image quality, etc. For example, the parameter adjustment interface is associated with the resolution setting option in the video player. In this example, the above-mentioned fI and / or σ can be taken as examples of the adjustment parameter.

[0071] In other examples, the super-resolution algorithm includes an edge extraction algorithm, the operator parameter includes a plurality of edge extraction operators of the edge extraction algorithm, and the image parameter is used to linearly combine the plurality of edge extraction operators.

[0072] In other examples, the first convolution network includes a reparameterization convolution, and the reparameterization convolution is obtained by reparameterizing the convolution parameters of a plurality of convolution groups, where the convolution parameters of the plurality of convolution groups respectively represent the operator parameters of a plurality of edge extraction operators. For example, Figure 3As shown, the plurality of convolution groups includes a first convolution group, a second convolution group, and a third convolution group. In one specific example, the first convolution group is composed of a 5x5 convolution, the second convolution group is composed of a 1x1 convolution, a 3x3 convolution, and a 3x3 convolution, and the third convolution group includes a 1x1 convolution, a 3x3 convolution, and a 3x3 convolution. In another example, the second convolution group and the third convolution group can also not include a 1x1 convolution. It should be understood that the 1x1 convolution described above can play a role of inter-channel feature fusion.

[0073] In the case where either of the second convolution group and the third convolution group includes a 1x1 convolution, the 1x1 convolution and the 3x3 convolution can be re-parameterized as a 3x3 convolution, which can be re-parameterized as a 5x5 convolution with another 3x3 convolution in the convolution group; in the case where neither of the second convolution group and the third convolution group includes a 1x1 convolution, the two 3x3 convolutions can be re-parameterized as a 5x5 convolution. Thus, the second convolution group and the third convolution group can both be re-parameterized as a 5x5 convolution, both of which can be merged as a 5x5 convolution with the 5x5 convolution as the first convolution group. It should be understood that the operation described above is only one example of re-parameterization, and the re-parameterization of the present embodiment can be applied to other numbers of convolution groups and can also be applied to various convolution kernel sizes.

[0074] In addition, either of the 3x3 convolution and the 3x3 convolution in the second convolution group can be a convolution for gradient / edge enhancement, and when the 3x3 convolution on the adjacent input side is used for gradient / edge enhancement, the other 3x3 convolution away from the input side can be set to be used for edge filtering. It should be understood that the convolution for gradient / edge enhancement includes but is not limited to laplace operator and sobel operator. In the present example, the convolution for gradient / edge enhancement is included in the second convolution group, but it should be understood that multiple convolution groups can include such a convolution, and the present example does not limit the number of convolution groups for image edge processing.

[0075] As one specific example, the parameters of the input feature map are [number of training batches; number of input channels; first dimension pixel number; second dimension pixel number], and the parameters of the output feature map are [number of training batches; number of output channels; first dimension pixel number; second dimension pixel number]. The convolution parameters of the 5x5 convolution include weight parameters and [number of output channels; number of input channels; convolution kernel size; convolution kernel size], bias B [number of output channels], and padding parameters, wherein the padding parameters can be preset to 0 or other values.

[0076] The parameters of the two 3x3 convolutions of the reparameterization are: W1, [intermediate channel number; input channel number; kernel size; kernel size], bias bias1: [intermediate channel number], stride = 1, padding = 0; W2: [intermediate channel number; output channel number; kernel size; kernel size], bias bias2: [out_chs], stride = 1, padding = 0. The specific formulas of the two 3x3 convolutions synthesizing a 5x5 convolution will be shown as follows:

[0077] W5x5[:,:,0,0] = W2[:,:,0,0] * W1[:,:,0,0]

[0078] W5x5[:,:,0,1] = W2[:,:,0,0] * W1[:,:,0,1] + W2[:,:,0,1] * W1[:,:,0,0]

[0079] W5x5[:,:,0,2] = W2[:,:,0,0] * W1[:,:,0,2] + W2[:,:,0,1] * W1[:,:,0,1] + W2[:,:,0,2] * W1[:,:,0,0]

[0080] W5x5[:,:,0,3] = W2[:,:,0,1] * W1[:,:,0,2] + W2[:,:,0,2] * W1[:,:,0,1]

[0081] W5x5[:,:,0,4] = W2[:,:,0,2] * W1[:,:,0,2];

[0082] W5x5[:,:,1,0] = W2[:,:,0,0] * W1[:,:,1,0] + W2[:,:,1,0] * W1[:,:,0,0]

[0083] W5x5[:,:,1,1] = W2[:,:,0,0] * W1[:,:,1,1] + W2[:,:,0,1] * W1[:,:,1,0] + W2[:,:,1,0] * W1[:,:,0,1] + W2[:,:,1,1] * W1[:,:,0,0]

[0084] W5x5[:,:,1,2] = W2[:,:,0,0] * W1[:,:,1,2] + W2[:,:,0,1] * W1[:,:,1,1] + W2[:,:,0,2] * W1[:,:,1,0] + W2[:,:,1,0] * W1[:,:,0,2] + W2[:,:,1,1] * W1[:,:,0,1] + W2[:,:,1,2] * W1[:,:,0,0]

[0085] W5x5[:,:,1,3] = W2[:,:,0,1] * W1[:,:,1,2] + W2[:,:,0,2] * W1[:,:,1,1] + W2[:,:,1,1] * W1[:,:,0,2] + W2[:,:,1,2] * W1[:,:,0,1]

[0086] W5x5[:,:,1,4] = W2[:,:,0,2] * W1[:,:,1,2] + W2[:,:,1,2] * W1[:,:,0,2];

[0087] W5x5[:,:,2,0] = W2[:,:,0,0] * W1[:,:,2,0] + W2[:,:,1,0] * W1[:,:,1,0] + W2[:,:,2,0] * W1[:,:,0,0]

[0088] W5x5[:,:,2,1] = W2[:,:,0,0] * W1[:,:,2,1] + W2[:,:,0,1] * W1[:,:,2,0] + W2[:,:,1,0] * W1[:,:,1,1] + W2[:,:,1,1] * W1[:,:,1,0] + W2[:,:,2,0] * W1[:,:,0,1] + W2[:,:,2,1] * W1[:,:,0,0]

[0089] W5x5[:,:,2,2] = W2[:,:,0,0] * W1[:,:,2,2] + W2[:,:,0,1] * W1[:,:,2,1] + W2[:,:,0,2] * W1[:,:,2,0] + W2[:,:,1,0] * W1[:,:,1,2] + W2[:,:,1,1] * W1[:,:,1,1] + W2[:,:,1,2] * W1[:,:,1,0] + W2[:,:,2,0] * W1[:,:,0,2] + W2[:,:,2,1] * W1[:,:,0,1] + W2[:,:,2,2] * W1[:,:,0,0]

[0090] W5x5[:,:,2,3] = W2[:,:,0,1] * W1[:,:,2,2] + W2[:,:,0,2] * W1[:,:,2,1] + W2[:,:,1,1] * W1[:,:,1,2] + W2[:,:,1,2] * W1[:,:,1,1] + W2[:,:,2,1] * W1[:,:,0,2] + W2[:,:,2,2] * W1[:,:,0,1]

[0091] W5x5[:,:,2,4] = W2[:,:,0,2] * W1[:,:,2,2] + W2[:,:,1,2] * W1[:,:,1,2] + W2[:,:,2,2] * W1[:,:,0,2];

[0092] W5x5[:,:,3,0] = W2[:,:,1,0] * W1[:,:,2,0] + W2[:,:,2,0] * W1[:,:,1,0]

[0093] W5x5[:,:,3,1] = W2[:,:,1,0] * W1[:,:,2,1] + W2[:,:,1,1] * W1[:,:,2,0] + W2[:,:,2,0] * W1[:,:,1,1] + W2[:,:,2,1] * W1[:,:,1,0]

[0094] W5x5[:,:,3,2] = W2[:,:,1,0] * W1[:,:,2,2] + W2[:,:,1,1] * W1[:,:,2,1] + W2[:,:,1,2] * W1[:,:,2,0] + W2[:,:,2,0] * W1[:,:,1,2] + W2[:,:,2,1] * W1[:,:,1,1] + W2[:,:,2,2] * W1[:,:,1,0]

[0095] W5x5[:,:,3,3] = W2[:,:,1,1] * W1[:,:,2,2] + W2[:,:,1,2] * W1[:,:,2,1] + W2[:,:,2,1] * W1[:,:,1,2] + W2[:,:,2,2] * W1[:,:,1,1]

[0096] W5x5[:,:,3,4] = W2[:,:,1,2] * W1[:,:,2,2] + W2[:,:,2,2] * W1[:,:,1,2];

[0097] W5x5[:,:,4,0] = W2[:,:,2,0] * W1[:,:,2,0]

[0098] W5x5[:,:,4,1] = W2[:,:,2,0] * W1[:,:,2,1] + W2[:,:,2,1] * W1[:,:,2,0]

[0099] W5x5[:,:,4,2] = W2[:,:,2,0] * W1[:,:,2,2] + W2[:,:,2,1] * W1[:,:,2,1] + W2[:,:,2,2] * W1[:,:,2,0]

[0100] W5x5[:,:,4,3] = W2[:,:,2,1] * W1[:,:,2,2] + W2[:,:,2,2] * W1[:,:,2,1]

[0101] W5x5[:,:,4,4] = W2[:,:,2,2] * W1[:,:,2,2].

[0102] In addition, in some other examples, the super-resolution algorithm includes a sharpening algorithm, and the image parameter includes a sharpening intensity factor. The operator parameter is used to sharpen according to the sharpening intensity factor, which improves the efficiency of image sharpening while reducing the cost of image sharpening. Figure 4A As shown in the figure, the image processing model includes a plurality of convolution groups 401-404. The convolution parameters of each convolution group include weight parameters and [output channel number m; input channel number n; convolution kernel size k; convolution kernel size k], bias B [output channel number], and padding parameters, where the padding parameters can be preset to 0 or other values. In addition, the output channel number m and the input channel number n corresponding to each channel graph can be any value as long as the output channel number m of a channel graph is the same as the input channel number n of the next channel graph. In this example, m can take a value of 1 or 8, and n can take a value of 1 or 8. The convolution kernel size k corresponding to each channel graph can also be defined as any value in advance, and in this example, the convolution kernel size is 3 or 5. However, it should not be understood that the input channel number n, the output channel number m, and the convolution kernel size k are limited.

[0103] In addition, the feature map parameter input to the image processing model is [training batch number; input channel number; first dimension pixel; second dimension pixel]. In this example, the first dimension and the second dimension can be the X axis and the Y axis, the training batch number (batchnorm) can be 1, the input channel number can be 8 channels, and the resolution of the feature map can be 180 x 320.

[0104] where the sharpening algorithm can follow the formula:

[0105] y = px - (p - 1)f(x) (3)

[0106] f(x) = boxfilter(x, kernelb) (4)

[0107]

[0108] y = (p - 1)(x - f(x)) + x = dx + x (6)

[0109] The convolution group 401 indicates that (x-f(x)) in the formula (5) is extracted from the input feature map. In this example, the convolution parameters of the convolution group 401 are W[8;1;3;3], B[8], in other words, the 8-channel feature map is processed by the convolution kernel size 3x3 to obtain a single-channel feature map.

[0110] The convolution group 402 indicates that (p-1) is extracted from the input feature map. Specifically, Figure 4A Exemplarily, the convolution group 402 including 4 convolutions is shown, the convolution parameters of the first convolution are W[8;8;5;5], B[8]; the convolution parameters of the second convolution are W[8;8;3;3], B[8]; the convolution parameters of the third convolution are W[8;8;3;3], B[8]; and the convolution parameters of the fourth convolution are W[8;8;5;5], B[8]. In other words, the 8-channel feature map is processed by the 4 convolutions, and the accurate (p-1) is extracted by different convolution kernels.

[0111] The convolution group 403 indicates that the output of the convolution group 401 is multiplied with the output of the convolution group 403 (for example, using the multiplication operator mul) to obtain (p-1)(x-f(x)).

[0112] The convolution group 404 indicates that the sum (for example, using the summation operator add) is calculated by the formula (5) to obtain y.

[0113] In addition, the convolution parameters of the first convolution network are characterized by the basic image parameters and the adjustment parameters. The image parameters are functions of the basic image parameters and the adjustment parameters, and the adjustment parameters are input via a parameter adjustment interface of the image processing model. For example, the parameter adjustment interface can be configured in a human-computer interaction interface of a device running the image processing model, and a user can set the adjustment parameters through the parameter adjustment interface. It should be understood that the parameter adjustment interface can be implemented as a resolution setting in the human-computer interaction interface (for example, establishing a corresponding relationship between the resolution and the adjustment parameters) to adjust the video quality or the image quality, etc. For example, the parameter adjustment interface is associated with the resolution setting option in the video player. In this example, the above-mentioned p or (1-p) can be taken as an example of the adjustment parameters.

[0114] In other examples, the image processing model is generated by connecting a first convolutional network to an open interface of a general image processing model, which is then built upon a second convolutional network. In other words, the general image processing model can be built upon the second convolutional network and configured with a parameter tuning interface. The first convolutional network, trained to meet different user needs or image processing metrics, can be connected to the general image processing model to obtain the final image processing model. This example improves model training efficiency and flexibility. As an example, the server can update and deploy only the first convolutional network while retaining the general image processing model, improving deployment efficiency. As another example, the update cycle of the general image processing model is longer than that of the first convolutional network, increasing its flexibility.

[0115] Accordingly, this embodiment of the invention also provides a video playback method. It can be applied to any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, tablets, etc.), and PCs. In one example, the electronic device can execute the method of this embodiment based on a machine learning software framework in conjunction with hardware configurations such as a GPU. The video playback method includes:

[0116] Acquire video data.

[0117] Based on image processing methods, super-resolution processing is performed on the video data.

[0118] Play back video data after it has been processed to super-resolution.

[0119] The image processing method can be any of the image processing methods described in the above embodiments, and the video playback method in this embodiment can be applied to... Figure 1A The architecture shown.

[0120] The communication module 111 can be used to acquire video data. The image processing module 112 can call the image processing model 114 to perform image processing on the video data. The video playback method 113 can play the super-resolution processed video data.

[0121] In other examples, the super-resolution algorithm includes a wavelet processing algorithm. The first convolutional network includes a linearly stacked N-level convolutional group for wavelet transform and an N-level convolutional group for inverse wavelet transform. The convolution parameters of the N-level convolutional group for wavelet transform are linearly superimposed with pre-defined image frequency parameters characterizing the wavelet transform and with parameters obtained through training. The N-level convolutional group for inverse wavelet transform is linearly superimposed with pre-defined image frequency parameters characterizing the inverse wavelet transform and with parameters obtained through training. In the N-level convolutional group for wavelet transform, the input of the convolution representing the i-th level low-frequency information and the input of the convolution representing the i-th level high-frequency information are connected to the output of the convolution representing the (i-1)-th level low-frequency information. In the N-level convolutional group for inverse wavelet transform, the output of the convolution representing the (i-1)-th level low-frequency information and the output of the convolution representing the (i-1)-th level high-frequency information are connected to the input of the convolution representing the i-th level low-frequency information. i is an integer, and 1 ≤ i ≤ N, where N is related to the accuracy of wavelet processing; the larger the value of N, the higher the accuracy of wavelet processing. This embodiment improves the efficiency of image wavelet processing while reducing the cost of image wavelet processing.

[0122] like Figure 4B The diagram illustrates an N-level convolutional group for wavelet transform and an N-level convolutional group for inverse wavelet transform. The convolutional parameters for each convolutional group include weight parameters and [number of output channels m; number of input channels n; kernel size k], bias B [number of output channels], and padding parameters, where the padding parameters can be preset to 0 or other values. Furthermore, the number of output channels m and the number of input channels n corresponding to each channel plot can be any predetermined value, as long as the number of output channels m of one channel plot is the same as the number of input channels n of the next channel plot. In this example, m can be 1 or 8, and n can be 1 or 8. The kernel size k corresponding to each channel plot can also be predefined as any value, and k is related to the wavelet processing type; for example, when k is 2, the wavelet processing is Haar wavelet processing. In this example, kernel sizes of 1, 3, or 5 are used as examples, but it should not be understood that the above-mentioned number of input channels n, number of output channels m, and kernel size k are limited. Convolutional group 4001 represents the input feature map obtained by convolution processing. Convolutional group 4011 represents the initial wavelet decomposition of the input feature map to obtain low-frequency information L0 and high-frequency information H0, which are then linearly superimposed with convolutional group 4012 to obtain an enhanced representation of high-frequency information H0. Convolutional group 4021 represents low-frequency information L0. Convolutional group 4022 represents the next-level wavelet decomposition of low-frequency information L0 to obtain low-frequency information L1 and high-frequency information H1. After decomposition, it is linearly superimposed with convolutional group 4023, and then linearly superimposed with 4024 to obtain an enhanced representation of low-frequency information L0. It should be understood that... Figure 4BThe example only shows two-level decomposition of the image frequency feature, and more convolution groups can be used to continue the L1 decomposition to obtain L2 and H2. The input frequency feature and the output frequency feature of each level of decomposition can satisfy the following relationship: L(i-1) = Li+Hi, i is an integer greater than 1 and less than N.

[0123] The convolution group 4031 is used to perform inverse wavelet transform on the image frequency information after decomposition. The inverse wavelet transform can use the same structure as the wavelet transform. It should also be understood that the input of the convolution group 4001 described above can be an image input or an image feature map.

[0124] In addition, the convolution parameters of the first convolutional network represent the image parameters through the basic image parameters and the adjustment parameters. The image parameters are functions of the basic image parameters and the adjustment parameters, and the adjustment parameters are input via a parameter adjustment interface of the image processing model. For example, the parameter adjustment interface can be configured in a human-computer interaction interface of a device running the image processing model, and a user can set the adjustment parameters through the parameter adjustment interface. It should be understood that the parameter adjustment interface can be implemented as a resolution setting (for example, a corresponding relationship between the resolution and the adjustment parameter is established) in the human-computer interaction interface to adjust the video quality or the image quality. For example, the parameter adjustment interface is associated with a resolution setting option in a video player. In this example, I described above can be an example of the adjustment parameter.

[0125] In addition, the convolutional network structure will be combined with the image processing method described above. Figure 4C Another example of an image processing method will be described. Unlike the case of Figure 4B , the network structure is different, Figure 4C The wavelet processing process when i = 0 is shown, and Figure 4C The network includes convolution groups 41-43 for wavelet transform and a convolution group 44 for inverse wavelet transform.

[0126] Specifically, the convolution group 41 represents the input feature map after convolution processing, the convolution group 42 represents the wavelet decomposition of the input feature map to obtain low-frequency information L0 and high-frequency information H0, and then linearly superimposed with the convolution group 43.

[0127] The convolution group 44 is used to perform inverse wavelet transform on the image frequency information after decomposition. The inverse wavelet transform can use the same structure as the wavelet transform. It should also be understood that the input of the convolution group 41 described above can be an image input or an image feature map.

[0128] In other examples, Figure 5 The convolutional network structure according to another embodiment of the present application is shown, Figure 5 The convolutional network structure of Figure 2A or Figure 2B , Figure 3 ,Figure 4A 、 Figure 4B and Figure 4C one combination example of the structure in Figure 5 The network structure of the present embodiment includes an edge processing network, a bilateral processing network and an up-sampling network connected in sequence. The input image is processed by the edge processing network, and the obtained feature map is input to the bilateral filtering network for further processing. Then the feature map obtained by the bilateral filtering processing is input to the up-sampling network to obtain the final output image. It should be understood that the convolutional network of wavelet processing can be connected at any position between the above-mentioned networks, and the present embodiment does not limit this. It should also be understood that the up-sampling network can also be connected to a sharpening processing network. Each of the above-mentioned networks can be connected and implemented by using each convolution group, and Figure 5 The convolutional network of the present embodiment can be trained as a whole.

[0129] Figure 6 is a structural block diagram of an image processing device according to another embodiment of the present application. The scheme of the present embodiment can be applied to any appropriate electronic device with data processing capability, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PC machines, etc. In one example, the electronic device can execute the method of the present embodiment based on a machine learning software framework in cooperation with a hardware configuration such as a GPU. Figure 6 The image processing device of the present embodiment includes:

[0130] The acquisition module 610 acquires a to-be-processed image.

[0131] The processing module 620 inputs the to-be-processed image into an image processing model to perform image processing according to a super-resolution algorithm. The image processing model includes a first convolutional network and a second convolutional network connected to each other, convolutional parameters of the first convolutional network are trained to represent image parameters of the super-resolution algorithm, and convolutional parameters of the second convolutional network are preset to represent operator parameters of the super-resolution algorithm.

[0132] In the scheme of the present embodiment, the convolutional parameters of the first convolutional network are trained to represent image parameters, and the convolutional parameters of the second convolutional network are preset to represent operator parameters, so that the super-resolution algorithm is efficiently and reliably implemented by the convolutional network, the efficiency of image super-resolution processing is improved, and the low cost of the image processing model is ensured by using a simple convolutional network structure.

[0133] In some examples, the super-resolution algorithm comprises a bilateral filtering algorithm, the image parameters comprise respective pixel difference values between respective pixel values and a center pixel value, and a standard deviation of the respective pixel difference values, and the operator parameters are used to determine respective pixel values processed based on the bilateral filtering algorithm according to the respective pixel difference values and the standard deviation, and the first convolutional network comprises a first convolutional group representing the respective pixel difference values and a second convolutional group representing the standard deviation which are linearly superimposed.

[0134] In some examples, the super-resolution algorithm comprises a wavelet processing algorithm, the first convolutional network comprises N-level convolutional groups for wavelet transform and N-level convolutional groups for inverse wavelet transform, in the N-level convolutional groups for wavelet transform, an output of a convolution representing i-1 level low frequency information and an output of a convolution representing i-1 level high frequency information are connected to an input of a convolution representing i level low frequency information, and in the N-level convolutional groups for inverse wavelet transform, an output of a convolution representing i-1 level low frequency information and an output of a convolution representing i-1 level high frequency information are connected to an input of a convolution representing i level low frequency information, where i is an integer and 1≤i≤N.

[0135] In some examples, the super-resolution algorithm comprises an edge extraction algorithm, the operator parameters comprise a plurality of edge extraction operators of the edge extraction algorithm, and the image parameters are used to linearly combine the plurality of edge extraction operators.

[0136] In some examples, the first convolutional network comprises a reparameterization convolution obtained by reparameterizing convolutional parameters of a plurality of convolutional groups, where the convolutional parameters of the plurality of convolutional groups respectively represent operator parameters of the plurality of edge extraction operators.

[0137] In some examples, the super-resolution algorithm comprises a sharpening processing algorithm, the image parameters comprise a sharpening intensity factor, and the operator parameters are used to perform sharpening processing according to the sharpening intensity factor.

[0138] In some examples, the image processing model is generated by connecting the first convolutional network to an open interface of a general image processing model, and the general image processing model is constructed based on the second convolutional network.

[0139] The apparatus of the present embodiment is used to implement the corresponding method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here again. In addition, the function implementation of each module in the apparatus of the present embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will not be described here again.

[0140] Figure 5A step flowchart of a model training method according to another embodiment of the present application. The scheme of the present embodiment can be applied to any appropriate electronic device with data processing capability, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.), and PC machines, etc. In one example, the electronic device can execute the method of the present embodiment based on a machine learning software framework in cooperation with a hardware configuration such as a GPU. Figure 5 The model training method comprises:

[0141] S510: Obtain training data. The training data includes initial image data and image super-resolution data.

[0142] S520: Train a neural network based on the training data to obtain an image processing model.

[0143] The image processing model is used for image processing based on a super-resolution algorithm, and the neural network comprises a first convolutional network and a second convolutional network connected to each other,

[0144] The convolutional parameters of the first convolutional network are adjusted to represent image parameters in the super-resolution algorithm, and the convolutional parameters of the second convolutional network are pre-set to represent operator parameters in the super-resolution algorithm.

[0145] Figure 7B A structural block diagram of a model training device according to another embodiment of the present application. The scheme of the present embodiment can be applied to any appropriate electronic device with data processing capability, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.), and PC machines, etc. In one example, the electronic device can execute the method of the present embodiment based on a machine learning software framework in cooperation with a hardware configuration such as a GPU. Figure 7B The model training device comprises:

[0146] An obtaining module 710 obtains training data.

[0147] A training module 720 trains a neural network based on the training data to obtain an image processing model, the image processing model being used for image processing based on a super-resolution algorithm, and the neural network comprising a first convolutional network and a second convolutional network connected to each other. The convolutional parameters of the first convolutional network are adjusted to represent image parameters in the super-resolution algorithm, and the convolutional parameters of the second convolutional network are pre-set to represent operator parameters in the super-resolution algorithm.

[0148] In the scheme of the embodiment of the present application, the convolution parameters of the first convolution network are represented via training image parameters, the convolution parameters of the second convolution network are represented via presetting operator parameters, the super-resolution algorithm is efficiently and reliably implemented through the convolution network, the efficiency of image super-resolution processing is improved, and the low cost of the image processing model is ensured by using a simple convolution network structure.

[0149] The device of the embodiment is used to implement the corresponding method in the plurality of method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here. In addition, the function implementation of each module in the device of the embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will not be described here either.

[0150] Referring to Figure 8 , a structural schematic diagram of an electronic device according to another embodiment of the present application is shown, and the specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0151] As Figure 8 shown, the electronic device can include a processor 802, a communications interface 804, a memory 806, and a communications bus 808.

[0152] Among them:

[0153] The processor 802, the communications interface 804, and the memory 806 complete the communication with each other through the communications bus 808.

[0154] The communications interface 804 is configured to communicate with other electronic devices or servers.

[0155] The processor 802 is configured to execute the program 810, and specifically can execute the related steps in the foregoing method embodiments.

[0156] Specifically, the program 810 can include program code, and the program code includes computer operation instructions.

[0157] The processor 802 can be a processor CPU, or an application specific integrated circuit ASIC, or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0158] The memory 806 is configured to store a program 810. The memory 806 can include a high-speed RAM memory, and can further include a non-volatile memory such as at least one disk memory.

[0159] The program 810 can be specifically configured to enable the processor 802 to perform the following operations: obtaining a to-be-processed image; inputting the to-be-processed image into an image processing model to perform image processing according to a super-resolution algorithm. The image processing model includes a first convolutional network and a second convolutional network connected with each other, convolutional parameters of the first convolutional network are trained to represent image parameters of the super-resolution algorithm, and convolutional parameters of the second convolutional network are preset to represent operator parameters of the super-resolution algorithm.

[0160] Alternatively, the program 810 can be specifically configured to enable the processor 802 to perform the following operations: obtaining training data; training a neural network based on the training data to obtain an image processing model, the image processing model being configured to perform image processing based on a super-resolution algorithm, the neural network including a first convolutional network and a second convolutional network connected with each other, convolutional parameters of the first convolutional network being adjusted to represent image parameters in the super-resolution algorithm, and convolutional parameters of the second convolutional network being preset to represent operator parameters in the super-resolution algorithm.

[0161] In addition, specific implementation of each step in the program 810 can refer to the corresponding description in the corresponding steps and units of the foregoing method embodiments, and details are not described herein. It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, the specific working process of the foregoing described devices and modules can refer to the corresponding process description in the foregoing method embodiments, and details are not described herein.

[0162] It should be noted that, according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or part of the operations of the components / steps can be combined into a new component / step, to achieve the purpose of the embodiments of the present application.

[0163] The above-described methods according to embodiments of the application can be implemented in hardware, firmware, or software, or any combination thereof, and can be stored in or implemented with the aid of one or more data storage media (e.g., removable storage media like CD-ROM, floppy disks, hard disks, optical disks, or memory such as ROM, RAM, flash memory, etc.), or implemented with the aid of one or more processors (e.g., general-purpose, special-purpose or programmable hardware) of a processing system, or any combination thereof. The methods described herein can be stored as software or computer code on a non-transitory recording medium which can be any available media that can be read by a general- purpose or special-purpose processing system, such as a computer, processor, microprocessor controller, or programmable or dedicated hardware such as an ASIC or FPGA. It is understood that a storage medium as described herein includes one or both of a computer program product or a computer readable storage medium. By way of example, and not intending to limit the application of the principles described herein to particular programming languages, an example programming language can include an object-oriented programming system created using the Java® programming language. It is further understood that the methods described herein can be implemented using any number of programming or scripting languages such as C#, C++, Java®, JavaScript®, PHP, VBScript, Python, Ruby, etc., and that any

[0164] Those skilled in the art can understand that the units and method steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the application.

[0165] The above embodiments are only used to illustrate but not limit the embodiments of the application, and ordinary skilled in the related art can make various changes and modifications without departing from the spirit and scope of the embodiments of the application, therefore all equivalent technical solutions belong to the scope of the embodiments of the application, and the patent protection scope of the embodiments of the application should be defined by the claims.

Claims

1. An image processing method, comprising: Obtain the image to be processed; The image to be processed is input into an image processing model to perform image processing according to a super-resolution algorithm. The image processing model includes a first convolutional network and a second convolutional network that are interconnected. The convolutional parameters of the first convolutional network are trained using image parameters to characterize the super-resolution algorithm, and the convolutional parameters of the second convolutional network are preset using operator parameters to characterize the super-resolution algorithm. The super-resolution algorithm includes a wavelet processing algorithm. The first convolutional network includes linearly stacked N-level convolutional groups for wavelet transform and N-level convolutional groups for inverse wavelet transform. The convolution parameters of the N-level convolutional groups for wavelet transform are obtained by linearly stacking pre-set image frequency parameters characterizing the wavelet transform with trained parameters. The N-level convolutional groups for inverse wavelet transform are obtained by linearly stacking pre-set image frequency parameters characterizing the inverse wavelet transform with trained parameters. In an N-level convolutional group used for wavelet transform, the inputs of the convolution representing the i-th level low-frequency information and the inputs of the convolution representing the i-th level high-frequency information are connected to the output of the convolution representing the (i-1)-th level low-frequency information. In the N-level convolutional group used for inverse wavelet transform, the output of the convolution representing the (i-1)th level low-frequency information and the output of the convolution representing the (i-1)th level high-frequency information are connected to the input of the convolution representing the i-th level low-frequency information, where i is an integer and 1≤i≤N; The N-level convolutional group used for wavelet transform includes a first convolutional group, a second convolutional group, a third convolutional group, and a fourth convolutional group. The first convolutional group is used to perform convolution processing on the image to be processed or the image feature map to obtain the input feature map; The second convolutional group is used to perform primary wavelet decomposition on the input feature map to obtain first low-frequency information and first high-frequency information, and is linearly superimposed with the first parameters obtained through training to obtain an enhanced representation of the first high-frequency information. The fourth convolutional group performs a next-level wavelet decomposition on the first low-frequency information represented by the third convolutional group to obtain the second low-frequency information and the second high-frequency information. After decomposition, the information is linearly superimposed with the second parameters obtained through training, and then linearly superimposed with the third parameters obtained through training to obtain the enhanced representation of the first low-frequency information.

2. The method according to claim 1, wherein, The super-resolution algorithm includes a bilateral filtering algorithm. The image parameters include the pixel differences between each pixel value and the center pixel value, and the standard deviation of each pixel difference. The operator parameters are used to determine the pixel values ​​processed by the bilateral filtering algorithm based on the pixel differences and the standard deviation. The first convolutional network includes a first convolutional group and a second convolutional group. The first convolutional group is linearly superimposed with parameters obtained through training, and is pre-defined to represent the differences between the individual pixels. The second convolutional group is linearly superimposed with parameters obtained through training, and is pre-defined to represent the standard deviation.

3. The method according to claim 1, wherein, The super-resolution algorithm includes an edge extraction algorithm. The operator parameters include multiple edge extraction operators of the edge extraction algorithm, and the image parameters are used to linearly combine the multiple edge extraction operators.

4. The method according to claim 3, wherein, The first convolutional network includes reparameterized convolutions, which are obtained by reparameterizing the convolution parameters of multiple convolutional groups. The convolution parameters of the multiple convolutional groups respectively characterize the operator parameters of the various edge extraction operators.

5. The method according to claim 1, wherein, The super-resolution algorithm includes a sharpening algorithm, the image parameters include a sharpening intensity factor, and the operator parameters are used to perform sharpening processing based on the sharpening intensity factor.

6. The method according to claim 1, wherein, The convolution parameters of the first convolutional network characterize the image parameters through basic image parameters and adjustment parameters, wherein the image parameters are functions of the basic image parameters and the adjustment parameters, and the adjustment parameters are input via the parameter adjustment interface of the image processing model.

7. A video playback method, comprising: Acquire video data; The video data is subjected to super-resolution processing according to an image processing method, wherein the image processing method is the method according to any one of claims 1-6; Play back video data after it has been processed to super-resolution.

8. A model training method, comprising: Obtain training data; Based on the training data, the neural network is trained to obtain an image processing model. This model is used for image processing based on a super-resolution algorithm. The neural network includes interconnected first and second convolutional networks. The convolution parameters of the first convolutional network are adjusted to characterize image parameters in the super-resolution algorithm, wherein the convolution parameters of the second convolutional network are preset to characterize operator parameters in the super-resolution algorithm. The super-resolution algorithm includes a wavelet processing algorithm. The first convolutional network includes linearly stacked N-level convolutional groups for wavelet transform and N-level convolutional groups for inverse wavelet transform. The convolution parameters of the N-level convolutional groups for wavelet transform are obtained by linearly stacking pre-set image frequency parameters characterizing the wavelet transform with trained parameters. The N-level convolutional groups for inverse wavelet transform are obtained by linearly stacking pre-set image frequency parameters characterizing the inverse wavelet transform with trained parameters. In an N-level convolutional group used for wavelet transform, the inputs of the convolution representing the i-th level low-frequency information and the inputs of the convolution representing the i-th level high-frequency information are connected to the output of the convolution representing the (i-1)-th level low-frequency information. In the N-level convolutional group used for inverse wavelet transform, the output of the convolution representing the (i-1)th level low-frequency information and the output of the convolution representing the (i-1)th level high-frequency information are connected to the input of the convolution representing the i-th level low-frequency information, where i is an integer and 1≤i≤N; The N-level convolutional group used for wavelet transform includes a first convolutional group, a second convolutional group, a third convolutional group, and a fourth convolutional group. The first convolutional group is used to perform convolution processing on the training data or image feature map to obtain the input feature map; The second convolutional group is used to perform primary wavelet decomposition on the input feature map to obtain first low-frequency information and first high-frequency information, and is linearly superimposed with the first parameters obtained through training to obtain an enhanced representation of the first high-frequency information. The fourth convolutional group performs a next-level wavelet decomposition on the first low-frequency information represented by the third convolutional group to obtain the second low-frequency information and the second high-frequency information. After decomposition, the information is linearly superimposed with the second parameters obtained through training, and then linearly superimposed with the third parameters obtained through training to obtain the enhanced representation of the first low-frequency information.

9. An image processing apparatus, comprising: The acquisition module acquires the image to be processed. The processing module inputs the image to be processed into the image processing model to perform image processing according to the super-resolution algorithm. The image processing model includes a first convolutional network and a second convolutional network that are interconnected. The convolutional parameters of the first convolutional network are trained using image parameters to characterize the super-resolution algorithm, and the convolutional parameters of the second convolutional network are pre-set using operator parameters to characterize the super-resolution algorithm. The super-resolution algorithm includes a wavelet processing algorithm. The first convolutional network includes linearly stacked N-level convolutional groups for wavelet transform and N-level convolutional groups for inverse wavelet transform. The convolution parameters of the N-level convolutional groups for wavelet transform are obtained by linearly stacking pre-set image frequency parameters characterizing the wavelet transform with trained parameters. The N-level convolutional groups for inverse wavelet transform are obtained by linearly stacking pre-set image frequency parameters characterizing the inverse wavelet transform with trained parameters. In an N-level convolutional group used for wavelet transform, the inputs of the convolution representing the i-th level low-frequency information and the inputs of the convolution representing the i-th level high-frequency information are connected to the output of the convolution representing the (i-1)-th level low-frequency information. In the N-level convolutional group used for inverse wavelet transform, the output of the convolution representing the (i-1)th level low-frequency information and the output of the convolution representing the (i-1)th level high-frequency information are connected to the input of the convolution representing the i-th level low-frequency information, where i is an integer and 1≤i≤N; The N-level convolutional group used for wavelet transform includes a first convolutional group, a second convolutional group, a third convolutional group, and a fourth convolutional group. The first convolutional group is used to perform convolution processing on the image to be processed or the image feature map to obtain the input feature map; The second convolutional group is used to perform primary wavelet decomposition on the input feature map to obtain first low-frequency information and first high-frequency information, and is linearly superimposed with the first parameters obtained through training to obtain an enhanced representation of the first high-frequency information. The fourth convolutional group performs a next-level wavelet decomposition on the first low-frequency information represented by the third convolutional group to obtain the second low-frequency information and the second high-frequency information. After decomposition, the information is linearly superimposed with the second parameters obtained through training, and then linearly superimposed with the third parameters obtained through training to obtain the enhanced representation of the first low-frequency information.

10. A model training device, comprising: The module retrieves training data. The training module trains the neural network based on the training data to obtain an image processing model. This image processing model is used for image processing based on a super-resolution algorithm. The neural network includes interconnected first and second convolutional networks. The convolution parameters of the first convolutional network are adjusted to characterize image parameters in the super-resolution algorithm, wherein the convolution parameters of the second convolutional network are preset to characterize operator parameters in the super-resolution algorithm. The super-resolution algorithm includes a wavelet processing algorithm. The first convolutional network includes linearly stacked N-level convolutional groups for wavelet transform and N-level convolutional groups for inverse wavelet transform. The convolution parameters of the N-level convolutional groups for wavelet transform are obtained by linearly stacking pre-set image frequency parameters characterizing the wavelet transform with trained parameters. The N-level convolutional groups for inverse wavelet transform are obtained by linearly stacking pre-set image frequency parameters characterizing the inverse wavelet transform with trained parameters. In an N-level convolutional group used for wavelet transform, the inputs of the convolution representing the i-th level low-frequency information and the inputs of the convolution representing the i-th level high-frequency information are connected to the output of the convolution representing the (i-1)-th level low-frequency information. In the N-level convolutional group used for inverse wavelet transform, the output of the convolution representing the (i-1)th level low-frequency information and the output of the convolution representing the (i-1)th level high-frequency information are connected to the input of the convolution representing the i-th level low-frequency information, where i is an integer and 1≤i≤N; The N-level convolutional group used for wavelet transform includes a first convolutional group, a second convolutional group, a third convolutional group, and a fourth convolutional group. The first convolutional group is used to perform convolution processing on the training data or image feature map to obtain the input feature map; The second convolutional group is used to perform primary wavelet decomposition on the input feature map to obtain first low-frequency information and first high-frequency information, and is linearly superimposed with the first parameters obtained through training to obtain an enhanced representation of the first high-frequency information. The fourth convolutional group performs a next-level wavelet decomposition on the first low-frequency information represented by the third convolutional group to obtain the second low-frequency information and the second high-frequency information. After decomposition, the information is linearly superimposed with the second parameters obtained through training, and then linearly superimposed with the third parameters obtained through training to obtain the enhanced representation of the first low-frequency information.

11. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in any one of claims 1-8.

12. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-6.

Images