Image processing method, device, equipment and storage medium

Abstract

The present disclosure provides an image processing method, device, equipment and storage medium, which relate to the field of artificial intelligence, specifically computer vision and deep learning technology, and can be used in video restoration, image restoration and other scenarios. The image processing method includes: determining an image to be processed; performing fusion processing on the image to be processed to obtain fusion information; and obtaining a restoration image corresponding to the image to be processed based on the fusion information. The present disclosure can realize intelligent image processing.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, specifically to computer vision and deep learning technology, which can be used in video restoration and image restoration scenarios, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] As a special art, film is an important cultural heritage. Due to the backward preservation technology of early film films and the life of the film itself, the number of old films has decreased sharply. In order to restore old films, they need to be restored.

[0003] In the related art, old movies are usually restored manually. Summary of the invention

[0004] The present disclosure provides an image processing method, apparatus, device and storage medium.

[0005] According to one aspect of the present disclosure, an image processing method is provided, comprising: determining an image to be processed; performing fusion processing on the image to be processed to obtain fusion information; and obtaining a restored image corresponding to the image to be processed based on the fusion information.

[0006] According to another aspect of the present disclosure, an image processing device is provided, including: a determination module, used to determine an image to be processed; a fusion module, used to perform fusion processing on the image to be processed to obtain fusion information; and a processing module, used to obtain a repaired image corresponding to the image to be processed based on the fusion information.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method as described in any of the above aspects.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods according to any one of the above aspects.

[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the methods described in any one of the above aspects.

[0010] According to the technical solution of the present disclosure, intelligent image processing can be achieved.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0014] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0016] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0018] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0019] Figure 7 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0020] Figure 8 is a schematic diagram according to an eighth embodiment of the present disclosure;

[0021] Fig. 9 is a schematic diagram according to a ninth embodiment of the present disclosure;

[0022] Fig.10 is a schematic diagram according to a tenth embodiment of the present disclosure;

[0023] Fig.11 is a schematic diagram according to an eleventh embodiment of the present disclosure;

[0024] Fig.12 is a schematic diagram of an electronic device used to implement the image processing method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0026] In the related art, old movies are usually restored manually, but the manual method has certain drawbacks, and intelligent processing of old movies is an urgent problem to be solved.

[0027] In order to realize intelligent image processing, the present disclosure provides the following embodiments.

[0028] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure. This embodiment provides an image processing method, the method comprising:

[0029] Step 101: Determine an image to be processed.

[0030] Step 102: Perform fusion processing on the image to be processed to obtain fusion information.

[0031] Step 103: Based on the fusion information, obtain a repaired image corresponding to the image to be processed.

[0032] The execution subject of this embodiment can be called an image processing device, which can be software, hardware, or a combination of software and hardware, and can be located in an electronic device. The electronic device can be located at a server or a user terminal, the server can be a local server or a cloud, and the user terminal can include a personal computer (PC), a mobile device (such as a mobile phone, a tablet computer), a vehicle terminal (such as a car computer), a wearable device (such as a smart watch, a smart bracelet), a smart home device (such as a smart TV, a smart speaker), etc.

[0033] Image processing can be applied to a variety of scenarios, such as old movie restoration, blurry video processing, picture restoration, etc.

[0034] Wherein, taking a video as an example, the video includes multiple frames of images, and processing can be performed based on the images in the video. The image to be processed can be one or more frames of images in the video. Multiple frames refer to at least two frames. Furthermore, multiple frames of images can refer to continuous multiple frames of images in the video.

[0035] Among them, processing can be performed based on the fusion information of the image.

[0036] Among them, performing fusion processing on the image to be processed to obtain fusion information means obtaining fusion information based on different information of the image to be processed, that is, the fusion information is related to the different information based on. For example, A1 and A2 are two types of information to be fused, A=A1+A2, then A can be called fusion information that fuses A1 and A2.

[0037] In the embodiments of the present disclosure, different information to be fused may come from different images to be processed. Specifically, the images to be processed may be two adjacent frames in a video. Accordingly, different information comes from the two adjacent frames, so that the information of the two adjacent frames can be fused to obtain fused information. Alternatively, different information to be fused may come from a single frame image. For example, feature information of different scales (or called different layers) of a single frame image may be extracted, so that feature information of different scales can be fused to obtain fused information.

[0038] The above description uses the addition of two types of information as an example. It can be understood that the fusion method is not limited and can be selected according to actual needs. For example, one type of information can be convolved, and the other type of information can also be convolved, and then the convolved information can be added.

[0039] The repaired image may include at least one of the following items: a high-definition image, a frame-filled image, and a detail-filled image.

[0040] Depending on the different restored images, the fusion information can be obtained based on different images, or based on a single frame image. The specific fusion processing method can refer to the subsequent related description.

[0041] After obtaining the fusion information, a repaired image can be obtained based on the fusion information.

[0042] The image processing may include: one or more of high-definition processing, frame supplementation processing, and detail supplementation processing.

[0043] Accordingly, the repaired image may include one or more of: a high-definition image, a frame-filled image, and a detail-filled image.

[0044] A high-definition image can also be called a high-resolution image, which means an image with a higher resolution than the original image. For example, assuming that the image to be processed is I0 and the high-definition image is I1, the resolution of I1 is higher than I0.

[0045] The interpolated frame image can also be called an intermediate image, which refers to the image to be inserted between two frames of images. t and I t-1 , in order to improve coherence, you can t and I t-1Insert a frame of image between, assuming that the frame of image to be inserted is represented by Ix, then the video after the frame is inserted will include: I t-1 ,I x and I t . Among them, I x For the interpolated frame image.

[0046] A detail supplemented image refers to an image after the original image is supplemented with details. For example, if a certain area of ​​a frame image is missing or blurred, such problems can be improved through detail supplementation. Assuming that the image to be processed is I0 and the detail supplemented image is I2, the details of I2 are better than those of I0.

[0047] In the embodiment of the present disclosure, by processing the image to be processed, intelligent image processing can be achieved. Furthermore, by processing based on the fusion information of the image to be processed, the image processing effect can be improved.

[0048] As described above, when processing an image, it may include: high-definition processing, frame supplementation image and detail supplementation processing. The timing of these three types of processing is not limited. These three types of processing are described below respectively.

[0049] In some embodiments, the images to be processed are multiple frames of images to be processed in a video, and the fusing processing of the images to be processed to obtain fusion information includes: for a target image to be processed in the multiple frames of images to be processed, fusing the target image to be processed and adjacent images of the target image to be processed to obtain the fusion information.

[0050] If not specifically stated, the embodiments of the present disclosure will take the processing of images in a video as an example. The multiple frames of images to be processed may include multiple consecutive frames of images to be processed in the video.

[0051] For high-definition processing and frame interpolation processing, images of different frames can be fused to obtain fusion information.

[0052] Specifically, the fusion can be performed based on two adjacent frames of images (or two consecutive frames of images), and the two adjacent frames of images can be called the target image to be processed and the adjacent image of the target image to be processed. t Indicates that adjacent images may include: I t-1 , and / or, I t+1 . Where t is the time index of the image, I t-1 For I t The previous frame image can be called the forward adjacent image, I t+1 For I t The next frame of image can be called the backward adjacent image.

[0053] By fusing images of different frames, when processing a target image to be processed (hereinafter referred to as a target frame), information of other frames can be fused to improve the processing effect of the target frame.

[0054] As described above, during high-definition processing and frame interpolation processing, fusion processing can be performed based on the target image to be processed and the adjacent images of the target image to be processed, which are described below respectively.

[0055] For high definition processing, you can perform:

[0056] In some embodiments, the fusion processing adopts a multi-layer network, and the fusion processing of the target image to be processed and the adjacent images of the target image to be processed to obtain the fusion information includes: for the target layer in the multi-layer network: if the target layer is not the last layer in the multi-layer network, the first input image and the second input image are used as the input images of the target layer, and the input image of the target layer is processed to obtain the output image of the target layer, wherein the first input image is the output image of the target image to be processed in the previous layer of the target layer, and the second input image is the output image of the adjacent image of the target image to be processed in the target layer; if the target layer is the last layer of the multi-layer network, the output image of the target image to be processed in the previous layer of the last layer is used as the input image of the last layer; the input images are processed to obtain the output image of the last layer; and the output image of the last layer is used as the fusion information.

[0057] Accordingly, the repaired image includes a high-definition image, the fusion information includes a forward output image and a backward output image, the adjacent images include a forward adjacent image and a backward adjacent image, the forward output image is obtained based on the target image to be processed and the forward adjacent image, the backward output image is obtained based on the target image to be processed and the backward adjacent image, and the obtaining of the repaired image corresponding to the image to be processed based on the fusion information includes:

[0058] An addition operation is performed on the forward output image and the backward output image to obtain an added image, and the added image is used as a high-definition image corresponding to the target image to be processed.

[0059] Among them, the fusion processing during high-definition processing can be processed by a deep neural network, and the deep neural network can be a convolutional neural network and / or a recurrent convolutional neural network.

[0060] like Figure 2 As shown, the target image to be processed is I t Indicates that the high-definition image corresponding to the target image to be processed is represented by H N (It ) indicates that the deep neural network has a total of N (N is a positive integer) layers.

[0061] Among them, in an N-layer network, each layer can be used as a target layer. The previous layer of the target layer refers to a layer whose output is used as the input of the target layer. For example, the image first enters the first layer, and the image output by the first layer can enter the second layer. Then the previous layer of the second layer is the first layer.

[0062] Furthermore, for the first layer, the output image of the previous layer can be a set value, for example, combined with the subsequent: initial value, That is, the image to be processed I t As the input image of the first layer.

[0063] Among them, performing an addition operation on the forward output image and the backward output image to obtain an added image means performing an addition operation on matrices corresponding to the images to obtain an added matrix, and using the image corresponding to the added matrix as the added image.

[0064] Specifically, images can be represented by matrices during calculations. For example, an image can be represented by matrix A, and the values ​​of each element in matrix A can be the pixel values ​​of the corresponding pixels in the image. Similarly, another image can be represented by matrix B. After that, A+B can be calculated, and the image corresponding to A+B can be used as the added image.

[0065] For example, combined with the following:

[0066] in, is the forward output image, is the backward output image, H N (I t ) is the target image to be processed I t Corresponding high-definition images.

[0067] Figure 2 The deep neural network includes a forward network and a backward network. The forward network is used to process the target image It and the forward adjacent image I of the target image t-1 The backward network is used to process the target image I t and the backward adjacent image I of the target image to be processed t+1 The output image of the last layer (Nth layer) of the forward network is processed. It can be called the forward output image, and the output image of the last layer (Nth layer) of the backward network is represented by It can be called the backward output image. The output image of the i-th layer of the forward network is represented by Indicates that the output image of the i-th layer of the backward network is expressed as express.

[0068] See also Figure 2 , target image to be processed I t The corresponding high-definition image H N (I t ), is the forward output image and the backward output image The image after addition.

[0069] Based on the target image to be processed I t and the forward adjacent image I of the target image to be processed t-1 get, Based on the target image to be processed I t and the backward adjacent image I of the target image to be processed t+1 get.

[0070] i is a positive integer. When i is greater than or equal to 1 and less than N, the i-th layer of the forward network (similar to the backward network) can be a recurrent convolutional layer, and the N-th layer of the forward network (similar to the backward network) can be a convolutional layer. Figure 2 The circular curve in represents a loop operation, which may be a loop convolution operation.

[0071] See also Figure 3 , is a schematic diagram of the structure of the i-th layer of the forward network when i is greater than or equal to 1 and less than N. Figure 3 As shown, at the i-th layer, the output image of the i-th layer is based on the target image to be processed I t Output image at layer i-1 And, the forward adjacent image I t-1 Output image at layer i Obtain, among which, It can be called the first input image, is called the second output image. That is, the output image of the target image to be processed at the i-th layer Based on the first input image and the second input image get.

[0072] Specifically, the first input image can be convolved (represented by conv-i1), and the second input image can also be convolved (represented by conv-i2). The convolution kernels corresponding to the two convolutions can be the same or different, and are generally different, for example, they can be represented by W and V respectively. After that, the values ​​after the two convolutions are calculated and added to the preset bias value of the layer (represented by bias i), and then the added value is activated, for example, relu activation, to obtain

[0073] The above describes the processing flow of the i-th layer of the forward network (similar to the backward network) when i is greater than or equal to 1 and less than N.

[0074] When i=N, that is, the processing flow of the Nth layer of the forward network (similar to the backward network) can be seen in Figure 4 .like Figure 4 As shown, at layer N, the output image of layer N is is based on the target image to be processed I t Output image at layer N-1 get.

[0075] Specifically, Perform convolution (expressed as conv-N), add the convolved value to the preset bias value of the layer (expressed as bias N), and use the added value as the output image of the target image to be processed in the Nth layer. That is, corresponding to the forward network, is the forward output image, and corresponding to the backward network, is the backward output image.

[0076] The above high-definition processing process can be expressed as follows:

[0077]

[0078]

[0079]

[0080]

[0081]

[0082] Among them, the initial value,

[0083] In addition, use I t0 represents the initial frame (the frame before the first frame), then the initial value, Or other set values.

[0084] In the above formula, Represents the target image to be processed I t At the output of the i-th layer of the forward network, Represents the output of the target image to be processed in the i-th layer of the backward network, 1≤i≤N; Represents the forward adjacent image I t-1 At the output of the i-th layer of the forward network, Represents the backward adjacent image I t+1 At the output of the i-th layer of the backward network, 1≤i<N; is the preset bias value of the i-th layer of the forward network, is the preset bias value of the i-th layer of the backward network, 1≤i≤N; W i f and V i f (1≤i<N) and is the coefficient corresponding to the convolution operation of the i-th layer and the N-th layer of the forward network, W i b and V i b (1≤i<N) and H is the coefficient corresponding to the convolution operation of the i-th layer and the N-th layer of the backward network; N (I t ) is the target image to be processed I t The corresponding high-definition image, is the forward output image, is the backward output image; * is the convolution operation.

[0085] The above t is the frame number index of the image. For example, a video includes (2T+1) frames, and each frame image is represented as: -T ,I -T+1 , ..., I 0 , ..., I T-1 ,I T , then t=-T,-T+1,...,T-1,T.

[0086] Furthermore, images from different videos can also be represented with videos, for example, t L Represents the image in the video numbered L.

[0087] The target image to be processed and the adjacent images are processed by the deep neural network. Further, before the deep neural network processing, each image in the video can be upsampled, and then the upsampled images are processed. That is, the above I t ,I t-1 and I t+1 It can be an image obtained by upsampling the original image.

[0088] Specifically, during upsampling, a bilinear interpolation method may be used to upsample the original image according to a specific magnification.

[0089] By obtaining the output image of the target layer based on the target image to be processed and the adjacent images of the target image to be processed when the target layer is not the last layer, the target image to be processed and its adjacent images can be fused to improve the high-definition processing effect.

[0090] Furthermore, by dividing the network into a forward network and a backward network, a high-definition image is obtained based on the output image of the forward network and the output image of the backward network. Since the forward network is based on the target image to be processed and the forward adjacent image, and the backward network is based on the target image to be processed and the backward adjacent image, the high-definition image can refer to the information of the target image to be processed and the front and back adjacent images of the target image to be processed, thereby improving the high-definition processing effect.

[0091] For frame interpolation processing, you can execute:

[0092] In some embodiments, the target image to be processed and the adjacent image of the target image to be processed are fused to obtain the fusion information, including: determining a difference image between the target image to be processed and the adjacent image; obtaining fusion information based on the difference image; the repaired image includes a supplementary frame image, and obtaining the repaired image corresponding to the image to be processed based on the fusion information includes: using the fusion information as the supplementary frame image.

[0093] Correspondingly, obtaining fusion information based on the difference image includes: determining a weight of a convolution operation based on the difference image; and performing a convolution operation on the difference image based on the weight to obtain the fusion information.

[0094] Among them, the fusion processing during frame interpolation processing can be processed by a deep neural network, and the deep neural network can be a convolutional neural network.

[0095] like Figure 5 As shown, it is assumed that the target image to be processed is I t Indicated by I, adjacent images t-1 Indicates that the image to be inserted between two adjacent frames of images, that is, the interpolated frame image is represented by z t It is understood that the present embodiment is described by taking the forward adjacent image as an example, and the adjacent image may also be represented by the backward adjacent image I t+1 .

[0096] like Figure 5 As shown, skip-conv can be used to transform two adjacent frames of image I t and I t-1 Processing is performed to obtain the interpolated frame image. For more information about skip-conv, see Figure 6 .

[0097] Generally speaking, it can be based on the target image to be processed I t , to I t Perform convolution to obtain the complement image z tConvolution, for example, uses a deep convolutional neural network (DCNN), but this method requires a huge amount of computation.

[0098] In order to reduce the amount of calculation and improve the synthesis effect of the interpolated image, some operations can be introduced into the ordinary DCNN, and the modified network can be called skip-conv.

[0099] like Figure 6 As shown, we can first analyze the two adjacent frames of image I t and I t-1 Perform a difference operation to obtain a difference image D t , and then to D t Perform convolution operations, such as DCNN operations, to obtain the convolved image w*D t , and then with the known value z t-1 Add and get the complement image z t .

[0100] It can be expressed as:

[0101] z t =w*x t =w*x t-1 +w*x t -w*x t-1

[0102] =z t-1 +w*(x t -x t-1 )

[0103] =z t-1 +w*D t

[0104] Among them, z t For image I t and I t-1 The interpolated frame image between them, * is the convolution operation, w is the coefficient of the convolution operation, z t-1 The initial value is z 0 =w*I 0 , I 0 is the initial image, which can be 0 or other set values, usually 0, so z 0 Can be 0.

[0105] Due to the particularity of the video, the similarity between every two frames is relatively high. t Only some areas have non-zero values. tThe convolution is applied to the rest of the image except for the non-zero part. The results are the same at time t and time t-1. Therefore, when synthesizing the middle frame of the video (the interpolated frame image), the weight of the unimportant part can be directly reduced. t The non-zero part of t The non-zero part of can be achieved by adjusting the convolution weights, which can be done by adding a gating function to the calculation formula, namely:

[0106] z t =z t-1 +g(D t )⊙w*D t

[0107] Among them, ⊙ is the Hadamard product, g(D t ) is the gating function, and its expression is as follows:

[0108] g(D t )=round(σ||w*D t ||-ε)

[0109] Among them, σ is the sigmoid function, ε is the preset threshold, and round is the rounding function.

[0110] The DCNN model can be used for convolution operation. The basic components of the DCNN model are convolution module and deconvolution module. The convolution module is designed with reference to the standard convolution neural network. It consists of convolution layer and activation function layer arranged alternately and repeatedly three times, and a pooling layer is added at the end. For the convolution layer, the receptive field size, stride and inner margin are set to 3, 1, and 1 respectively. The activation function layer uses PRelu as the activation function; the receptive field size of the pooling layer is 3 and the stride is 1. Every time the data passes through the convolution module, the spatial size is halved. The second half of the model network is constructed using the deconvolution module. Each deconvolution module contains a convolution transpose layer and two convolution layers. The parameters of the convolution layer are consistent with the parameters of the convolution layer in the convolution module. The receptive field size of the convolution transpose layer is 4, the stride is 1, and the inner margin is 1. Every time the data passes through the deconvolution module, the spatial size doubles. In this way, after the input data passes through 5 convolution modules and 5 deconvolution modules, the spatial size remains unchanged. It is understandable that the above specific values ​​are examples and can be set according to actual needs.

[0111] When skip-conv is introduced, the convolutional layer of DCNN can be processed by adding a gating function to each convolutional layer to distinguish the valid area from the invalid background area so as to perform convolution operation on the valid area.

[0112] By obtaining a supplementary frame image based on a difference image between a target image to be processed and its adjacent images, the amount of computation can be reduced, the computation efficiency can be improved, and the accuracy of the synthesized supplementary frame image can be improved.

[0113] By determining weights based on the difference image and performing convolution operations based on the weights, the amount of computation can be further reduced and the computational efficiency can be improved.

[0114] The above description of high-definition processing and frame supplementation processing is based on the fusion of two adjacent frames of images. For detail supplementation processing, the features of different layers of a single frame image can be fused.

[0115] For additional details, you can execute:

[0116] In some embodiments, the fusing processing of the image to be processed to obtain fusion information includes: performing semantic segmentation processing based on fusion features on the image to be processed to obtain a feature map of the image to be processed, and using the feature map as the fusion information.

[0117] Correspondingly, the repaired image includes a detail supplement image, and the method of obtaining the repaired image corresponding to the image to be processed based on the fusion information includes: taking the feature map as the input of a generation network, using the generation network to process the feature map to obtain the output of the generation network, and using the output of the generation network as the detail supplement image, wherein the generation network is obtained by training a generative adversarial network (GAN) using a temporal consistency algorithm.

[0118] Among them, Figure 7 As shown, the corresponding single frame image I to be supplemented with details t , you can first t Perform semantic segmentation to obtain the feature map of the single frame image, use the feature map as the input of the generative network, and use the generative network to process the feature map to obtain the output of the generative network. The output of the generative network is I t Corresponding details in supplementary images.

[0119] Among them, the semantic segmentation network can fuse the input image I tThe semantic segmentation network is a network that combines the underlying features and high-level features of the image to be processed, that is, the fused features are the features after the underlying features and high-level features of the image to be processed are fused. The semantic segmentation network can be specifically a Deeplab v3+ network, whose network is an "encoding-decoding" structure. The encoding module uses DCNN for feature extraction, and then connects the spatial pyramid pooling module (Atrous Spatial Pyramid Pooling, ASPP) of the dilated convolution to extract and fuse the multi-scale features of the image. The decoding module uses upsampling to obtain the segmentation result. Because the Deeplab v3+ model introduces multi-scale information, compared with other image segmentation models, it further fuses the underlying features with the high-level features, greatly improving the accuracy of boundary segmentation.

[0120] For the generated network, Figure 8 As shown, it can be specifically divided into a G1 network and a G2 network. The G1 network can be called a global generation network, and the G2 network can be called a local generation network. The specific structure can include convolution, residual blocks and transposed convolution. The difference is that the input of the residual block of G2 also includes the output of the transposed convolution of G1, so that global information and local information can be integrated. It can be understood that the specific operations of convolution, residual blocks, and transposed convolution can refer to the existing technology and will not be described in detail here.

[0121] Optionally, the generative network can be obtained based on GAN, which is a learning model including a generative network G and a discriminative network D. The final G and D can be obtained through a learning process. In this embodiment, a generative network G after learning (or training) can be used, and the specific G can include the above-mentioned G1 and G2.

[0122] Optionally, during the training process, a deep video prior (DVP) algorithm may be used to train the GAN model. The DVP algorithm is an algorithm that can solve the problem of temporal stability.

[0123] Optionally, since the detail supplementation is performed on a single frame image, there may be video discontinuity, jitter, and flickering in the video. In order to solve the jitter and flicker problems, the DVP algorithm can be used. The DVP algorithm uses a single video to train a convolutional network from scratch to achieve temporal consistency. Compared with other algorithms that ensure temporal consistency, this method has a simple structure, and can achieve better temporal consistency and retain the original algorithm effect to a greater extent. In addition, the iterative weighting (IRT) strategy of the model also solves the multimodal temporal inconsistency problem well.

[0124] By performing semantic segmentation processing based on fusion features on the image to be processed, the segmentation effect of instances in the image (such as each person, vehicle, tree, etc. in the image) can be improved, thereby improving the detail supplement effect.

[0125] The generative network that generates detailed supplementary images is obtained by training GAN using a temporal consistency algorithm, which can reduce problems such as jitter and flicker in the video.

[0126] Various processes are described above. In addition, the image processing process may also include a process of interacting with the user.

[0127] In some embodiments, determining the images to be processed includes: by default, using all images in the video to be processed as the images to be processed; or, based on the user's selection, using images selected by the user as the images to be processed.

[0128] For example, if a video has (2T+1) frames of images, all of them can be used as images to be processed by default. Or,

[0129] For example, an interactive interface can be provided, and the user can select the information of the image to be processed through the interactive interface, for example, select the time information of the image to be processed (such as from the 1st minute to the 2nd minute), so that the image between the 1st minute and the 2nd minute in the video to be processed is used as the image to be processed.

[0130] By using the image selected by the user as the image to be processed, the user's personal needs can be met, and the image to be processed can be selected based on the user's needs. By using the system default image to be processed, the user operation can be simplified.

[0131] In some embodiments, obtaining a repaired image corresponding to the image to be processed based on the fusion information includes:

[0132] A preset configuration or a model selected by a user is used to obtain a restored image corresponding to the image to be processed based on the fusion information.

[0133] Among them, the various networks (convolution, etc.) used in the above processing process can be called models, network models, etc., that is, the first model can be used for high-definition processing, the second model can be used for frame supplementation processing, and the third model can be used for detail supplementation processing. The corresponding various models can have different version numbers. Taking the first model as an example, it can include version v1 and version v2, and the first model of version v1 and version v2 can be saved. After that, the user can select the first model of version v1 or version v2.

[0134] By selecting a model, the system can be equipped with the ability to publish models, and users can switch to a model with good effects for image processing at any time. By presetting the configuration model, user operations can be simplified.

[0135] Fig. 9 is a schematic diagram of a ninth embodiment of the present disclosure. This embodiment provides an image processing method. Fig.10 yes Fig. 9 Corresponding system architecture diagram, this embodiment takes movie restoration as an example, see Fig. 9 and Fig.10 , the method comprising:

[0136] Step 901: Import video data.

[0137] Among them, see Fig.10 ,The movie optimization system (may be referred to as the system) may include a user interaction ,module, and the user may upload the video to be processed to the ,system through the user interaction model.

[0138] In addition, in addition to user uploads, the movie optimization system can also be connected to the system that generates the video to be processed to obtain the video to be processed.

[0139] If the movie is a film movie, the film movie can be converted into an electronic video, and the specific conversion method can adopt existing technologies.

[0140] Step 902: Video storage.

[0141] After the system receives the video, it can store the video, for example, Fig.10 The video is stored in the module shown.

[0142] The video storage module is used to store videos. Uploaded videos, user-processed videos, and processed videos can all be saved. You can then retrieve and view related videos through the interactive interface.

[0143] Step 903: high definition video processing.

[0144] Step 904: Video frame interpolation.

[0145] Step 905: Complete the video details.

[0146] For the specific contents of steps 903-905, please refer to the above related description.

[0147] Steps 903-905 can correspond to Fig.10 High-definition video processing, video frame interpolation, and video detail supplementation.

[0148] High-definition video processing can perform super-resolution processing on videos, turning blurry videos into clear videos.

[0149] Video frame interpolation can insert synthesized video frames into adjacent video frames to make the stuck video smooth and coherent.

[0150] To supplement video details, image segmentation and GAN models can be used to complete the supplement of video details.

[0151] Step 906: Save the model.

[0152] For example, high-definition processing is based on the first model, video frame interpolation is based on the second model, and detail supplementation is based on the third model. The first model, the second model, and the third model can be saved.

[0153] Furthermore, different versions of the model can be saved, that is, different versions of the model can be obtained through different training processes. Taking the first model as an example, it can include version v1 and version v2, and the first model of version v1 and version v2 can be saved. Afterwards, the user can select version v1 or version v2 of the first model.

[0154] In addition, one or more of the first model, the second model and the third model may also be uploaded by the user.

[0155] The storage, update, and upload of models can be done through Fig.10 The video model management module shown is executed.

[0156] The video model management module is mainly used for the storage and release of trained models. Since multiple model services are called during the entire video processing process, a model management module is required to uniformly manage the released models.

[0157] The model publishing process can also include a testing phase. Users can upload videos and then test them in this module. After checking the results, if the model processing effect is good, users can choose to publish the model.

[0158] After the model is released, a version number will be automatically generated. All released models will be saved in this module. If the user wants to apply a historical version of the model, just search for the corresponding version number to find the corresponding model, and then click Re-Publish Online to re-apply it.

[0159] Step 907: Process and save the video.

[0160] That is, after the video to be processed is processed through high-definition processing, video frame filling, and video detail filling, a processed video can be obtained. Afterwards, the processed video can be saved, for example, stored in Fig.10 The video is stored in the module shown.

[0161] Step 908: Video display.

[0162] Among them, it can be displayed on the front-end page through the user interaction module.

[0163] In the disclosed embodiments, a variety of processing methods are provided, including video high definition, video frame interpolation, video detail addition, etc., to meet the needs of video processing, especially old movie processing. The video frame interpolation operation can be accelerated through the skip-conv network. By managing the model, the system can have the ability to publish models. For different videos, users can switch to models with good effects for video processing at any time. It also supports users to upload video processing models by themselves, and has strong scalability. The above processing flow is an automated process that can automatically complete the video processing process, thereby greatly saving the cost and time of manual video processing.

[0164] Fig.11 is a schematic diagram of the eleventh embodiment of the present disclosure. This embodiment provides an image processing device. The device 1100 includes: a determination module 1101, a fusion module 1102 and a processing module 1103.

[0165] The determination module 1101 is used to determine the image to be processed; the fusion module 1102 is used to perform fusion processing on the image to be processed to obtain fusion information; the processing module 1103 is used to obtain a repaired image corresponding to the image to be processed based on the fusion information.

[0166] In some embodiments, the repaired image includes at least one of the following items: a high-definition image, a frame-filled image, and a detail-filled image.

[0167] In some embodiments, the image to be processed is a plurality of frames of image to be processed in a video, and the fusion module 1102 is further used to: for a target image to be processed in the plurality of frames of image to be processed, perform fusion processing on the target image to be processed and adjacent images of the target image to be processed to obtain the fusion information.

[0168] In some embodiments, the fusion processing adopts a multi-layer network, and the fusion module 1102 is further used for: for the target layer in the multi-layer network: if the target layer is not the last layer in the multi-layer network, taking the first input image and the second input image as the input images of the target layer, processing the input images of the target layer to obtain the output image of the target layer, wherein the first input image is the output image of the target image to be processed in the previous layer of the target layer, and the second input image is the output image of the adjacent image of the target image to be processed in the target layer; if the target layer is the last layer of the multi-layer network, taking the output image of the target image to be processed in the previous layer of the last layer as the input image of the last layer; processing the input images to obtain the output image of the last layer; and using the output image of the last layer as the fusion information.

[0169] In some embodiments, the repaired image includes a high-definition image, the fusion information includes a forward output image and a backward output image, the adjacent images include a forward adjacent image and a backward adjacent image, the forward output image is obtained based on the target image to be processed and the forward adjacent image, and the backward output image is obtained based on the target image to be processed and the backward adjacent image, and the processing module 1103 is further used to: add the forward output image and the backward output image to obtain an added image, and use the added image as the high-definition image corresponding to the target image to be processed.

[0170] In some embodiments, the fusion module 1102 is further used to: determine a difference image between the target image to be processed and the adjacent image; obtain fusion information based on the difference image; the repaired image includes a supplementary frame image, and the processing module 1103 is further used to: use the fusion information as the supplementary frame image.

[0171] In some embodiments, the fusion module 1102 is further used to: determine a weight of a convolution operation based on the difference image; and perform a convolution operation on the difference image based on the weight to obtain the fusion information.

[0172] In some embodiments, the fusion module 1102 is further used to: perform semantic segmentation processing on the image to be processed based on fusion features to obtain a feature map of the image to be processed, and use the feature map as the fusion information.

[0173] In some embodiments, the repaired image includes a detail supplement image, and the processing module 1103 is further used to: use the feature map as the input of the generation network, use the generation network to process the feature map to obtain the output of the generation network, and use the output of the generation network as the detail supplement image, and the generation network is obtained by training GAN using a temporal consistency algorithm.

[0174] In the disclosed embodiment, by processing the image to be processed in the video to be processed, intelligent image processing can be achieved. Furthermore, by processing based on the fusion information of the image to be processed, the image processing effect can be improved.

[0175] It can be understood that in the embodiments of the present disclosure, the same or similar contents in different embodiments can be referenced to each other.

[0176] It can be understood that the “first”, “second”, etc. in the embodiments of the present disclosure are only used for distinction and do not indicate the degree of importance, time sequence, etc.

[0177] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0178] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0179] Fig.12 A schematic block diagram of an example electronic device 1200 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0180] like Fig.12 As shown, the electronic device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the electronic device 1200 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0181] Multiple components in the electronic device 1200 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the electronic device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0182] The computing unit 1201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform the image processing method in any other appropriate manner (e.g., by means of firmware).

[0183] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0184] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0185] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0186] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0187] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0188] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0189] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0190] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: Determine the image to be processed; Performing fusion processing on the image to be processed to obtain fusion information; Based on the fusion information, obtaining a repaired image corresponding to the image to be processed; The images to be processed are multiple frames of images to be processed in a video, and the fusion processing is performed on the images to be processed to obtain fusion information, including: For a target image to be processed in the multiple frames of images to be processed, performing fusion processing on the target image to be processed and an adjacent image of the target image to be processed to obtain the fusion information; The repaired image includes a high-definition image, the fusion information includes a forward output image and a backward output image, the adjacent images include a forward adjacent image and a backward adjacent image, the forward output image is obtained based on the target image to be processed and the forward adjacent image, the backward output image is obtained based on the target image to be processed and the backward adjacent image, and the obtaining of the repaired image corresponding to the image to be processed based on the fusion information includes: An addition operation is performed on the forward output image and the backward output image to obtain an added image, and the added image is used as a high-definition image corresponding to the target image to be processed.

2. The method according to claim 1, wherein: The repaired image also includes at least one of the following items: a frame-filled image and a detail-filled image.

3. The method according to claim 1, wherein: The fusion process uses a multi-layer network, and the fusion process is performed on the target image to be processed and the adjacent image of the target image to be processed to obtain the fusion information, including: For the target layer in the multi-layer network: If the target layer is a non-last layer in the multi-layer network, taking the first input image and the second input image as input images of the target layer, processing the input image of the target layer to obtain an output image of the target layer, wherein the first input image is an output image of a layer before the target image to be processed, and the second input image is an output image of an adjacent image of the target image to be processed at the target layer; If the target layer is the last layer of the multi-layer network, the output image of the target image to be processed in the previous layer of the last layer is used as the input image of the last layer; the input image is processed to obtain the output image of the last layer; and the output image of the last layer is used as the fusion information.

4. The method according to claim 1, wherein: The step of fusing the target image to be processed and an adjacent image of the target image to be processed to obtain the fusion information includes: Determine a difference image between the target image to be processed and the adjacent image; Based on the difference image, obtaining fusion information; The repaired image includes a frame-filled image, and obtaining the repaired image corresponding to the image to be processed based on the fusion information includes: The fusion information is used as the interpolated frame image.

5. The method according to claim 4, wherein: The obtaining fusion information based on the difference image includes: Determining a weight for a convolution operation based on the difference image; Based on the weight, a convolution operation is performed on the difference image to obtain the fusion information.

6. The method according to claim 1, wherein: The performing fusion processing on the image to be processed to obtain fusion information includes: The image to be processed is subjected to semantic segmentation processing based on fusion features to obtain a feature map of the image to be processed, and the feature map is used as the fusion information.

7. The method according to claim 6, wherein: The repaired image includes a detail supplemented image, and obtaining the repaired image corresponding to the image to be processed based on the fusion information includes: The feature map is used as the input of the generative network, and the feature map is processed by the generative network to obtain the output of the generative network, and the output of the generative network is used as the detailed supplementary image. The generative network is obtained by training the generative adversarial network GAN using a temporal consistency algorithm.

8. An image processing device, comprising: A determination module, used for determining an image to be processed; A fusion module, used for performing fusion processing on the image to be processed to obtain fusion information; A processing module, used for obtaining a repaired image corresponding to the image to be processed based on the fusion information; The image to be processed is a plurality of frames of image to be processed in a video, and the fusion module is further used for: For a target image to be processed in the multiple frames of images to be processed, performing fusion processing on the target image to be processed and an adjacent image of the target image to be processed to obtain the fusion information; The repaired image includes a high-definition image, the fusion information includes a forward output image and a backward output image, the adjacent images include a forward adjacent image and a backward adjacent image, the forward output image is obtained based on the target image to be processed and the forward adjacent image, the backward output image is obtained based on the target image to be processed and the backward adjacent image, and the processing module is further used to: An addition operation is performed on the forward output image and the backward output image to obtain an added image, and the added image is used as a high-definition image corresponding to the target image to be processed.

9. The device according to claim 8, wherein: The repaired image also includes at least one of the following items: a frame-filled image and a detail-filled image.

10. The device according to claim 8, wherein: The fusion process uses a multi-layer network, and the fusion module is further used for: For the target layer in the multi-layer network: If the target layer is a non-last layer in the multi-layer network, taking the first input image and the second input image as input images of the target layer, processing the input image of the target layer to obtain an output image of the target layer, wherein the first input image is an output image of a layer before the target image to be processed, and the second input image is an output image of an adjacent image of the target image to be processed at the target layer; If the target layer is the last layer of the multi-layer network, the output image of the target image to be processed in the previous layer of the last layer is used as the input image of the last layer; the input image is processed to obtain the output image of the last layer; and the output image of the last layer is used as the fusion information.

11. The device according to claim 8, wherein: The fusion module is further used to: determine a difference image between the target image to be processed and the adjacent image; and obtain fusion information based on the difference image; The repaired image includes a supplemented frame image, and the processing module is further used to: use the fusion information as the supplemented frame image.

12. The device according to claim 11, wherein The fusion module is further used for: Determining a weight for a convolution operation based on the difference image; Based on the weight, a convolution operation is performed on the difference image to obtain the fusion information.

13. The device according to claim 8, wherein: The fusion module is further used for: The image to be processed is subjected to semantic segmentation processing based on fusion features to obtain a feature map of the image to be processed, and the feature map is used as the fusion information.

14. The device according to claim 13, wherein: The repaired image includes a detail supplemented image, and the processing module is further used for: The feature map is used as the input of the generative network, and the feature map is processed by the generative network to obtain the output of the generative network, and the output of the generative network is used as the detail supplementary image. The generative network is obtained by training the generative adversarial network GAN using a temporal consistency algorithm.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images