Image processing method, image restoration method, and data processing method for image processing

Abstract

Embodiments of the present specification provide an image processing method, an image repairing method and a data processing method for image processing, wherein the image processing method comprises: obtaining a to-be-processed image; encoding the to-be-processed image by using an image coding and decoding model to obtain first image features; processing the first image features by using an image processing model to obtain second image features; and decoding the second image features by using the image coding and decoding model to obtain a target image, wherein the image coding and decoding model is obtained by training based on a prediction image and a label image, the prediction image is obtained by decoding fusion features by using the image coding and decoding model, and the fusion features are obtained by fusing sample image features of a sample image and label image features of a label image. The decoupling of the image processing model and the image coding and decoding model in training is realized, the training effect and the image processing effect are improved, and the flexibility of selection of the image processing model and the generalization of image processing are improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of image processing, and in particular, to an image processing method. BACKGROUND

[0002] With the development of neural network models, very excellent results have been achieved in image processing.

[0003] At present, an image coding and decoding model and an image processing model are jointly used to process a to-be-processed image to obtain a target image. However, the joint use of the image coding and decoding model and the image processing model brings information loss to the processing process of the to-be-processed image, resulting in poor image processing effect. SUMMARY

[0004] Therefore, embodiments of the present specification provide an image processing method. One or more embodiments of the present specification also relate to an image repairing method, another image processing method, a data processing method for image processing, an image processing device, an image repairing method, another image processing device, a data processing device for image processing, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects in the prior art.

[0005] In an embodiment of the present specification, an image processing method is provided, comprising:

[0006] obtaining a to-be-processed image;

[0007] encoding the to-be-processed image using an image coding and decoding model to obtain first image features;

[0008] processing the first image features using an image processing model to obtain second image features;

[0009] decoding the second image features using the image coding and decoding model to obtain a target image, wherein the image coding and decoding model is trained based on a predicted image and a label image, the predicted image is obtained by decoding the fusion features using the image coding and decoding model, and the fusion features are obtained by fusing sample image features of a sample image and label image features of the label image.

[0010] In an embodiment of the present specification, a to-be-processed image is obtained; an image coding and decoding model is used to encode the to-be-processed image to obtain first image features; the image processing model is used to process the first image features to obtain second image features; and the image coding and decoding model is used to decode the second image features to obtain a target image, wherein the image coding and decoding model is trained based on a prediction image and a label image, the prediction image is obtained by decoding the fusion features using the image coding and decoding model, and the fusion features are obtained by fusing sample image features of a sample image and label image features of a label image. The fusion features obtained by fusing the sample image features of the sample image and the label image features of the label image using the image coding and decoding model are decoded to pre-train the image coding and decoding model, the label image features of the label image are directly fused as image features processed by the image processing model, the introduction of the image processing model into the pre-training of the image coding and decoding model is avoided, the decoupling of the image processing model and the image coding and decoding model in training is realized, the training effect and the image processing effect are effectively improved, the selection of the image processing model has higher flexibility, and the image processing has higher generalization. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present specification;

[0012] Figure 2 is a model architecture diagram in an image processing method provided by an embodiment of the present specification;

[0013] Figure 3 is a training schematic diagram of an image coding and decoding model in an image processing scheme provided by an embodiment of the present specification;

[0014] Figure 4 is a flowchart of an image repairing method provided by an embodiment of the present specification;

[0015] Figure 5 is a flowchart of another image processing method provided by an embodiment of the present specification;

[0016] Figure 6 is a flowchart of a data processing method of image processing provided by an embodiment of the present specification;

[0017] Figure 7 is a processing process flowchart of an image processing method applied to video picture quality improvement provided by an embodiment of the present specification;

[0018] Figure 8 is a structural schematic diagram of an image processing device provided by an embodiment of the present specification;

[0019] Figure 9is a structural schematic diagram of an image repairing device provided by an embodiment of the present specification;

[0020] Figure 10 is a structural schematic diagram of another image processing device provided by an embodiment of the present specification;

[0021] Figure 11 is a structural schematic diagram of a data processing device for image processing provided by an embodiment of the present specification;

[0022] Figure 12 is a structural block diagram of a computing device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0023] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples described herein. In other instances, well-known methods have not been described in detail in order not to unnecessarily obscure the present specification.

[0024] The terminology used in one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present specification. As used in one or more embodiments of the present specification and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0025] It will be understood that, although the terms first, second, etc. can be employed in describing various information, such information should not be limited by these terms. These terms are only used to differentiate one piece of information from another. For example, a first can be termed a second, and, similarly, a second can be termed a first, without departing from the scope of one or more embodiments of the present specification. As used herein, the word "if' can be interpreted to mean "when" or "upon" or "in response to determining" depending on the context.

[0026] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0027] In one or more embodiments of the present specification, a large model refers to a deep learning model with a large number of model parameters, usually containing hundreds of millions, billions, tens of billions, hundreds of billions, or even tens of billions of model parameters. The large model can also be referred to as a foundation model. Through large-scale unlabeled corpus pre-training of the large model, a pre-trained model with hundreds of millions of parameters is output. Such a model can adapt to a wide range of downstream tasks, and the model has good generalization ability. For example, a large language model (LLM) and a multi-modal pre-training model.

[0028] In actual application, the large model only needs a small amount of samples to fine-tune the pre-trained model and can be applied to different tasks. The large model can be widely applied to natural language processing (NLP) and computer vision fields. Specifically, it can be applied to computer vision field tasks such as visual question answering (VQA), image captioning (IC), and image generation, as well as natural language processing field tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the large model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0029] First, the technical terms related to one or more embodiments of the present specification are explained.

[0030] Image processing: using computer technology to operate and process images, specifically extracting image features and performing deep feature calculation and processing on image features to achieve certain image processing tasks, including but not limited to image inpainting, image super-resolution, and image denoising.

[0031] Image inpainting: a technique for restoring missing or damaged parts of an image by inpainting the image. It is used to repair noise, damage, and missing parts in images to make them clearer and more complete.

[0032] Image super-resolution: a technique for converting low-resolution images into high-resolution images by super-resolution processing of images.

[0033] Image denoising: a technique for reducing or removing noise in digital images by denoising images.

[0034] Pixel Space: A low-dimensional, concrete feature space of an image. Each pixel in the image has a corresponding feature value, such as an RGB (Red Green Blue) value.

[0035] Latent Space: A high-dimensional, abstract feature space. By dimensionally compressing the pixel space features of an image, the key features of the image are extracted.

[0036] Diffusion Model: A reasoning and prediction model that includes a forward noise addition process and a reverse noise removal process. It generates images through forward and reverse reasoning and prediction methods. In the forward process, Gaussian noise (white noise) is added to the original input latent space features in an iterative manner. After a sufficient number of iterations, the latent space features become pure Gaussian noise. In the reverse process, the pure Gaussian noise is removed in an iterative manner. After a sufficient number of iterations, the pure Gaussian noise becomes the output latent space features.

[0037] CNN (Convolutional Neural Networks): A multi-convolution layer neural network model with forward propagation and backward propagation.

[0038] VIT Model (Vision Transformer): A translation model for visual image processing with visual attention mechanism.

[0039] U-Net Model (U-shaped Neural Network Model): A U-shaped (encoding layer size decreases and decoding layer size increases) neural network model that fuses deep and shallow layer features through concatenation.

[0040] VQGAN (Vector Quantized Generative Adversarial Network): An image encoding and decoding model with an encoder, a feature dictionary, and a decoder. The encoder maps the input pixel space features to the latent space, then determines the corresponding features of the latent space features from the feature dictionary, and then the decoder maps the determined corresponding features back to the pixel space.

[0041] VAEGAN (Variational Autoencoder Generative Adversarial Network): an image encoding and decoding model, which has an encoder and a decoder. The encoder maps the input pixel space features to the latent space, then determines the latent space features to be Gaussian distribution, and finally the decoder samples features from the Gaussian distribution and maps them back to the pixel space.

[0042] AIGC (AI Generated Content): a content generation technology based on artificial intelligence technology.

[0043] At present, through the way of freezing model parameters and multi-stage training, for example, only the image processing model is trained in the first stage, only part of the newly added training parameters of the image encoding and decoding model is trained in the second stage, and the model parameters of the image encoding and decoding model remain unchanged. However, in the training process, because the overall architecture of the image processing model is still bound with the image encoding and decoding model, the image processing model relies on the image encoding and decoding model to realize encoding and decoding, and there is no decoupling in the training, so the problem of information loss in image encoding cannot be effectively solved, the training effect is insufficient, and the image processing effect is insufficient.

[0044] To solve the above problems, the present specification provides an image processing method, and simultaneously relates to an image restoration method, another image processing method, a data processing method for image processing, an image processing device, an image restoration device, another image processing device, a data processing device for image processing, a computing device, a computer-readable storage medium, and a computer program, which are described in detail one by one in the following embodiments.

[0045] Referring to Figure 1 , Figure 1 A flowchart of an image processing method provided by an embodiment of the present specification is shown, which includes the following specific steps:

[0046] Step 102: obtaining an image to be processed.

[0047] The embodiment of the present specification is applied to a client or server of an application, website or mini-program with image processing function, and the client or server is deployed with an image encoding and decoding model and an image processing model to realize the image processing function.

[0048] The to-be-processed image is a to-be-processed visual image, which is a low-quality visual image. The to-be-processed image can be a real image collected by an image collection device, for example, a photo, a video frame, a remote sensing image, and the like, can be an artificially generated image, for example, a picture and an animation, and the like, and can also be a virtual image generated by an image generation algorithm, for example, a picture, an animation, a photo, and a video frame, and the like, which are not limited herein.

[0049] The to-be-processed image can be received from a client, for example, a photo uploaded by a user through an application front-end interface on a mobile terminal. The to-be-processed image can also be obtained from an image database, for example, an open source database, a cloud database, and the like. For example, a video image of an A area recorded by a camera in the past 24 hours is obtained from a cloud database. The to-be-processed image can also be received from an image collection device, for example, four-directional image sensors on a car send collected images to an image stitching function application in a car computer, and the four-directional collected images are stitched on the server to obtain the surrounding environment image of the car, which are not limited herein.

[0050] For example, a monitoring video stream of a road intersection in rainy weather collected and sent by a camera at the intersection is received, and a video frame of the monitoring video stream is extracted.

[0051] The to-be-processed image is obtained to provide an image basis for subsequent image processing.

[0052] Step 104: encoding the to-be-processed image by using the image coding and decoding model to obtain a first image feature.

[0053] The image coding and decoding model is a neural network model with image coding and decoding functions. The image coding and decoding model includes an encoder and a decoder, and realizes the coding and decoding conversion between image features and pixel features. The image coding and decoding model includes but is not limited to: a CNN model, a ViT model, a UNet model, a GAN model, a VQGAN model, and a VAEGAN model. The image coding and decoding model is obtained by pre-supervised training.

[0054] The first image feature is an image coding feature of the to-be-processed image, which is expressed as a feature vector of a specific scale (dimension). The first image feature represents high-dimensional and abstract key features such as structures, shapes, and textures of the to-be-processed image, for example, a feature of the to-be-processed image in a hidden space, i.e., a first hidden space feature. For example, a certain landscape photo is a to-be-repaired image, and the first image feature represents high-dimensional and abstract key features such as structures, shapes, and textures of various topographies in the landscape photo.

[0055] The image coding and decoding model is used to encode the to-be-processed image to obtain first image features. Specifically, the image coding and decoding model is used to map the pixel space features of the to-be-processed image to a latent space to obtain first latent space features.

[0056] Exemplarily, the pixel space features Pixel Feature of the video frames of the monitoring video stream are mapped to a latent space by using a pre-trained VAEGAN model to obtain first latent space features Latent Feature_1.

[0057] The image coding and decoding model is used to encode the to-be-processed image to obtain first image features, which provides a feature basis for image processing of a subsequent image processing model.

[0058] Step 106: The image processing model is used to perform image feature processing on the first image features to obtain second image features.

[0059] The image processing model is a neural network model that performs feature processing at the image feature level. Based on the first image features of the to-be-processed image, the image processing model determines the key features such as the structure, shape and texture of the to-be-processed image, and performs image processing based on these features to obtain second image features after image processing. It includes but is not limited to: a generative model and an image processing model. The generative model is an image processing model based on predictive generation. The generative model realizes image generation through a generator, for example, a generative pre-trained Transformer model (GPT, Generative Pre-Trained Transformer), a GAN model, a VQGAN model and a VAEGAN model. The diffusion model is an image processing model based on inferential prediction. The diffusion model realizes image processing through the way of adding noise forward and removing noise backward, for example, a Stable Diffusion model. Based on the first image features of the input to-be-processed image, the image processing model determines the key features such as the structure, shape and texture of the to-be-processed image, and performs image feature processing based on these features to obtain second image features. The higher the accuracy of image feature processing of the image processing model, the better the image processing effect of the processed target image obtained by subsequent decoding. In the embodiments of the present specification, the image processing model is not jointly trained with the image coding and decoding model, but is trained separately and decoupled, so that the image processing model can be changed according to the needs of image processing, for example, for portrait image processing, the image processing model 1 is trained specifically, for landscape image processing, the image processing model 2 is trained specifically, and according to the image type of the to-be-processed image, the corresponding image processing model is selected for image processing, so that the selection of the image processing model has higher flexibility, and the image processing has higher generalization.

[0060] The second image feature is an image coding feature of the processed image, which is expressed as a feature vector of a specific scale (dimension), and the second image feature represents high-dimensional and abstract key features such as structure, shape, and texture of the processed image. For example, the second image feature is a feature in a second latent space after the image processing model processes the first image feature.

[0061] The image processing model is used to process the first image feature to obtain the second image feature. Specifically, the image processing model is used to process the first latent space feature to obtain the second latent space feature.

[0062] For example, the diffusion model Stable Diffusion is used to infer and predict the first latent space feature Latent Feature_1 to obtain the second latent space feature Latent Feature_2.

[0063] The image processing model is used to process the first image feature to obtain the second image feature, which completes the image feature processing and provides a feature basis for subsequent decoding of the processed target image.

[0064] Step 108: Decoding the second image feature using an image coding and decoding model to obtain a target image, wherein the image coding and decoding model is trained based on a predicted image and a label image, the predicted image is obtained by decoding the fusion feature using the image coding and decoding model, and the fusion feature is obtained by fusing the sample image feature of the sample image and the label image feature of the label image.

[0065] The target image is a visual image processed by the image processing model, which is a high-quality visual image. The target image corresponds to the image to be processed, which can be a real image collected by an image collection device after processing, such as a processed photo, video frame, and remote sensing image, or a processed image generated by a human, such as a processed drawing and animation, or a virtual image generated by an image generation algorithm, such as a processed drawing, animation, photo, and video frame, without limitation.

[0066] The sample image is a training sample image for training the image coding model, and is a visual image to be processed, which is a low-quality visual image. For example, 10,000 visual images of A place collected under bad weather are taken as the training sample images for training the image coding model. The sample image feature of the sample image is an image coding feature of the sample image, representing high-dimensional and abstract key features such as structure, shape and texture of the sample image, for example, a feature of the sample image in a hidden space, i.e., a sample hidden space feature.

[0067] The label image is a training label image for training the image coding model, and is a visual image for image processing as a reference, which is a high-quality visual image. For example, 10,000 visual images of A place collected under good weather are taken as the training label images for training the image coding model. The label image feature of the label image is an image coding feature of the label image, representing high-dimensional and abstract key features such as structure, shape and texture of the label image, for example, a feature of the label image in a hidden space, i.e., a label hidden space feature.

[0068] The predicted image is an image output by the image coding model in the process of training the image coding model, and is a visual image after image processing of the sample image. For example, in the process of training the image coding model, the image coding model performs image inpainting on 10,000 visual images of A place collected under bad weather, and outputs 10,000 visual images, which are the predicted images.

[0069] The fusion feature is an image coding feature fused based on the sample image feature of the sample image and the label image feature of the label image, and is represented as a feature vector of a specific scale (dimension). The fusion feature represents high-dimensional and abstract key features such as structure, shape and texture of the fusion of the sample image and the label image, for example, a feature in a hidden space, i.e., a fusion hidden space feature. From the perspective of the sample image, the fusion feature is an image coding feature obtained by image processing of the sample image. For example, the image quality of the sample image is taken as 0, the image quality of the label image is taken as 1, and the image quality represented by the fusion feature is 0.5, which is image processing relative to the sample image. In the training process of the image processing model, the predicted feature output by the image processing model also represents an image quality in the interval (0, 1). Therefore, the fusion feature can be understood as the predicted feature output by the image processing model. The predicted image obtained by decoding the fusion feature can be equivalently understood as a predicted image obtained by coupling the image coding model and the image processing model in the training process. Training the image coding model in this way can achieve the same training effect, but at the same time avoids introducing the image processing model to participate in the pre-training of the image coding model, realizes the decoupling of the image processing model and the image coding model in training, and effectively improves the training effect and the image processing effect.

[0070] Optionally, the predicted image is obtained by decoding the fused feature by using the image encoding and decoding model, specifically, the predicted image is obtained by decoding the fused feature based on the multi-scale encoded feature by using the image encoding and decoding model. Further, the predicted image is obtained by performing fused decoding on the fused feature based on the multi-scale encoded feature by using the image encoding and decoding model, where the fused decoding is performed by taking the corresponding scale encoded feature as a guide. In this way, the information loss between scales caused in the image encoding process is compensated, the image distortion problem in the image processing process is effectively overcome, the training effect of the model is improved, and thus the image processing effect is improved.

[0071] The second image feature is decoded by using the image encoding and decoding model to obtain a target image, specifically, the second image feature is mapped to a pixel space by using the image encoding and decoding model to obtain the processed target image.

[0072] For example, the second latent space feature Latent Feature_2 is mapped to the pixel space by using the pre-trained VAEGAN model to obtain the video frame of the repaired monitoring video stream, and the monitoring video stream of the road intersection collected and sent by the road intersection camera under the rain-free weather is integrated.

[0073] In the embodiments of the present specification, a to-be-processed image is obtained; the to-be-processed image is input into an encoder of an image encoding and decoding model to obtain a first image feature; the first image feature is input into an image processing model to obtain a second image feature; and the second image feature is input into a decoder of the image encoding and decoding model to obtain a processed target image, where the image encoding and decoding model is trained based on a predicted image and a label image, the predicted image is obtained by decoding a fused feature by the decoder, and the fused feature is obtained by fusing a sample image feature of a sample image and a label image feature of the label image. The fused feature obtained by fusing the sample image feature of the sample image and the label image feature of the label image by using the decoder of the image encoding and decoding model is decoded to pre-train the image encoding and decoding model, the label image feature of the label image is directly fused as the latent space feature after image processing, the latent space feature predicted by the image processing model is avoided to participate in the pre-training of the image encoding and decoding model, the decoupling of the image processing model and the image encoding and decoding model in training is realized, the training effect and the image processing effect are effectively improved, and meanwhile, the selection of the image processing model has higher flexibility and the image processing has higher generalization.

[0074] In an optional embodiment of the present specification, the image encoding and decoding model includes an encoder and a decoder.

[0075] Correspondingly, step 104 includes the following specific steps:

[0076] The image to be processed is input into the encoder, and the first image feature is obtained by encoding.

[0077] Correspondingly, step 108 includes the following specific steps:

[0078] The second image features are input into the decoder to obtain the target image.

[0079] Figure 2 This specification illustrates a model architecture diagram of an image processing method according to an embodiment of this specification. The overall architecture of the image encoding / decoding model and the image processing model is as follows: Figure 2 As shown: The encoder, image processing model, and decoder of the image encoding and decoding model are connected in series. The image to be processed is input into the encoder of the image encoding and decoding model to obtain the first image feature. The first image feature is input into the image processing model and processed to obtain the second image feature. The second image feature is input into the decoder of the image encoding and decoding model to obtain the target image.

[0080] The encoder in an image encoding / decoding model is a model module used to encode image features of the input image. By encoding these features, the encoder extracts high-dimensional, abstract key features from the input image. However, information loss is inevitable during the encoding process. For example, the encoder maps the pixel space features of an image to the latent space. By encoding pixel space features into latent space features, the encoder maps the 512×512-scale pixel space features of the input image to 128×1-scale latent space features, extracting key features but also causing information loss.

[0081] The decoder in an image encoding / decoding model is a model module used to decode image features. By decoding image features, the decoder reconstructs a low-dimensional, concrete visual image based on the high-dimensional, abstract key features of the image. For example, the decoder maps the latent space features of an image to the pixel space. By decoding the latent space features into pixel space features, mapping 128×1 scale latent space features to 512×512 scale pixel space features, the decoder obtains the processed visual image.

[0082] The image to be processed is input into the encoder to obtain the first image feature. Specifically, the image to be processed is input into the encoder, and the pixel space features of the image to be processed are mapped to the latent space to obtain the first image feature.

[0083] The second image features are input into the decoder to obtain the target image. Specifically, the second image features are input into the decoder and mapped to the pixel space to obtain the target image.

[0084] Exemplarily, the video frames of the monitoring video stream are input into the pre-trained VAEGAN model, the pixel space features of the video frames of the monitoring video stream are mapped to the latent space to obtain first latent space features, the first latent space features are input into the diffusion model Stable Diffusion, the first latent space features are inferred and predicted to obtain second latent space features, the second latent space features are input into the VAEGAN model, and the second latent space features Latent Feature_2 are mapped to the pixel space to obtain the video frames of the repaired monitoring video stream, and the monitoring video stream of the road intersection in the rain-free weather collected and sent by the road intersection camera is integrated.

[0085] In the embodiments of the present specification, the decoder of the image coding and decoding model is used to decode the fusion features obtained by fusing the latent space features of the sample images and the latent space features of the label images, to pre-train the image coding and decoding model, directly fuse the latent space features of the label images as the latent space features after image processing, avoid introducing the latent space features predicted by the diffusion model to participate in the pre-training of the image coding and decoding model, realize the decoupling of the diffusion model and the image coding and decoding model in training, effectively improve the training effect and the image processing effect, and at the same time, make the selection of the diffusion model have higher flexibility and the image processing have higher generalization.

[0086] In an optional embodiment of the present specification, before step 104, the following specific steps are further included:

[0087] Obtaining a first sample set, wherein the first sample set includes a plurality of sample pairs, and each sample pair includes a sample image and a label image;

[0088] Extracting a first sample pair from the first sample set, wherein the first sample pair includes a first sample image and a first label image, and the first sample pair is any sample pair in the sample set;

[0089] Inputting the first sample image and the first label image into an encoder of an image coding and decoding model respectively, and encoding to obtain a sample image feature of the first sample image and a label image feature of the first label image;

[0090] Fusing the sample image feature of the first sample image and the label image feature of the first label image to obtain a fusion feature;

[0091] Inputting the fusion feature into a decoder of the image coding and decoding model to obtain a first predicted image;

[0092] Based on the first prediction image and the first label image, the model parameters of the image coding and decoding model are adjusted until a training stop condition is reached, and a trained image coding and decoding model is obtained.

[0093] Currently, in the scheme of training the integrated image coding and decoding model and the image processing model, the encoder and the decoder of the image processing model and the image coding and decoding model are coupled, and therefore, the training effect of the encoder and the decoder is affected by the image processing model. Since the image processing model needs to iteratively perform inference prediction, a large inference prediction overhead is generated on the training samples, and after the model parameters of the image processing model are adjusted, the encoder and the decoder need to be trained correspondingly, resulting in a very high training cost of the entire training. In the embodiments of the present specification, the number of sample images and label images is sufficient, and after decoupling, a large overhead is avoided on inference prediction, and the training cost is reduced.

[0094] The first sample set is a set of training samples for training the image coding and decoding model, and the first sample set includes a plurality of sample pairs. Each sample pair includes a low-quality sample image and a high-quality label image. For example, the first sample set includes 10,000 sample pairs, and the 10,000 sample pairs include 10,000 visual images of A place collected under bad weather and 10,000 visual images of A place collected under good weather. The first sample set can be obtained from a sample database, for example, an open source sample database, or can be artificially constructed, for example, generated by a generation model represented by an image processing model, or can be obtained from a historical database, for example, visual images before processing and visual images after processing obtained from a historical database, which is not limited here.

[0095] Feature fusion is to combine a plurality of feature vectors to form a feature vector with more information. The feature fusion methods include spatial fusion, channel fusion, and time domain fusion. Spatial fusion refers to combining a plurality of feature images in space, for example, averaging or weighted summing the feature vectors of a plurality of convolution layers. Channel fusion refers to combining a plurality of feature vectors in channels, for example, splicing or weighted summing the feature vectors of a plurality of convolution layers. Time domain fusion refers to combining a plurality of time step feature vectors, for example, averaging or weighted summing the video feature vectors of a plurality of time steps. According to the stage and complexity of feature fusion, it is divided into pre-fusion (early fusion), intermediate fusion, and post-fusion (late fusion).

[0096] The training stop condition is a pre-set judgment condition for stopping training, including but not limited to: a pre-set number of iterations, a pre-set loss value threshold, a pre-set training time, and a pre-set model convergence condition.

[0097] The first sample image and the first label image are respectively input into an encoder of the image coding and decoding model to obtain sample image features of the first sample image and label image features of the first label image. Specifically, the first sample image is input into the encoder of the image coding and decoding model to map pixel space features of the first sample image to a hidden space to obtain the sample image features of the first sample image. The first label image is input into the encoder of the image coding and decoding model to map pixel space features of the first label image to the hidden space to obtain the label image features of the first label image. It should be noted that, since the fusion features are obtained based on the sample image features of the first sample image and the label image features of the first label image, and the training of the image coding and decoding model is iteratively performed, the sample image and the label image need to be respectively input into two encoders, the encoder input with the label image is parameter frozen to ensure the coding characteristics of the encoder and further ensure the stability of the fusion features.

[0098] The fusion features are input into a decoder of the image coding and decoding model to obtain the first predicted image. Specifically, the fusion features are input into the decoder of the image coding and decoding model to map the fusion features to a pixel space based on the multi-scale coding features to obtain the first predicted image. Further, the fusion features are input into the decoder of the image coding and decoding model to map the fusion features to a pixel space of a corresponding scale based on the multi-scale coding features to obtain the first predicted image.

[0099] Based on the first predicted image and the first label image, model parameters of the image coding and decoding model are adjusted. Specifically, based on the first predicted image and the first label image, a loss value is calculated, and based on the loss value, the model parameters of the image coding and decoding model are adjusted. The loss value is a difference value between the predicted image and the label image, and is used to evaluate the model performance of the image coding and decoding model, including but not limited to cross-entropy loss, mean square error loss, average error loss and logarithmic loss.

[0100] Exemplarily, a sample set from an open source sample database is acquired, the sample set including 8000000 images from 365 different real scenes, the sample set including 4000000 sample pairs, any sample pair including a sample image and a label image. A first sample pair is extracted from the 4000000 sample pairs for training of the current image coding model, the first sample pair including a first sample image Lr_1 of a city road scene under a rainy day weather and a first label image Hr_1 of the city road scene under a sunny day weather, the first sample image is input into an encoder Encoder_A of the VAEGAN model, pixel space features of the first sample image are mapped to a latent space to obtain latent space features Z of the first sample image, the first label image is input into an encoder Encoder_B of the VAEGAN model, pixel space features of the first label image are mapped to the latent space to obtain latent space features Z' of the first label image, the latent space features of the first sample image and the latent space features of the first label image are averaged to obtain a fusion feature Fusion=1 / 2(Z+Z'), the fusion feature is input into a decoder Decoder of the VAEGAN model, based on multi-scale coding features, the fusion feature is mapped to a pixel space to obtain a first predicted image Preditct.Hr_1 through decoding, based on the first predicted image and the first label image, a loss value Loss is calculated, based on the loss value, model parameters of the encoder Encoder_A and the decoder Decoder in the VAEGAN model are adjusted until input and output of all 4000000 sample pairs are completed, and a trained VAEGAN model is obtained.

[0101] In the embodiments of the present specification, the decoder of the image coding model is used to decode the fusion feature obtained by fusing the sample image features of the sample image and the label image features of the label image, to pre-train the image coding model, directly fusing the label image features of the label image as the latent space features after image processing, avoiding the introduction of the latent space features predicted by the image processing model to participate in the pre-training of the image coding model, realizing the decoupling of the image processing model and the image coding model in training, effectively improving the training effect, and reducing the training cost of model training.

[0102] In an optional embodiment of the present specification, after the first sample image is input into the encoder of the image coding model, the following specific steps are further included:

[0103] The multi-scale coding features of the first sample image are obtained;

[0104] Correspondingly, the fusion feature is input into the decoder of the image coding model, and a first predicted image is decoded and obtained, including the following specific steps:

[0105] The decoder using the image coding model calculates multi-scale decoding features based on the multi-scale coding features and the fusion features, and decodes to obtain a first predicted image based on the multi-scale decoding features.

[0106] At present, when the encoder encodes the image, a mapping process from pixel space features to latent space features is inevitably caused to result in information loss, so that when the decoder maps back to the pixel space, the predicted image decoded by the decoder must exist image distortion, for example, the problem of text distortion in the image, and therefore, in order to avoid excessive image distortion of the predicted image, the multi-scale coding features obtained by the encoder in the encoding process must be used to compensate for the information to a certain extent, and based on this, the present specification embodiment provides a fine-tuning strategy for guiding the decoder to decode.

[0107] The image encoding performed by the encoder is to gradually map from a large-scale pixel space to a small-scale latent space, and the encoder has a plurality of coding units of different scales, for example, the encoder has five coding units, and the scales of the five coding units are: 512*512 scale, 512*256 scale, 256*128 scale, 128*32 scale and 128*1 scale. The pixel space features of a sample image of 512*512 scale are gradually mapped to the latent space features of 128*1 scale, and in this process, information loss is caused. Meanwhile, each coding unit outputs coding features of a corresponding scale, including 512*256 scale coding features, 256*128 scale coding features and 128*32 scale coding features. Correspondingly, the image decoding performed by the decoder is to gradually map from a small-scale latent space back to a large-scale pixel space, and the decoder has a plurality of decoding units of corresponding scales.

[0108] The multi-scale coding features are the coding features output by the multi-scale coding units in the encoder, and are intermediate features in the process of mapping from pixel space features to latent space features.

[0109] The multi-scale decoding features are the decoding features output by the multi-scale decoding units in the decoder, and are intermediate features in the process of mapping from latent space features to pixel space features.

[0110] The multi-scale coding features of the first sample image are obtained in the following specific manner: the coding features output by the multi-scale coding units in the encoder are obtained to obtain the multi-scale coding features of the first sample image.

[0111] The decoder of the image coding and decoding model calculates multi-scale decoding features based on the multi-scale coding features and the fusion features, and decodes a first predicted image based on the multi-scale decoding features.

[0112] Exemplarily, the encoder of the VAEGAN model includes 7 coding units of different scales, and the multi-scale coding features of the first sample image Lr_1 are obtained by obtaining coding features output by the 7 coding units of different scales. The decoder of the VAEGAN model includes 7 decoding units of different scales, and the decoding features of 7 scales are calculated based on the multi-scale coding features and the fusion features Fusion. And a first predicted image Preditct.Hr_1 is decoded based on the multi-scale decoding features.

[0113] In the embodiments of the present specification, the multi-scale coding features encoded by the encoder are introduced into the decoding process of the decoder, which makes up for the information loss in the encoding process, realizes the joint adjustment of the model parameters of the encoder and the decoder, solves the image distortion problem caused by information loss, and further improves the training effect of the image coding and decoding model.

[0114] In an optional embodiment of the present specification, the decoder of the image coding and decoding model calculates multi-scale decoding features based on the multi-scale coding features and the fusion features, and decodes a first predicted image based on the multi-scale decoding features, including the following specific steps:

[0115] The first decoding unit calculates the decoding features of the first scale based on the coding features of the first scale and the fusion features;

[0116] For any decoding unit other than the first decoding unit, the decoding features of the corresponding scale of any decoding unit are calculated based on the coding features of the corresponding scale of any decoding unit and the decoding features output by the previous decoding unit;

[0117] Until the decoding features of the target scale are calculated, the first predicted image is decoded based on the decoding features of the target scale.

[0118] The scales of the coding units in the encoder are sequentially decreased in order, and correspondingly, the scales of the decoding units in the decoder are sequentially increased in order.

[0119] The first decoding unit is the first decoding unit in the decoder, and is the decoding unit with the smallest scale among the multiple decoding units of different scales, and the first decoding unit directly performs decoding feature calculation on the fusion features.

[0120] Any decoding unit other than the first decoding unit is a decoding unit of other orders, and is a decoding unit of other scales among multiple decoding units of different scales except the smallest scale. Decoding feature calculation is not performed directly on the fused features.

[0121] The coding feature of the first scale is the coding feature output by the coding unit of the first scale, the coding feature output by the coding unit with the smallest scale among multiple coding units of different scales, and the coding feature output by the last coding unit in the encoder.

[0122] The decoding features of the first scale are the decoding features output by the first decoding unit.

[0123] The coding feature corresponding to any decoding unit at any scale is the coding feature output by the coding unit at the same scale as any decoding unit, and is the coding feature output by the coding units at other scales other than the smallest scale among multiple coding units at different scales.

[0124] The target scale is the largest among multiple different scales, which is the original scale of the image. The decoding features of the target scale are the decoding features of the original scale of the image, which are the pixel space features.

[0125] In the embodiments of this specification, the multi-scale coding features of the first sample image Lr_1 are obtained. Correspondingly, the multi-scale decoding unit in the decoder calculates the decoding features at the corresponding scale. Furthermore, the input of the decoding unit at the i-th scale includes the decoding features at the (i-1)-th scale. and the coding features output by the coding unit at the corresponding scale Therefore, the embodiments in this specification can be specifically represented by Formula 1:

[0126]

[0127]

[0128] in, The first-scale decoding features are represented by Net(), which performs computational processing, and F represents the fusion feature. The first-scale encoding feature, Let be the decoding feature corresponding to the scale of any decoding unit. The decoding feature output by the previous decoding unit. Let be the encoded feature corresponding to any decoding unit at that scale, and θ be the learnable model parameters. and The features are first fused, and then the corresponding decoding features are calculated.

[0129] Exemplarily, the first decoding unit is utilized to calculate a first scale decoding feature based on a first scale encoding feature and a fusion feature Fusion. Exemplarily, the first decoding unit is utilized to calculate a first scale decoding feature based on a first scale encoding feature and a fusion feature Fusion. For any decoding unit other than the first decoding unit, the corresponding scale decoding feature of the any decoding unit is calculated based on the corresponding scale encoding feature of the any decoding unit and the decoding feature output by the previous decoding unit. For any decoding unit other than the first decoding unit, the corresponding scale decoding feature of the any decoding unit is calculated based on the corresponding scale encoding feature of the any decoding unit and the decoding feature output by the previous decoding unit. For any decoding unit other than the first decoding unit, the corresponding scale decoding feature of the any decoding unit is calculated based on the corresponding scale encoding feature of the any decoding unit and the decoding feature output by the previous decoding unit. For any decoding unit other than the first decoding unit, the corresponding scale decoding feature of the any decoding unit is calculated based on the corresponding scale encoding feature of the any decoding unit and the decoding feature output by the previous decoding unit. Based on the seventh scale decoding feature, a first prediction image Preditct.Hr_1 is decoded.

[0130] In the embodiments of the present specification, the multi-scale encoding features encoded by the encoder are introduced into the decoding process of the decoder scale by scale, which makes up for the information loss at each scale in the encoding process, realizes the joint adjustment of the model parameters of the encoder and the decoder, solves the image distortion problem caused by information loss, and further improves the training effect of the image encoding and decoding model.

[0131] In an optional embodiment of the present specification, the sample image features of the first sample image and the label image features of the first label image are fused to obtain fusion features, including the following specific steps:

[0132] The sample image features of the first sample image and the label image features of the first label image are pre-fused to obtain fusion features.

[0133] The pre-fusion processing is a way of directly fusing multiple hidden space features in the hidden space, including space fusion, channel fusion, time domain fusion, etc. The specific fusion processing includes but is not limited to: Concat processing and Add processing.

[0134] Exemplarily, the sample image features Z of the first sample image and the label image features Z' of the first label image are subjected to Concat processing, input into a convolution layer, and the fusion features Fusion=Conv(Concat(Z,Z')) are obtained.

[0135] In the embodiments of the present specification, the fusion features are obtained by pre-fusion processing, which improves the accuracy of the fusion features and further improves the training effect of the image encoding and decoding model.

[0136] In an optional embodiment of the present specification, the model parameters of the image encoding and decoding model are adjusted based on the first prediction image and the first label image, including the following specific steps:

[0137] Based on the first prediction image and the first label image, a loss value is calculated.

[0138] Based on the loss value, the model parameters of the encoder and the decoder are jointly adjusted.

[0139] The loss value is a value for measuring the difference between the predicted image and the label image, and is used to evaluate the model performance of the image coding model, including but not limited to: cross-entropy loss, mean square error loss, mean error loss and logarithmic loss.

[0140] Based on the loss value, the model parameters of the encoder and the decoder are jointly adjusted, and the specific manner is: based on the loss value, the model parameters of the encoder and the decoder are jointly adjusted by using gradient descent method, wherein the gradient descent method is a commonly used model parameter adjustment algorithm, which is used to solve the minimum value of the loss value and realize the optimization path of the model performance. The idea of gradient descent method is: starting from the current loss value, iterating in the opposite direction of the gradient to find the minimum point of the loss value function. The iteration process of the gradient descent method can be represented as: X(k+1)=X(k)-α*g(X(k)), wherein X(k) represents the current loss value, α represents the learning rate, and g() represents the opposite direction of the gradient.

[0141] Exemplarily, based on the first predicted image and the first label image, the loss value Loss is calculated, based on the loss value, the model parameters of the encoder Encoder_A and the decoder Decoder in the VAEGAN model are jointly adjusted by using the gradient descent method, until the input and output of all 4000000 sample pairs are completed, and the trained VAEGAN model is obtained.

[0142] In the embodiments of the present specification, based on the first predicted image and the first label image, the loss value is calculated, and then the model parameters of the encoder and the decoder are jointly adjusted by using the gradient descent method, which improves the efficiency of model training, reduces the complexity of model training, and further improves the training effect.

[0143] In an optional embodiment of the present specification, before step 106, the following specific steps are further included:

[0144] Obtaining a second sample set, wherein the second sample set includes a plurality of sample pairs, and each sample pair includes a sample image and a label image;

[0145] Extracting a second sample pair from the second sample set, wherein the second sample pair includes a second sample image and a second label image, and the second sample pair is any sample pair in the second sample set;

[0146] Inputting the second sample image into the image processing model to obtain a second predicted image;

[0147] Based on the second prediction image and the second label image, the model parameters of the image processing model are adjusted until a training stop condition is reached, and a trained image processing model is obtained.

[0148] Because the training of the image processing model and the image coding and decoding model is decoupled, the image processing model can be trained separately, and the essence of inference prediction of the image processing model is forward noise addition and reverse noise removal, so the image processing model can be directly trained by sample images and label images.

[0149] The second sample set is a set of training samples for training the image processing model, and the second sample set includes a plurality of sample pairs. Any sample pair includes a low-quality sample image and a high-quality label image. For example, the second sample set includes 5000 sample pairs, and the 5000 sample pairs include 5000 face photos with face occlusion and 5000 face photos without face occlusion. The second sample set can be obtained from a sample database, such as an open source sample database, or can be artificially constructed, such as being generated by an image processing model representing an image image processing model, or can be obtained from a historical database, such as obtaining a visual image before processing and a visual image after processing from a historical database, without limitation.

[0150] The sample image is a training sample image for training the image processing model, which is a visual image to be processed, and is a low-quality visual image. The label image is a training label image for training the image processing model, which is a visual image for reference by the image processing, and is a high-quality visual image. The prediction image is an image output by the image processing model during training of the image processing model, which is a visual image processed by the image processing.

[0151] The training stop condition is a pre-set judgment condition for stopping training, including but not limited to a pre-set iteration number, a pre-set loss value threshold, a pre-set training time, and a pre-set model convergence condition.

[0152] The second sample image is input into the image processing model to obtain a second prediction image. Specifically, the second sample image is input into the image processing model, the image processing is performed based on the second sample image, and the second prediction image is obtained.

[0153] Based on the second prediction image and the second label image, the model parameters of the image processing model are adjusted. Specifically, based on the second prediction image and the second label image, a loss value is calculated, and based on the loss value, the model parameters of the image processing model are adjusted. The loss value is a difference value between the prediction image and the label image, and is used to evaluate the model performance of the image processing model, including but not limited to cross-entropy loss, mean square error loss, average error loss, and logarithmic loss.

[0154] Exemplarily, a sample set from an open source sample database is acquired, the sample set including images of 200,000 different faces, face occlusion is added to the 200,000 different face images, and a second sample set is constructed, the second sample set including 200,000 sample pairs, each sample pair including a sample image and a label image. A second sample pair is extracted from the 200,000 sample pairs for training of the current diffusion model Stable Diffusion, the second sample pair including a second sample image Lr_2 with face occlusion and a second label image Hr_2 without face occlusion, the second sample image is input into the diffusion model, inference prediction is performed based on the second sample image, a second predicted image Preditct.Hr_2 is obtained, based on the second predicted image and the second label image, a loss value Loss is calculated, based on the loss value, the model parameters of the diffusion model are adjusted, until the input and output of all 200,000 sample pairs are completed, and the trained diffusion model Stable Diffusion is obtained.

[0155] In an embodiment of the present specification, through supervised training of the image processing model, independent of the training of the image coding and decoding model, the image processing function of the image processing model is improved, and the image processing effect is improved.

[0156] In an alternative embodiment of the present specification, step 102 includes the following specific steps:

[0157] Receiving a to-be-processed video sent by a front-end user;

[0158] Extracting a to-be-processed video frame from the to-be-processed video;

[0159] Correspondingly, after step 108, the following specific steps are further included:

[0160] Synthesizing a target video based on the processed target video frame;

[0161] Feeding back the target video to the front-end user.

[0162] At present, by transcoding a low-quality video into a high-quality video, including video processing enhancement pre-processing and video compression encoding, and improving the video quality mainly relies on pre-video processing enhancement pre-processing, which is implemented by the embodiments of the present specification, effectively improving the video quality of the transcoded video and improving the user experience of the front-end user.

[0163] The to-be-processed video is a video with damaged image quality or missing information, which is a low-quality video. The to-be-processed video is caused by problems in video acquisition, transmission, storage and other processes, or problems in video editing, synthesis and other processes. For example, a video with blurred image quality collected by a camera due to weather reasons. The to-be-processed video frame is a video frame with damaged image quality or missing information in the to-be-processed video, which is a low-quality visual image.

[0164] The target video is a video with damaged image quality or missing information, which is a high-quality video. The target video frame is a video frame processed by image processing, which is obtained by processing steps 104 to 108, which is a high-quality visual image.

[0165] Exemplarily, the receiving front-end user sends a to-be-processed video video.mp4 (IPBIPPBIPBPB…) on the client of the application program, extracts 120 to-be-processed video frames I, P, B, I… from the to-be-processed video, processes the 120 to-be-processed video frames through steps 104 to 108, obtains 120 processed target video frames, synthesizes a target video video(restored).mp4 based on the processed target video frames, and feeds back the target video to the front-end user.

[0166] In the embodiments of the present specification, the to-be-processed video sent by the front-end user is processed by steps 104 to 108 in an interactive manner, which effectively improves the video quality and improves the user experience.

[0167] In an optional embodiment of the present specification, after feeding back the target video to the front-end user, the following specific steps are further included:

[0168] Receiving adjustment information sent by the front-end user for the target video;

[0169] Adjusting the model parameters of the image coding model based on the adjustment information.

[0170] The adjustment information is feedback information of the front-end user generated and sent for the target video, including but not limited to: missing information and damaged image quality.

[0171] Based on the adjustment information, the model parameters of the image coding model are adjusted. The specific manner is: based on the adjustment information, the sample image and the label image are updated, and based on the updated sample image and the label image, the model parameters of the image coding model are adjusted. Wherein, based on the updated sample image and the label image, the model parameters of the image coding model are adjusted. The specific manner is: based on the updated sample image and the label image, positive and negative samples are determined, and the model parameters of the image coding model are adjusted by using the positive and negative samples.

[0172] Exemplarily, the feedback information generated and sent by the receiving front-end user for the target video video (restored).mp4 still has information loss in the first frame and quality damage in the third frame, based on the adjustment information, the sample image and the label image are updated, based on the updated sample image and label image, the positive and negative samples are determined, and the model parameters of the VAEGAN model are adjusted by using the positive and negative samples.

[0173] In the embodiments of the present specification, the further adjustment of the image coding and decoding model parameters is completed through the interactive mode, and the subsequent video processing effect is targetedly improved.

[0174] Figure 3 A training schematic diagram of an image coding and decoding model in an image processing scheme provided by one embodiment of the present specification is shown, as shown in Figure 3

[0175] The sample image and the label image are input into the encoder of the image coding and decoding model, the sample image features corresponding to the input sample image and the label image features corresponding to the input label image are obtained, the feature fusion is performed by using the feature fusion module based on the sample image features and the label image features, the fusion features are obtained, the fusion features are input into the encoder of the image coding and decoding model, and the multi-scale features of the sample image obtained in the encoding process are introduced into the decoding process of the decoder of the image coding and decoding model by using the feature fusion module, the decoding features of the target scale are calculated, the prediction image is decoded based on the decoding features of the target scale, and the parameters of the encoder and the decoder of the image coding and decoding model are adjusted based on the prediction image and the label image.

[0176] Referring to Figure 4 , Figure 4 A flowchart of an image repairing method provided by one embodiment of the present specification is shown, including the following specific steps:

[0177] Step 402: obtaining a to-be-repaired image.

[0178] Step 404: encoding the to-be-repaired image by using the image coding and decoding model to obtain first image features.

[0179] Step 406: repairing the first image features by using the image repairing model to obtain second image features.

[0180] Step 408: decoding the second image features by using the image coding and decoding model to obtain a repaired target image, wherein the image coding and decoding model is trained based on a prediction image and a label image, the prediction image is obtained by decoding the fusion features by using the image coding and decoding model, and the fusion features are obtained by fusing the sample image features of the sample image and the label image features of the label image.​

[0181] The embodiment of the present specification is applied to a client or a server of an application program, a website or a mini program with an image inpainting function, and the client or the server is deployed with an image coding and decoding model and an image processing model to realize the image inpainting function.

[0182] The embodiment of the present specification is the same as the above Figure 1 The embodiment of the present specification is the same as the above Figure 1 The embodiment of the present specification is the same as the above

[0183] In the embodiment of the present specification, the image to be repaired is obtained; the image to be repaired is encoded by using the image coding and decoding model to obtain first image features; the first image features are inpainted by using the image inpainting model to obtain second image features; and the second image features are decoded by using the image coding and decoding model to obtain the target image after inpainting, wherein the image coding and decoding model is trained based on a prediction image and a label image, the prediction image is obtained by decoding the fusion features by using the image coding and decoding model, and the fusion features are obtained by fusing sample image features of a sample image and label image features of a label image. The fusion features obtained by fusing the sample image features of the sample image and the label image features of the label image by using the image coding and decoding model are decoded to pre-train the image coding and decoding model, the label image features of the label image are directly fused as the image features after image inpainting, the image inpainting model is avoided to participate in the pre-training of the image coding and decoding model, the image inpainting model and the image coding and decoding model are decoupled in training, the training effect and the image inpainting effect are effectively improved, the selection of the image inpainting model has higher flexibility, and the image inpainting has higher generalization.

[0184] The embodiment of the present specification is the same as the above Figure 5 , Figure 5 A flowchart of another image processing method provided by an embodiment of the present specification is shown, which is applied to a cloud-side device and includes the following specific steps:

[0185] Step 502: receiving an image to be processed sent by an end-side device.

[0186] Step 504: encoding the image to be processed by using the image coding and decoding model to obtain first image features.

[0187] Step 506: processing the first image features by using the image processing model to obtain second image features.

[0188] Step 508: decoding the second image feature by using the image coding and decoding model to obtain a target image, wherein the image coding and decoding model is trained based on a prediction image and a label image, the prediction image is obtained by decoding the fusion feature by using the image coding and decoding model, and the fusion feature is obtained by fusing the sample image feature of the sample image and the label image feature of the label image.

[0189] Step 510: feeding back the target image to the terminal side device.

[0190] The embodiment of the present specification is applied to a network cloud device where a server of a webpage, an application program or a mini program with an image processing function, which is a virtual device, and the cloud side device is deployed with an image coding and decoding model and an image processing model with an image processing function. The terminal side device is a terminal where a client of a webpage, an application program or a mini program with an image processing function logged in by a user, which is an entity device. The cloud side device and the terminal side device are connected through a network transmission channel for data transmission. The computing power and storage performance of the cloud side device are higher than those of the terminal side device.

[0191] The embodiment of the present specification and the above Figure 1 The embodiment of the present specification is based on the same inventive concept, and the specific manners of steps 504 to 508 are described in the above Figure 1 The content of steps 104 to 108 in the embodiment of the present specification.

[0192] In the embodiment of the present specification, an image processing method is provided, which is applied to a cloud side device and includes: receiving a to-be-processed image sent by a terminal side device; encoding the to-be-processed image by using an image coding and decoding model to obtain a first image feature; processing the first image feature by using an image processing model to obtain a second image feature; decoding the second image feature by using the image coding and decoding model to obtain a target image, wherein the image coding and decoding model is trained based on a prediction image and a label image, the prediction image is obtained by decoding a fusion feature by using the image coding and decoding model, and the fusion feature is obtained by fusing a sample image feature of a sample image and a label image feature of a label image; and feeding back the target image to the terminal side device. The fusion feature obtained by fusing the sample image feature of the sample image and the label image feature of the label image by using the image coding and decoding model is decoded to pre-train the image coding and decoding model, the label image feature of the label image is directly fused as the image feature after image processing, the introduction of the image processing model into the pre-training of the image coding and decoding model is avoided, the decoupling of the image processing model and the image coding and decoding model in training is realized, the training effect and the image processing effect are effectively improved, the selection of the image processing model has higher flexibility, the image processing has higher generalization, and the high computing performance and high storage performance of the cloud side device improve the efficiency and stability of the image processing.

[0193] In an optional embodiment of the present specification, after step 510, the following specific steps are further included:

[0194] receiving the adjustment information for the target image sent by the terminal-side device;

[0195] adjusting the model parameters of the image coding model based on the adjustment information.

[0196] The embodiments of the present specification and Figure 1 The embodiments of the present specification are based on the same inventive concept, and the specific manners of the embodiments of the present specification are consistent with the scheme of further adjusting the image coding model parameters through interaction as described above, which will not be repeated here.

[0197] In the embodiments of the present specification, the further adjustment of the image coding model parameters is completed through interaction, and the subsequent image processing effect is targetedly improved.

[0198] Referring to Figure 6 , Figure 6 A flowchart of a data processing method for image processing provided by an embodiment of the present specification is shown, which is applied to a cloud-side device and includes the following specific steps:

[0199] Step 602: obtaining a sample set, wherein the sample set includes a plurality of sample pairs, and each sample pair includes a sample image and a label image.

[0200] Step 604: extracting a first sample pair from the sample set, wherein the first sample pair includes a first sample image and a first label image, and the first sample pair is any sample pair in the sample set.

[0201] Step 606: inputting the first sample image and the first label image into an encoder of an image coding model respectively, and encoding to obtain a sample image feature of the first sample image and a label image feature of the first label image.

[0202] Step 608: performing feature fusion on the sample image feature of the first sample image and the label image feature of the first label image to obtain a fusion feature.

[0203] Step 610: inputting the fusion feature into a decoder of the image coding model, and decoding to obtain a first predicted image.

[0204] Step 612: adjusting the model parameters of the image coding model based on the first predicted image and the first label image until a training stop condition is reached, and obtaining a trained image coding model.

[0205] Step 614: feeding back the model parameters of the image coding model to the terminal-side device.

[0206] The embodiment of the present specification is applied to a network cloud device with a model training function, which is a kind of virtual device. The terminal side device is the terminal where the client of the webpage, application or applet with the model training function is logged in by the user, which is a kind of physical device. The cloud side device and the terminal side device are connected through a network transmission channel to perform data transmission. The computing power and storage performance of the cloud side device are higher than those of the terminal side device.

[0207] The embodiment of the present specification is the same as the above Figure 1 The embodiment of the present specification is the same as the above Figure 1 The embodiment of the present specification is the same as the above

[0208] In the embodiment of the present specification, a data processing method for image processing is provided, which is applied to a cloud side device and includes: obtaining a sample set, wherein the sample set includes a plurality of sample pairs, and each sample pair includes a sample image and a label image; extracting a first sample pair from the sample set, wherein the first sample pair includes a first sample image and a first label image, and the first sample pair is any sample pair in the sample set; inputting the first sample image and the first label image into an encoder of an image encoding and decoding model respectively to obtain sample image features of the first sample image and label image features of the first label image by encoding; performing feature fusion on the sample image features of the first sample image and the label image features of the first label image to obtain fused features; inputting the fused features into a decoder of the image encoding and decoding model to obtain a first predicted image by decoding; adjusting model parameters of the image encoding and decoding model based on the first predicted image and the first label image until a training stop condition is reached to obtain a trained image encoding and decoding model; and feeding back the model parameters of the image encoding and decoding model to a terminal side device. The decoder of the image encoding and decoding model is used to decode the fused features obtained by fusing the sample image features of the sample image and the label image features of the label image, to pre-train the image encoding and decoding model. The label image features of the label image are directly fused as the hidden space features after image processing, which avoids introducing the hidden space features predicted by the image processing model into the pre-training of the image encoding and decoding model, realizes the decoupling of the image processing model and the image encoding and decoding model in training, effectively improves the training effect and image processing effect, and at the same time, makes the selection of the image processing model more flexible, the image processing more general, and the efficiency and training effect of model training improved by using the high computing performance and high storage performance of the cloud side device.

[0209] The following describes the embodiment of the present specification in combination with the accompanying Figure 7 The image processing method provided in the present specification is further described by taking the application of the image processing method in video quality improvement as an example. Wherein, Figure 7A processing process flowchart of an image processing method applied to video quality improvement provided by an embodiment of the present specification is shown, including the following specific steps:

[0210] Step 702: Obtain a sample set, wherein the sample set includes 800,000 sample pairs, and each sample pair includes an SD sample image and an HD label image.

[0211] Step 704: Extract a first sample pair from the sample set, wherein the first sample pair includes a first SD sample image and a first HD label image, and the first sample pair is any sample pair in the sample set.

[0212] Step 706: Input the first SD sample image and the first HD label image into the encoder of the VAEGAN model respectively, and encode to obtain the latent space feature of the first SD sample image, the latent space feature of the first HD label image, and the multi-scale coding feature of the first SD sample image.

[0213] Step 708: Perform pre-fusion processing on the latent space feature of the first SD sample image and the latent space feature of the first HD label image to obtain a fusion feature.

[0214] Step 710: Use the decoder of the VAEGAN model to calculate a multi-scale decoding feature based on the multi-scale coding feature and the fusion feature, and decode to obtain a first HD prediction image based on the multi-scale decoding feature.

[0215] Step 712: Calculate a loss value based on the first HD prediction image and the first HD label image, and based on the loss value, jointly adjust the model parameters of the encoder and the decoder using the gradient descent method until a training stop condition is reached, and obtain a trained VAEGAN model.

[0216] Step 714: Obtain an SD movie video with to-be-improved quality sent by a front-end user, wherein the SD movie video includes 260,000 SD video frames.

[0217] Step 716: Input the SD video frame into the encoder of the VAEGAN model to encode and obtain a first latent space feature.

[0218] Step 718: Input the first latent space feature into the diffusion model to predict a second latent space feature.

[0219] Step 720: Input the second latent space feature into the decoder of the VAEGAN model to decode and obtain a transcoded HD video frame.

[0220] Step 722: Synthesize an HD movie video based on the 260,000 transcoded HD video frames, and feed back the HD movie video to the front-end user.

[0221] In the embodiments of the present specification, the multi-scale coding features coded by the encoder are introduced into the decoding process of the decoder, making up for the information loss in the coding process, realizing the joint adjustment of the model parameters of the encoder and the decoder, solving the image distortion problem caused by information loss, and improving the training effect of the VAEGAN model. The decoder of the VAEGAN model is used to decode the fusion features obtained by fusing the hidden space features of the sample image and the hidden space features of the label image, to pre-train the VAEGAN model, directly fusing the hidden space features of the label image as the hidden space features after image processing, avoiding the introduction of the hidden space features predicted by the diffusion model into the pre-training of the VAEGAN model, realizing the decoupling of the diffusion model and the VAEGAN model in training, effectively improving the training effect and the image processing effect, and at the same time, making the selection of the diffusion model have higher flexibility, and the image processing have higher generalization.

[0222] Corresponding to the method embodiments described above, the present specification also provides image processing device embodiments, Figure 8 The structure of an image processing device provided by an embodiment of the present specification is shown in a structural schematic diagram. As shown in the figure, Figure 8 The device comprises:

[0223] The first acquisition module 802 is configured to acquire a to-be-processed image;

[0224] The first encoding module 804 is configured to code the to-be-processed image by using an image coding and decoding model, to obtain first image features;

[0225] The first prediction module 806 is configured to perform image feature processing on the first image features by using an image processing model, to obtain second image features;

[0226] The first decoding module 808 is configured to decode the second image features by using the image coding and decoding model, to obtain a target image, wherein the image coding and decoding model is trained based on a predicted image and a label image, the predicted image is obtained by decoding the fusion features by using the image coding and decoding model, and the fusion features are obtained by fusing sample image features of a sample image and label image features of a label image.

[0227] Optionally, the image coding and decoding model comprises an encoder and a decoder.

[0228] Correspondingly, the first encoding module 804 is further configured to:

[0229] input the to-be-processed image into the encoder, to obtain the first image features by coding;

[0230] Correspondingly, the first decoding module 808 is further configured to:

[0231] The second image feature is input into the decoder to obtain the target image.

[0232] Optionally, the apparatus further comprises:

[0233] The first training module is configured to obtain a first sample set, wherein the first sample set comprises a plurality of sample pairs, and each sample pair comprises a sample image and a label image; extract a first sample pair from the first sample set, wherein the first sample pair comprises a first sample image and a first label image, and the first sample pair is any sample pair in the sample set; input the first sample image and the first label image into an encoder of an image coding and decoding model respectively to obtain a sample image feature of the first sample image and a label image feature of the first label image; perform feature fusion on the sample image feature of the first sample image and the label image feature of the first label image to obtain a fused feature; input the fused feature into a decoder of the image coding and decoding model to obtain a first predicted image; and adjust model parameters of the image coding and decoding model based on the first predicted image and the first label image until a training stop condition is reached to obtain a trained image coding and decoding model.

[0234] Optionally, the first training module is further configured to

[0235] The first training module is configured to obtain a first sample set, wherein the first sample set comprises a plurality of sample pairs, and each sample pair comprises a sample image and a label image; extract a first sample pair from the first sample set, wherein the first sample pair comprises a first sample image and a first label image, and the first sample pair is any sample pair in the sample set; input the first sample image and the first label image into an encoder of an image coding and decoding model respectively to obtain a sample image feature of the first sample image and a label image feature of the first label image; perform feature fusion on the sample image feature of the first sample image and the label image feature of the first label image to obtain a fused feature; input the fused feature into a decoder of the image coding and decoding model to obtain a first predicted image; and adjust model parameters of the image coding and decoding model based on the first predicted image and the first label image until a training stop condition is reached to obtain a trained image coding and decoding model.

[0236] Optionally, the first training module is further configured to

[0237] The first training module is configured to obtain a first sample set, wherein the first sample set comprises a plurality of sample pairs, and each sample pair comprises a sample image and a label image; extract a first sample pair from the first sample set, wherein the first sample pair comprises a first sample image and a first label image, and the first sample pair is any sample pair in the sample set; input the first sample image and the first label image into an encoder of an image coding and decoding model respectively to obtain a sample image feature of the first sample image and a label image feature of the first label image; perform feature fusion on the sample image feature of the first sample image and the label image feature of the first label image to obtain a fused feature; input the fused feature into a decoder of the image coding and decoding model to obtain a first predicted image; and adjust model parameters of the image coding and decoding model based on the first predicted image and the first label image until a training stop condition is reached to obtain a trained image coding and decoding model.

[0238] Optionally, the first training module is further configured to

[0239] The first training module is configured to obtain a first sample set, wherein the first sample set comprises a plurality of sample pairs, and each sample pair comprises a sample image and a label image; extract a first sample pair from the first sample set, wherein the first sample pair comprises a first sample image and a first label image, and the first sample pair is any sample pair in the sample set; input the first sample image and the first label image into an encoder of an image coding and decoding model respectively to obtain a sample image feature of the first sample image and a label image feature of the first label image; perform feature fusion on the sample image feature of the first sample image and the label image feature of the first label image to obtain a fused feature; input the fused feature into a decoder of the image coding and decoding model to obtain a first predicted image; and adjust model parameters of the image coding and decoding model based on the first predicted image and the first label image until a training stop condition is reached to obtain a trained image coding and decoding model.

[0240] Optionally, the first training module is further configured to

[0241] The first training module is configured to obtain a first sample set, wherein the first sample set comprises a plurality of sample pairs, and each sample pair comprises a sample image and a label image; extract a first sample pair from the first sample set, wherein the first sample pair comprises a first sample image and a first label image, and the first sample pair is any sample pair in the sample set; input the first sample image and the first label image into an encoder of an image coding and decoding model respectively to obtain a sample image feature of the first sample image and a label image feature of the first label image; perform feature fusion on the sample image feature of the first sample image and the label image feature of the first label image to obtain a fused feature; input the fused feature into a decoder of the image coding and decoding model to obtain a first predicted image; and adjust model parameters of the image coding and decoding model based on the first predicted image and the first label image until a training stop condition is reached to obtain a trained image coding and decoding model.

[0242] Optionally, the apparatus further comprises:

[0243] The second training module is configured to obtain a second sample set, wherein the second sample set comprises a plurality of sample pairs, and each sample pair comprises a sample image and a label image; extract a second sample pair from the second sample set, wherein the second sample pair comprises a second sample image and a second label image, and the second sample pair is any sample pair in the second sample set; input the second sample image into the image processing model to obtain a second predicted image; and adjust the model parameters of the image processing model based on the second predicted image and the second label image until a training stop condition is reached, thereby obtaining a trained image processing model.

[0244] Optionally, the first obtaining module 802 is further configured to:

[0245] receive a to-be-processed video sent by a front-end user; and extract a to-be-processed video frame from the to-be-processed video;

[0246] Correspondingly, the apparatus further comprises:

[0247] The first feedback module is configured to synthesize a target video based on the processed target video frame, and feed back the target video to the front-end user.

[0248] Optionally, the apparatus further comprises:

[0249] The first adjustment module is configured to receive adjustment information sent by the front-end user for the target video, and adjust the model parameters of the image coding model based on the adjustment information.

[0250] In the embodiments of the present specification, the image coding model is pre-trained by decoding the fusion features obtained by fusing the sample image features of the sample images and the label image features of the label images, the label image features of the label images are directly fused as the image features after image processing, the introduction of the image processing model into the pre-training of the image coding model is avoided, the decoupling of the image processing model and the image coding model in training is achieved, the training effect and the image processing effect are effectively improved, and the selection of the image processing model has higher flexibility and the image processing has higher generalization.

[0251] The above is a schematic scheme of the image processing apparatus of the present embodiment. It should be noted that the technical scheme of the image processing apparatus belongs to the same concept as the technical scheme of the image processing method described above, and the details of the technical scheme of the image processing apparatus that are not described in detail can be referred to the description of the technical scheme of the image processing method.

[0252] Corresponding to the method embodiments described above, the present specification also provides image processing apparatus embodiments, Figure 9A structural diagram of an image repairing device provided by an embodiment of the present specification is shown. As shown in Figure 9 includes:

[0253] The second acquisition module 902 is configured to acquire a to-be-repaired image.

[0254] The second encoding module 904 is configured to encode the to-be-repaired image by using an image encoding and decoding model to obtain first image features.

[0255] The second prediction module 906 is configured to perform image feature repairing on the first image features by using an image repairing model to obtain second image features.

[0256] The second decoding module 908 is configured to decode the second image features by using the image encoding and decoding model to obtain a repaired target image, wherein the image encoding and decoding model is obtained by training based on a predicted image and a label image, the predicted image is obtained by decoding the fusion features by using the image encoding and decoding model, and the fusion features are obtained by fusing sample image features of a sample image and label image features of the label image.

[0257] In the embodiment of the present specification, the image encoding and decoding model is pre-trained by decoding the fusion features obtained by fusing the sample image features of the sample image and the label image features of the label image, the label image features of the label image are directly fused as the image features after image repairing, the image repairing model is avoided to participate in the pre-training of the image encoding and decoding model, the decoupling of the image repairing model and the image encoding and decoding model in training is realized, the training effect and the image repairing effect are effectively improved, meanwhile, the selection of the image repairing model has higher flexibility, and the image repairing has higher generalization.

[0258] The above is a schematic scheme of the image repairing device of the present embodiment. It should be noted that the technical scheme of the image repairing device belongs to the same concept as the technical scheme of the image repairing method described above, and the details of the technical scheme of the image repairing device that are not described in detail can be referred to the description of the technical scheme of the image repairing method.

[0259] Corresponding to the above method embodiment, the present specification also provides an image processing device embodiment, Figure 10 A structural diagram of another image processing device provided by an embodiment of the present specification is shown. As shown in Figure 10 The device is applied to a cloud-side device and includes:

[0260] The receiving module 1002 is configured to receive a to-be-processed image sent by an end-side device.

[0261] The third encoding module 1004 is configured to encode the to-be-processed image by using the image coding and decoding model to obtain first image features.

[0262] The third prediction module 1006 is configured to perform image feature processing on the first image features by using the image processing model to obtain second image features.

[0263] The third decoding module 1008 is configured to decode the second image features by using the image coding and decoding model to obtain a target image, wherein the image coding and decoding model is obtained by training based on a prediction image and a label image, the prediction image is obtained by decoding the fusion features by using the image coding and decoding model, and the fusion features are obtained by fusing sample image features of a sample image and label image features of a label image.

[0264] The third feedback module 1010 is configured to feed back the target image to the terminal-side device.

[0265] Optionally, the apparatus further includes:

[0266] The third adjustment module is configured to receive adjustment information for the target image sent by the terminal-side device, and adjust model parameters of the image coding and decoding model based on the adjustment information.

[0267] In the embodiments of the present specification, the image coding and decoding model is pre-trained by decoding the fusion features obtained by fusing the sample image features of the sample image and the label image features of the label image by using the image coding and decoding model, the label image features of the label image are directly fused as image features after image processing, the introduction of the image processing model into the pre-training of the image coding and decoding model is avoided, the decoupling of the image processing model and the image coding and decoding model in training is achieved, the training effect and the image processing effect are effectively improved, meanwhile, the selection of the image processing model has higher flexibility, the image processing has higher generalization, and the high computing performance and high storage performance of the cloud-side device are utilized to improve the efficiency and stability of the image processing.

[0268] The above is a schematic scheme of another image processing apparatus of the present embodiment. It should be noted that the technical scheme of the image processing apparatus belongs to the same concept as the technical scheme of the image processing method described above, and the details of the technical scheme of the image processing apparatus that are not described in detail can be referred to the description of the technical scheme of the image processing method.

[0269] Corresponding to the method embodiments described above, the present specification also provides an image processing data processing apparatus embodiment, Figure 11 Fig. 1 shows a structural schematic diagram of an image processing data processing apparatus provided by one embodiment of the present specification. As shown in the figure, Figure 11 the apparatus is applied to a cloud-side device and includes:

[0270] The fourth acquisition module 1102 is configured to acquire a sample set, wherein the sample set includes a plurality of sample pairs, and each sample pair includes a sample image and a label image;

[0271] The extraction module 1104 is configured to extract a first sample pair from the first sample set, wherein the first sample pair includes a first sample image and a first label image, and the first sample pair is any sample pair in the sample set;

[0272] The fourth encoding module 1106 is configured to input the first sample image and the first label image into an encoder of an image coding and decoding model respectively, and encode to obtain sample image features of the first sample image and label image features of the first label image;

[0273] The fusion module 1108 is configured to perform feature fusion on the sample image features of the first sample image and the label image features of the first label image to obtain fusion features;

[0274] The fourth decoding module 1110 is configured to input the fusion features into a decoder of the image coding and decoding model to decode to obtain a first predicted image;

[0275] The adjustment module 1112 is configured to adjust model parameters of the image coding and decoding model based on the first predicted image and the first label image until a training stop condition is reached to obtain a trained image coding and decoding model;

[0276] The fourth feedback module 1114 is configured to feed back the model parameters of the image coding and decoding model to an end-side device.

[0277] In the embodiments of the present specification, the decoder of the image coding and decoding model is used to decode the fusion features obtained by fusing the sample image features of the sample image and the label image features of the label image, to pre-train the image coding and decoding model. The label image features of the label image are directly fused as the hidden space features after image processing, avoiding the introduction of the hidden space features predicted by the image processing model into the pre-training of the image coding and decoding model, realizing the decoupling of the image processing model and the image coding and decoding model in training, effectively improving the training effect and the image processing effect, at the same time, making the selection of the image processing model have higher flexibility, the image processing has higher generalization, and using the high computing performance and high storage performance of the cloud-side device improves the efficiency and training effect of the model training.

[0278] The above is a schematic scheme of the data processing apparatus for image processing according to an embodiment. It should be noted that the technical scheme of the data processing apparatus for image processing is the same concept as the technical scheme of the data processing method for image processing described above, and the details of the technical scheme of the data processing apparatus for image processing not described in detail can be seen from the description of the technical scheme of the data processing method for image processing.

[0279] Figure 12 A structural block diagram of a computing device is shown according to an embodiment of the present specification. The components of the computing device 1200 include, but are not limited to, a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 through a bus 1230, and a database 1250 is used to save data.

[0280] The computing device 1200 also includes an access device 1240, which enables the computing device 1200 to communicate via one or more networks 1260. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1240 can include one or more of any type of network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, near field communication (NFC).

[0281] In an embodiment of the present specification, the above-mentioned components of the computing device 1200 and other components not shown in the Figure 12 may be connected to each other, for example, through a bus. It should be understood that Figure 12 The structural block diagram of the computing device shown is only for the purpose of example, and is not a limitation on the scope of the present specification. Those skilled in the art can add or replace other components as needed.

[0282] The computing device 1200 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1200 can also be a mobile or stationary server.

[0283] The processor 1220 is configured to execute computer-executable instructions to perform the steps of the image processing method or the data processing method of image processing.

[0284] The above is a schematic solution of the computing device of the embodiment. It should be noted that the technical solution of the computing device belongs to the same concept as the technical solutions of the image processing method and the data processing method of image processing, and details of the technical solution of the computing device that are not described in detail can be referred to the description of the technical solution of the image processing method or the data processing method of image processing.

[0285] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the image processing method or the data processing method of image processing.

[0286] The above is a schematic solution of the computer-readable storage medium of the embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solutions of the image processing method and the data processing method of image processing, and details of the technical solution of the storage medium that are not described in detail can be referred to the description of the technical solution of the image processing method or the data processing method of image processing.

[0287] An embodiment of the present specification further provides a computer program, which, when executed in a computer, causes the computer to perform the steps of the image processing method or the data processing method of image processing.

[0288] The above is a schematic solution of the computer program of the embodiment. It should be noted that the technical solution of the computer program belongs to the same concept as the technical solutions of the image processing method and the data processing method of image processing, and details of the technical solution of the computer program that are not described in detail can be referred to the description of the technical solution of the image processing method or the data processing method of image processing.

[0289] The above-described embodiments of the application have several aspects, no single one of which is solely responsible for the application's desirable attributes. Without limiting the scope of the application as expressed by the claims which follow, some further embodiments make these aspects even more useful. Other embodiments can result in less desirable attributes.

[0290] The computer readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate contents according to the requirements of patent practice, for example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0291] It should be noted that for the foregoing method embodiments, the acts described can be performed in a different order than that described, and that various interlocking and / or parallel configurations are also possible according to the present application. Furthermore, specific operational details mentioned only specify implementations for descriptive purposes and can not necessarily be exhaustive. Also, the order in which some or all of the described acts are listed-they are not necessarily intended to be limiting, since some acts can be done in different orders or concurrently. Moreover, described acts can be implemented by one or more components of a claim, and many components can benefit from being implemented by one or more of the described acts. Additionally, the scope of the application extends to numerous combinations of the various features described.

[0292] In the above-described embodiments, the description of each embodiment focuses on different aspects, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0293] The above-disclosed specification of the present application is only used to help explain the present application. Alternative embodiments do not describe all the details, nor limit the application to only the specific embodiments described. Obviously, according to the content of the embodiments of the present application, many modifications and changes can be made. The present application selects and describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present application, so that those skilled in the art can well understand and use the present application. The present application is limited by the claims and their full scope and equivalents.

Claims

1. An image processing method comprising: obtaining a to-be-processed image; encoding the to-be-processed image using an image coding model to obtain first image features; processing the first image features using an image processing model to obtain second image features; decoding the second image features using the image coding model to obtain a target image, wherein the image coding model is trained based on a predicted image and a label image, the predicted image is obtained by decoding fusion features using the image coding model, the fusion features are obtained by fusing sample image features of a sample image and label image features of the label image, and the sample image features and the label image features are obtained by respectively inputting the sample image and the label image into an encoder of the image coding model.

2. The method of claim 1, wherein the image coding model comprises an encoder and a decoder; the encoding the to-be-processed image using the image coding model to obtain the first image features comprises: inputting the to-be-processed image into the encoder to obtain the first image features; the decoding the second image features using the image coding model to obtain the target image comprises: inputting the second image features into the decoder to obtain the target image.

3. The method of claim 2, further comprising, before the inputting the to-be-processed image into the encoder to obtain the first image features: obtaining a first sample set, wherein the first sample set comprises a plurality of sample pairs, and each sample pair comprises a sample image and a label image; extracting a first sample pair from the first sample set, wherein the first sample pair comprises a first sample image and a first label image, and the first sample pair is any sample pair in the sample set; inputting the first sample image and the first label image into an encoder of the image coding model to obtain sample image features of the first sample image and label image features of the first label image; fusing the sample image features of the first sample image and the label image features of the first label image to obtain fusion features; inputting the fusion features into a decoder of the image coding model to obtain a first predicted image; adjusting model parameters of the image coding model based on the first predicted image and the first label image until a training stop condition is reached to obtain the trained image coding model.

4. The method of claim 3, further comprising, after the inputting the first sample image into the encoder of the image coding model: obtaining multi-scale encoding features of the first sample image; the inputting the fusion features into the decoder of the image coding model to obtain the first predicted image comprises: using the decoder of the image coding model to calculate multi-scale decoding features based on the multi-scale encoding features and the fusion features, and to decode the first predicted image based on the multi-scale decoding features.

5. The method of claim 4, wherein the decoder of the image coding model calculates multi-scale decoding features based on the multi-scale encoding features and the fused features, and decodes a first predicted image based on the multi-scale decoding features, comprising: calculating, by a first decoding unit, the decoding features of a first scale based on the encoding features of the first scale and the fused features; calculating, by any decoding unit other than the first decoding unit, the decoding features of a corresponding scale of the any decoding unit based on the encoding features of the corresponding scale of the any decoding unit and the decoding features output by a previous decoding unit; and calculating, until the decoding features of a target scale are calculated, the decoding features of the target scale based on the decoding features of the target scale, and decoding the first predicted image based on the decoding features of the target scale.

6. The method of claim 3, wherein the adjusting the model parameters of the image coding model based on the first predicted image and the first label image comprises: calculating a loss value based on the first predicted image and the first label image; and jointly adjusting the model parameters of the encoder and the decoder based on the loss value.

7. The method of claim 1, wherein before the processing the first image features using the image processing model to obtain second image features, the method further comprises: obtaining a second sample set, wherein the second sample set comprises a plurality of sample pairs, and each sample pair comprises a sample image and a label image; extracting a second sample pair from the second sample set, wherein the second sample pair comprises a second sample image and a second label image, and the second sample pair is any sample pair in the second sample set; inputting the second sample image into the image processing model to obtain a second predicted image; and adjusting the model parameters of the image processing model based on the second predicted image and the second label image until a training stop condition is reached, and obtaining the trained image processing model.

8. The method of claim 1, wherein the obtaining the image to be processed comprises: receiving a video to be processed sent by a front-end user; and extracting a video frame to be processed from the video to be processed; and wherein after the decoding the second image features using the image coding model to obtain a target image, the method further comprises: synthesizing a target video based on the processed target video frame; and feeding back the target video to the front-end user.

9. The method of claim 8, wherein after the feeding back the target video to the front-end user, the method further comprises: receiving adjustment information sent by the front-end user for the target video; and adjusting the model parameters of the image coding model based on the adjustment information.

10. An image inpainting method, comprising: obtaining an image to be inpainted; encoding the image to be inpainted using an image coding model to obtain first image features; and processing the first image features using an image inpainting model to obtain second image features. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ decoding, by using the image coding and decoding model, the second image feature to obtain a target image, wherein the image coding and decoding model is trained based on a prediction image and a label image, the prediction image is obtained by decoding, by using the image coding and decoding model, fused features, the fused features are obtained by fusing a sample image feature of a sample image and a label image feature of the label image, and the sample image feature and the label image feature are obtained by respectively inputting the sample image and the label image into an encoder of the image coding and decoding model.

11. An image processing method applied to a cloud-side device, comprising: receiving an image to be processed sent by an end-side device; encoding, by using an image coding and decoding model, the image to be processed to obtain a first image feature; processing, by using an image processing model, the first image feature to obtain a second image feature; decoding, by using the image coding and decoding model, the second image feature to obtain a target image, wherein the image coding and decoding model is trained based on a prediction image and a label image, the prediction image is obtained by decoding, by using the image coding and decoding model, fused features, the fused features are obtained by fusing a sample image feature of a sample image and a label image feature of the label image, and the sample image feature and the label image feature are obtained by respectively inputting the sample image and the label image into an encoder of the image coding and decoding model; feeding back the target image to the end-side device.

12. A data processing method of image processing applied to a cloud-side device, comprising: obtaining a sample set, wherein the sample set comprises a plurality of sample pairs, and each sample pair comprises a sample image and a label image; extracting a first sample pair from the sample set, wherein the first sample pair comprises a first sample image and a first label image, and the first sample pair is any sample pair in the sample set; inputting the first sample image and the first label image into an encoder of an image coding and decoding model to obtain a sample image feature of the first sample image and a label image feature of the first label image; fusing the sample image feature of the first sample image and the label image feature of the first label image to obtain fused features; inputting the fused features into a decoder of the image coding and decoding model to obtain a first prediction image; adjusting model parameters of the image coding and decoding model based on the first prediction image and the first label image until a training stop condition is reached to obtain the image coding and decoding model trained; feeding back the model parameters of the image coding and decoding model to an end-side device.

13. A computing device, comprising: a memory and a processor; the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, implement steps of the method in any one of claims 1 to 12.

14. A computer readable storage medium storing computer executable instructions which, when executed by a processor, implement the steps of the method of any one of claims 1 to 12.

Images