Image processing method, image restoration method and data processing method for image processing

Abstract

The embodiment of the invention provides an image processing method, an image restoration method and a data processing method for image processing. The image processing method comprises the following steps: acquiring a to-be-processed image; encoding the to-be-processed image by using the image encoding and decoding model to obtain a first image feature; performing image feature processing on the first image feature by using an image processing model to obtain a second image feature; and decoding the second image feature by using an image coding and decoding model to obtain a target image, the image coding and decoding model being obtained by training based on a predicted image and a tag image, the predicted image being obtained by decoding the fusion feature by using the image coding and decoding model, and the tag image being obtained by decoding the fusion feature by using the tag image. The fusion feature is obtained by fusing the sample image feature of the sample image and the label image feature of the label image. Decoupling of the image processing model and the image coding and decoding model in training is achieved, the training effect and the image processing effect are improved, and meanwhile the selection flexibility of the image processing model and the generalization of image processing are improved.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of image processing technology, and in particular to an image processing method. Background Art

[0002] With the development of neural network models, excellent results have been achieved in image processing.

[0003] At present, the image to be processed is processed by jointly using the image coding model and the image processing model to obtain the target image. However, the joint use of the image coding model and the image processing model will cause information loss in the processing process of the image to be processed, resulting in poor image processing effect. Summary of the invention

[0004] In view of this, an embodiment of the present specification provides an image processing method. One or more embodiments of the present specification also relate to an image restoration method, another image processing method, a data processing method for image processing, an image processing device, an image restoration method, another image processing device, a data processing device for image processing, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects existing in the prior art.

[0005] In an embodiment of the present specification, an image processing method is provided, comprising:

[0006] Get the image to be processed;

[0007] Encode the image to be processed by using the image encoding and decoding model to obtain a first image feature;

[0008] Using an image processing model, performing image feature processing on the first image feature to obtain a second image feature;

[0009] The second image feature is decoded by using an image encoding and decoding model to obtain a target image, wherein the image encoding and decoding model is trained based on a predicted image and a label image, the predicted image is obtained by decoding a fusion feature using the image encoding and decoding model, and the fusion feature is obtained by fusing a sample image feature of the sample image and a label image feature of the label image.

[0010] In one embodiment of the present specification, an image to be processed is obtained; an image encoding and decoding model is used to encode the image to be processed to obtain a first image feature; an image processing model is used to perform image feature processing on the first image feature to obtain a second image feature; and an image encoding and decoding model is used to decode the second image feature to obtain a target image, wherein the image encoding and decoding model is obtained by training based on a predicted image and a label image, the predicted image is obtained by decoding a fusion feature using the image encoding and decoding model, and the fusion feature is obtained by fusion of a sample image feature of a sample image and a label image feature of a label image. The image encoding and decoding model is used to decode a fusion feature obtained by fusion of the sample image feature of the sample image and the label image feature of the label image, so as to pre-train the image encoding and decoding model, directly fuse the label image feature of the label image as the image feature after image processing, avoid introducing the image processing model to participate in the pre-training of the image encoding and decoding model, realize the decoupling of the image processing model and the image encoding and decoding model in training, effectively improve the training effect and the image processing effect, and at the same time, make the selection of the image processing model more flexible and the image processing more generalized. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a flow chart of an image processing method provided by an embodiment of this specification;

[0012] Figure 2 It is a model architecture diagram of an image processing method provided in one embodiment of this specification;

[0013] Figure 3 It is a schematic diagram of training an image encoding and decoding model in an image processing solution provided by an embodiment of this specification;

[0014] Figure 4 is a flow chart of an image restoration method provided by an embodiment of this specification;

[0015] Figure 5 is a flow chart of another image processing method provided by an embodiment of this specification;

[0016] Figure 6 is a flow chart of a data processing method for image processing provided by one embodiment of this specification;

[0017] Figure 7 is a processing flow chart of an image processing method for improving video quality provided by an embodiment of this specification;

[0018] Figure 8 is a structural schematic diagram of an image processing device provided by an embodiment of this specification;

[0019] Fig. 9is a structural schematic diagram of an image restoration device provided by an embodiment of this specification;

[0020] Fig.10 is a schematic diagram of the structure of another image processing device provided by an embodiment of this specification;

[0021] Fig.11 is a structural schematic diagram of a data processing device for image processing provided by an embodiment of this specification;

[0022] Fig.12 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0023] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0024] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0025] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0026] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0027] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM), a multi-modal pre-training model, etc.

[0028] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0029] First, the terms involved in one or more embodiments of this specification are explained.

[0030] Image processing: using computer technology to calculate and process images, specifically to extract image features, perform in-depth feature calculations on image features, and achieve image processing tasks for a specific purpose, including but not limited to: image restoration, image super-resolution, and image denoising.

[0031] Image restoration: A technique that restores lost or damaged parts of an image by restoring it. It is used to repair noise, damage, missing parts, etc. in an image, making the image clearer and more complete.

[0032] Image super-resolution: A technique for converting low-resolution images into high-resolution images by performing super-resolution processing on the images.

[0033] Image denoising: A technique that reduces or removes noise in digital images by performing denoising processing on the image.

[0034] Pixel Space: A low-dimensional, concrete feature space of an image. Each pixel in the image has a corresponding feature value, for example, RGB (Red Green Blue) value.

[0035] Latent Space: A high-dimensional, abstract feature space. The key features of the image are extracted by reducing the dimensionality of the pixel space features of the image.

[0036] Diffusion Model: An inference prediction model, including a forward denoising process and a reverse denoising process, which generates images through forward and reverse inference prediction. In the forward process, Gaussian noise (white noise) is iteratively added to the latent space features of the original input. After a sufficient number of iterations, the latent space features become pure Gaussian noise; in the reverse process, pure Gaussian noise is iteratively removed. After a sufficient number of iterations, pure Gaussian noise becomes the output latent space features.

[0037] CNN (Convolutional Neural Networks): A multi-convolutional layer neural network model with forward propagation and back propagation.

[0038] VIT model (Vision Transformer): a translation model for visual image processing with a visual attention mechanism.

[0039] U-Net model (U-type neural network model): a U-shaped (decreasing encoding layer size and increasing decoding layer size) neural network model that uses splicing to fuse deep and shallow layer features.

[0040] VQGAN (Vector Quantized GenerativeAdversarial Network): An image encoding and decoding model with an encoder, a feature dictionary, and a decoder. The encoder maps the input pixel space features to the latent space, then determines the corresponding features of the latent space features from the feature dictionary, and then the decoder maps the determined corresponding features back to the pixel space.

[0041] VAEGAN (Variational Autoencoder Generative Adversarial Network): An image encoding and decoding model with an encoder and a decoder. The encoder maps the input pixel space features to the latent space, then determines the latent space features as a Gaussian distribution, and finally the decoder samples features from the Gaussian distribution and maps them back to the pixel space.

[0042] AIGC (AI Generated Content): technology for generating content based on artificial intelligence technology.

[0043] At present, by freezing model parameters and multi-stage training, for example, in the first stage, only the image processing model is trained, and in the second stage, only some newly added training parameters of the image encoding and decoding model are trained, while the model parameters of the image encoding and decoding model remain unchanged. However, during the training process, because the image processing model is still bound to the image encoding and decoding model in the overall architecture, the image processing model relies on the image encoding and decoding model to implement encoding and decoding, and no decoupling is achieved in training, it is still impossible to effectively solve the information loss problem of image encoding, and the training effect is insufficient, and the image processing effect is insufficient.

[0044] In response to the above problems, this specification provides an image processing method. This specification also involves an image restoration method, another image processing method, a data processing method for image processing, an image processing device, an image restoration device, another image processing device, a data processing device for image processing, a computing device, a computer-readable storage medium and a computer program, which are described in detail one by one in the following embodiments.

[0045] See also Figure 1 , Figure 1 A flowchart of an image processing method provided by an embodiment of the present specification is shown, comprising the following specific steps:

[0046] Step 102: Obtain an image to be processed.

[0047] The embodiments of this specification are applied to the client or server of an application, website or applet with image processing function, and the client or server is deployed with an image encoding and decoding model and an image processing model for realizing the image processing function.

[0048] The image to be processed is a visual image to be processed, which is a low-quality visual image. The image to be processed can be a real image acquired by an image acquisition device, such as a photo, a video frame, and a remote sensing image, etc., or an artificially generated image, such as a picture and an animation, etc., or a virtual image generated by an image generation algorithm, such as a picture, an animation, a photo, and a video frame, etc., which is not limited here.

[0049] Acquiring the image to be processed may be receiving the image to be processed sent by the client, for example, receiving the photo to be processed uploaded by the user through the front-end interface of the application on the mobile terminal. It may also be acquiring the image to be processed from an image database, which includes a local database and a remote database, for example, an open source database, a cloud database, etc., and acquiring the video image of area A recorded by the recording lens in the past 24 hours from the cloud database. It may also be receiving the image to be processed sent by the image acquisition device, for example, the image sensors in four directions on the car send the acquired images to the server of the application of the image stitching function in the driving computer, and the server implements the stitching of the acquired images in the four directions to obtain the image of the surrounding environment of the car, which is not limited here.

[0050] Exemplarily, a monitoring video stream of a road intersection in rainy weather that is captured and sent by a camera located at a road intersection is received, and video frames of the monitoring video stream are extracted.

[0051] Acquiring the image to be processed provides an image basis for subsequent image processing.

[0052] Step 104: Encode the image to be processed using the image encoding and decoding model to obtain a first image feature.

[0053] The image encoding and decoding model is a neural network model with image encoding and decoding functions. The image encoding and decoding model includes an encoder and a decoder. The image encoding and decoding model realizes the encoding and decoding conversion between image features and pixel features. The image encoding and decoding model includes but is not limited to: CNN model, ViT model, UNet model, GAN model, VQGAN model and VAEGAN model. The image encoding and decoding model is obtained through pre-supervised training.

[0054] The first image feature is the image coding feature of the image to be processed, which is expressed as a feature vector of a specific scale (dimension). The first image feature represents the high-dimensional, abstract key features such as the structure, shape, and texture of the image to be processed. For example, the features of the image to be processed in the latent space, that is, the first latent space feature. For example, a landscape photo is the image to be restored, and the first image feature represents the high-dimensional, abstract key features such as the structure, shape, and texture of the topography in the landscape photo.

[0055] The image to be processed is encoded by using the image encoding and decoding model to obtain the first image feature. The specific method is: the pixel space feature of the image to be processed is mapped to the latent space by using the image encoding and decoding model to obtain the first latent space feature.

[0056] Exemplarily, the pre-trained VAEGAN model is used to map the pixel space feature Pixel Feature of the video frame of the monitoring video stream to the latent space to obtain the first latent space feature Latent Feature_1.

[0057] The image to be processed is encoded using the image encoding and decoding model to obtain the first image features, which provides a feature basis for image processing by the subsequent image processing model.

[0058] Step 106: Using the image processing model, perform image feature processing on the first image feature to obtain a second image feature.

[0059] The image processing model is a neural network model that performs feature processing at the image feature level. The image processing model determines the key features of the image to be processed, such as the structure, shape, and texture, based on the first image feature of the image to be processed, and performs image processing based on this to obtain the second image feature after image processing. Including but not limited to: a generation model and an image processing model. The generation model is an image processing model that performs image processing based on predictive generation. The generation model realizes image generation through a generator, such as a generative pre-trained transformer model (GPT, Generative Pre-Trained Transformer), a GAN model, a VQGAN model, and a VAEGAN model. The diffusion model is an image processing model that performs image processing based on inference prediction. The diffusion model realizes image processing by adding noise forward and removing noise backward, such as a stable diffusion model. The image processing model determines the key features of the image to be processed, such as the structure, shape, and texture, based on the first image feature of the input image to be processed, and performs image feature processing based on this to obtain the second image feature. The higher the accuracy of the image processing model in image feature processing, the better the image processing effect of the processed target image obtained by subsequent decoding. In the embodiments of the present specification, the image processing model is not trained jointly with the image encoding and decoding model, but is trained separately by being decoupled. Therefore, the image processing model can be changed according to the needs of image processing. For example, for image processing of portrait images, image processing model 1 is trained specifically, and for image processing of landscape images, image processing model 2 is trained specifically. According to the image type of the image to be processed, the corresponding image processing model is selected for image processing, so that the selection of the image processing model has higher flexibility and the image processing has higher generalization.

[0060] The second image feature is the image coding feature of the processed image, which is expressed as a feature vector of a specific scale (dimension). The second image feature represents the high-dimensional, abstract key features such as the structure, shape, and texture of the processed image. For example, the feature in the latent space obtained after processing the image to be processed is the second latent space feature. For example, a landscape photo is an image to be restored. The image processing model performs image feature processing on its first image feature, and the obtained second image feature represents the high-dimensional, abstract key features such as the structure, shape, and texture of various landforms in the restored landscape photo.

[0061] The image processing model is used to perform image feature processing on the first image feature to obtain the second image feature. The specific method is: the image processing model is used to perform image feature processing on the first latent space feature to obtain the second latent space feature.

[0062] Exemplarily, the diffusion model Stable Diffusion is used to perform inference prediction on the first latent space feature Latent Feature_1 to obtain the second latent space feature Latent Feature_2.

[0063] The image processing model is used to perform image feature processing on the first image feature to obtain the second image feature, thereby completing the image feature processing and providing a feature basis for subsequent decoding to obtain the processed target image.

[0064] Step 108: Use the image coding and decoding model to decode the second image features to obtain a target image, wherein the image coding and decoding model is trained based on the predicted image and the label image, the predicted image is obtained by decoding the fusion features using the image coding and decoding model, and the fusion features are obtained by fusing the sample image features of the sample image and the label image features of the label image.

[0065] The target image is a visual image that has been processed, and is a high-quality visual image. The target image corresponds to the image to be processed, and can be a processed image of a real image acquired by an image acquisition device, such as a processed photo, video frame, and remote sensing image, etc., or a processed image of an artificially generated image, such as a processed picture and animation, etc., or a processed image of a virtual image generated by an image generation algorithm, such as a processed picture, animation, photo, and video frame, etc., which is not limited here.

[0066] The sample image is a training sample image for training the image encoding and decoding model, which is a visual image to be processed and a low-quality visual image. For example, 10,000 visual images of place A collected under bad weather are used as training sample images for training the image encoding and decoding model. The sample image features of the sample image are the image coding features of the sample image, which characterize the high-dimensional and abstract key features of the sample image such as structure, shape and texture, for example, the features of the sample image in the latent space, that is, the sample latent space features.

[0067] The label image is a training label image used to train the image encoding and decoding model. It is a visual image used as a reference for image processing and is a high-quality visual image. For example, 10,000 visual images of place A collected in good weather are used as training label images for training the image encoding and decoding model. The label image features of the label image are the image coding features of the label image, which characterize the high-dimensional and abstract key features of the label image, such as the structure, shape and texture, for example, the features of the label image in the latent space, i.e., the label latent space features.

[0068] The predicted image is the image output by the image encoding and decoding model during the training process of the image encoding and decoding model, which is a visual image after image processing is completed on the sample image. For example, during the training process of the image encoding and decoding model, the image encoding and decoding model performs image restoration on 10,000 visual images of place A collected under bad weather conditions, and outputs 10,000 visual images, which are the predicted images.

[0069] The fused feature is an image coding feature obtained by fusing the sample image feature of the sample image and the label image feature of the label image, and is expressed as a feature vector of a specific scale (dimension). The fused feature represents the high-dimensional, abstract key features such as the structure, shape and texture of the sample image and the label image, for example, the feature under the latent space, that is, the fused latent space feature. From the perspective of the sample image, the fused feature is the image coding feature obtained by image processing. For example, the image quality of the sample image is regarded as 0, the image quality of the label image is regarded as 1, and the image quality represented by the fused feature is 0.5. The image processing is performed relative to the sample image. In the training process of the image processing model, the image quality represented by the prediction feature output by the image processing model is also in the interval (0, 1). Therefore, the fused feature can be understood as the prediction feature output by the image processing model. The predicted image obtained by decoding the fusion features can be equivalently understood as the predicted image obtained in the training process of coupling the image encoding and decoding model and the image processing model. Training the image encoding and decoding model in this way can achieve the same training effect, but at the same time avoids introducing the image processing model to participate in the pre-training of the image encoding and decoding model, thereby realizing the decoupling of the image processing model and the image encoding and decoding model in training, and effectively improving the training effect and image processing effect.

[0070] Optionally, the predicted image is obtained by decoding the fusion features using an image coding and decoding model, specifically: the predicted image is obtained by decoding the fusion features based on multi-scale coding features using an image coding and decoding model. Furthermore, the predicted image is obtained by performing fusion decoding of the fusion features at corresponding scales based on multi-scale coding features using an image coding and decoding model, wherein the fusion decoding is performed correspondingly guided by the coding features at corresponding scales. In this way, the information loss between scales caused by the image coding process is compensated, the image distortion problem in the image processing process is effectively overcome, the training effect of the model is improved, and thus the image processing effect is improved.

[0071] The image encoding and decoding model is used to decode the second image feature to obtain the target image. The specific method is: the image encoding and decoding model is used to map the second image feature to the pixel space, and the processed target image is obtained by decoding.

[0072] Exemplarily, the pre-trained VAEGAN model is used to map the second latent space feature Latent Feature_2 to the pixel space, decode to obtain the video frame of the repaired monitoring video stream, and integrate to obtain the monitoring video stream of the intersection in rainless weather collected and sent by a camera at a road intersection.

[0073] In the embodiment of the present specification, an image to be processed is obtained; the image to be processed is input into the encoder of the image coding and decoding model, and the first image feature is encoded; the first image feature is input into the image processing model, and the second image feature is predicted; the second image feature is input into the decoder of the image coding and decoding model, and the processed target image is decoded, wherein the image coding and decoding model is trained based on the predicted image and the label image, the predicted image is obtained by decoding the fusion feature by the decoder, and the fusion feature is obtained by fusion of the sample image feature of the sample image and the label image feature of the label image. The decoder of the image coding and decoding model is used to decode the fusion feature obtained by fusion of the sample image feature of the sample image and the label image feature of the label image, so as to pre-train the image coding and decoding model, directly fuse the label image feature of the label image as the latent space feature after image processing, avoid introducing the latent space feature predicted by the image processing model to participate in the pre-training of the image coding and decoding model, realize the decoupling of the image processing model and the image coding and decoding model in training, effectively improve the training effect and the image processing effect, and at the same time, make the selection of the image processing model more flexible, and the image processing has higher generalization.

[0074] In an optional embodiment of the present specification, the image coding and decoding model includes an encoder and a decoder;

[0075] Correspondingly, step 104 includes the following specific steps:

[0076] Inputting the image to be processed into an encoder and encoding it to obtain a first image feature;

[0077] Correspondingly, step 108 includes the following specific steps:

[0078] The second image feature is input into the decoder and decoded to obtain the target image.

[0079] Figure 2 The model architecture diagram of an image processing method provided by an embodiment of the present specification is shown, and the overall architecture of the image encoding and decoding model and the image processing model is as follows Figure 2 As shown: the encoder of the image coding and decoding model, the image processing model and the decoder of the image coding and decoding model are connected in series in sequence, the image to be processed is input into the encoder of the image coding and decoding model, encoded to obtain the first image feature, the first image feature is input into the image processing model, after image feature processing, the second image feature is obtained, the second image feature is input into the decoder of the image coding and decoding model, and decoded to obtain the target image.

[0080] The encoder of the image coding and decoding model is a model module used to encode the image features of the input image. The encoder extracts the high-dimensional, abstract key features of the input image by encoding the image features of the input image. At the same time, information loss is inevitably caused during the encoding process. For example, the encoder is an encoder that maps the pixel space features of the image to the latent space. The encoder encodes the pixel space features into latent space features, maps the pixel space features of the input image at a scale of 512×512 to the latent space features at a scale of 128×1, extracts the key features of the image, and causes information loss.

[0081] The decoder of the image encoding and decoding model is a model module for decoding image features. The decoder decodes the image features and restores the low-dimensional, concrete visual image based on the high-dimensional, abstract key features of the image. For example, the decoder maps the latent space features of the image to the pixel space decoder. The decoder decodes the latent space features into pixel space features, maps the 128×1 scale latent space features into 512×512 scale pixel space features, and decodes the processed visual image.

[0082] The image to be processed is input into the encoder, and the first image feature is obtained by encoding. Specifically, the image to be processed is input into the encoder, and the pixel space feature of the image to be processed is mapped to the latent space to obtain the first image feature.

[0083] The second image feature is input into the decoder, and the target image is obtained by decoding. Specifically, the second image feature is input into the decoder, and the second image feature is mapped to the pixel space to obtain the target image.

[0084] Exemplarily, the video frames of the monitoring video stream are input into the pre-trained VAEGAN model, the pixel space features Pixel Feature of the video frames of the monitoring video stream are mapped to the latent space to obtain the first latent space feature LatentFeature_1, the first latent space feature is input into the diffusion model Stable Diffusion, the first latent space feature is inferred and predicted to obtain the second latent space feature Latent Feature_2, the second latent space feature is input into the VAEGAN model, the second latent space feature Latent Feature_2 is mapped to the pixel space, the video frames of the monitoring video stream are decoded to obtain the repaired video frames, and the monitoring video stream of the intersection in rainless weather collected and sent by a camera at a road intersection is integrated.

[0085] In the embodiments of the present specification, the decoder of the image coding and decoding model is used to decode the fused features obtained by fusing the latent space features of the sample image and the latent space features of the label image, so as to pre-train the image coding and decoding model. The latent space features of the label image are directly fused as the latent space features after image processing, thereby avoiding the introduction of the latent space features predicted by the diffusion model to participate in the pre-training of the image coding and decoding model, realizing the decoupling of the diffusion model and the image coding and decoding model in training, effectively improving the training effect and the image processing effect, and at the same time, making the selection of the diffusion model more flexible and the image processing more generalizable.

[0086] In an optional embodiment of the present specification, before step 104, the following specific steps are also included:

[0087] Acquire a first sample set, wherein the first sample set includes a plurality of sample pairs, and the sample pairs include a sample image and a label image;

[0088] Extracting a first sample pair from the first sample set, wherein the first sample pair includes a first sample image and a first label image, and the first sample pair is any sample pair in the sample set;

[0089] Inputting the first sample image and the first label image into the encoder of the image encoding and decoding model respectively, encoding to obtain the sample image feature of the first sample image and the label image feature of the first label image;

[0090] Performing feature fusion on the sample image feature of the first sample image and the label image feature of the first label image to obtain a fusion feature;

[0091] Inputting the fused features into a decoder of an image encoding and decoding model, and decoding to obtain a first predicted image;

[0092] Based on the first predicted image and the first label image, the model parameters of the image encoding and decoding model are adjusted until the training stop condition is reached, thereby obtaining a trained image encoding and decoding model.

[0093] At present, in the scheme of training integrated image coding and decoding models and image processing models, the encoder and decoder of the image coding and decoding model are coupled to train the image processing model, so the training effect of the encoder and decoder is affected by the image processing model. Since the image processing model needs to perform inference prediction iteratively, a large inference prediction overhead is generated on the training samples, and after the model parameters of the image processing model are adjusted, the encoder and decoder need to be trained accordingly, resulting in a very high training cost for the entire training. In the embodiment of this specification, the number of sample images and label images is sufficient, and after decoupling, a large overhead on inference prediction is avoided, thereby reducing the training cost.

[0094] The first sample set is a set of training samples for training the image encoding and decoding model, and the first sample set includes multiple sample pairs, and any sample pair includes a low-quality sample image and a high-quality label image. For example, the first sample set includes 10,000 sample pairs, and the 10,000 sample pairs include 10,000 visual images of place A collected in bad weather and 10,000 visual images of place A collected in good weather. The first sample set can be obtained from a sample database, such as an open source sample database, or can be artificially constructed, such as generated by a generative model represented by an image processing model, or can be obtained from a historical database, such as obtaining visual images before processing and visual images after processing from a historical database, which is not limited here.

[0095] Feature fusion is to combine multiple feature vectors to form a feature vector with more information. The methods of feature fusion include spatial fusion, channel fusion, and time domain fusion. Spatial fusion refers to combining multiple feature images in space, for example, averaging or weighted summing the feature vectors of multiple convolutional layers. Channel fusion refers to combining multiple feature vectors in channels, for example, concatenating or weighted summing the feature vectors of multiple convolutional layers. Time domain fusion refers to combining feature vectors of multiple time steps, for example, averaging or weighted summing the feature vectors of multiple time steps of videos. According to the stage and complexity of feature fusion, it is divided into pre-fusion (early fusion), intermediate fusion, and post-fusion (late fusion).

[0096] The training stop condition is a pre-set judgment condition for stopping training, including but not limited to: a preset number of iterations, a preset loss value threshold, a preset training time and a preset model convergence condition.

[0097] The first sample image and the first label image are respectively input into the encoder of the image coding and decoding model, and the sample image features of the first sample image and the label image features of the first label image are obtained by encoding. Specifically, the first sample image is input into the encoder of the image coding and decoding model, and the pixel space features of the first sample image are mapped to the latent space to obtain the sample image features of the first sample image. The first label image is input into the encoder of the image coding and decoding model, and the pixel space features of the first label image are mapped to the latent space to obtain the label image features of the first label image. It should be noted that since the fusion feature is obtained by fusion of the sample image features of the first sample image and the label image features of the first label image, and the training of the image coding and decoding model is iterative, the sample image and the label image need to be respectively input into two encoders, and the parameters of the encoder inputted with the label image are frozen to ensure the encoding characteristics of the encoder, thereby ensuring the stability of the fusion feature.

[0098] The fused features are input into the decoder of the image coding and decoding model, and the first predicted image is obtained by decoding. Specifically, the fused features are input into the decoder of the image coding and decoding model, and the fused features are mapped to the pixel space based on the multi-scale coding features, and the first predicted image is obtained by decoding. Furthermore, the fused features are input into the decoder of the image coding and decoding model, and the fused features are mapped to the pixel space of the corresponding scale based on the multi-scale coding features, and the first predicted image is obtained by decoding.

[0099] Based on the first predicted image and the first labeled image, the model parameters of the image encoding and decoding model are adjusted, specifically, based on the first predicted image and the first labeled image, a loss value is calculated, and based on the loss value, the model parameters of the image encoding and decoding model are adjusted. The loss value is a measure of the difference between the predicted image and the labeled image, and is used to evaluate the model performance of the image encoding and decoding model, including but not limited to: cross entropy loss, mean square error loss, average error loss, and logarithmic loss.

[0100] Exemplarily, a sample set is obtained from an open source sample database, the sample set includes 8,000,000 images from 365 different real scenes, the sample set includes 4,000,000 sample pairs, and any sample pair includes a sample image and a label image. A first sample pair is extracted from the 4,000,000 sample pairs for training the current image encoding and decoding model, the first sample pair includes a first sample image Lr_1 of an urban road scene in rainy weather and a first label image Hr_1 of an urban road scene in sunny weather, the first sample image is input into the encoder Encoder_A of the VAEGAN model, the pixel space feature of the first sample image is mapped to the latent space, and the latent space feature Z of the first sample image is obtained, the first label image is input into the encoder Encoder_B of the VAEGAN model, the pixel space feature of the first label image is mapped to the latent space, and the latent space feature Z' of the first label image is obtained, and the first sample image is decoded. The latent space features of the first label image and the latent space features of the first label image are averaged to obtain the fusion feature Fusion=1 / 2(Z+Z'), and the fusion feature is input into the decoder Decoder of the VAEGAN model. Based on the multi-scale coding features, the fusion feature is mapped to the pixel space and decoded to obtain the first prediction image Preditct.Hr_1. Based on the first prediction image and the first label image, the loss value Loss is calculated. Based on the loss value, the model parameters of the encoder Encoder_A and the decoder Decoder in the VAEGAN model are adjusted until the input and output of all 4,000,000 sample pairs are completed, and the trained VAEGAN model is obtained.

[0101] In the embodiments of the present specification, the decoder of the image coding and decoding model is used to decode the fused features obtained by fusing the sample image features of the sample image and the label image features of the label image to pre-train the image coding and decoding model, and the label image features of the label image are directly fused as the latent space features after image processing, thereby avoiding the introduction of the latent space features predicted by the image processing model to participate in the pre-training of the image coding and decoding model, realizing the decoupling of the image processing model and the image coding and decoding model in training, effectively improving the training effect, and reducing the training cost of model training.

[0102] In an optional embodiment of the present specification, after the first sample image is input into the encoder of the image coding and decoding model, the following specific steps are also included:

[0103] Obtaining a multi-scale coding feature of a first sample image;

[0104] Correspondingly, the fusion feature is input into the decoder of the image encoding and decoding model, and the first predicted image is obtained by decoding, which includes the following specific steps:

[0105] A decoder of the image encoding and decoding model is used to calculate multi-scale decoding features based on the multi-scale encoding features and the fusion features, and a first predicted image is decoded based on the multi-scale decoding features.

[0106] At present, when the encoder encodes the image and realizes the mapping from pixel space features to latent space features, information loss is inevitably caused. Therefore, when it is mapped back to the pixel space in the decoder, the predicted image obtained by decoding will inevitably have image distortion, for example, the text in the image is distorted. Therefore, in order to avoid excessive image distortion of the predicted image, it is necessary to use the multi-scale coding features obtained by the encoder in the encoding process to perform a certain degree of information compensation. Based on this, the embodiment of this specification provides a fine-tuning strategy for the decoder decoding guided by the multi-scale coding features.

[0107] The encoder performs image encoding by gradually mapping from a large-scale pixel space to a small-scale latent space. The encoder has multiple encoding units of different scales. For example, the encoder has 5 encoding units, and the scales of the 5 encoding units are: 512×512 scale, 512×256 scale, 256×128 scale, 128×32 scale and 128×1 scale. The pixel space features of the sample image of 512×512 scale are gradually mapped to the latent space features of 128×1 scale, which causes information loss in the process. At the same time, each encoding unit outputs the encoding features of the corresponding scale, including the encoding features of 512×256 scale, the encoding features of 256×128 scale, and the encoding features of 128×32 scale. Correspondingly, the decoder performs image decoding by gradually mapping from a small-scale latent space back to a large-scale pixel space. The decoder has multiple decoding units of corresponding scales.

[0108] The multi-scale coding feature is the coding feature output by the multi-scale coding unit in the encoder, and is the intermediate feature in the process of mapping from pixel space features to latent space features.

[0109] The multi-scale decoding feature is the decoding feature output by the multi-scale decoding unit in the decoder, and is an intermediate feature in the process of mapping from latent space features to pixel space features.

[0110] The multi-scale coding feature of the first sample image is obtained by obtaining the coding feature output by the multi-scale coding unit in the encoder to obtain the multi-scale coding feature of the first sample image.

[0111] A decoder of the image coding and decoding model is used to calculate multi-scale decoding features based on multi-scale coding features and fusion features, and a first predicted image is decoded based on the multi-scale decoding features. The specific method is as follows: a multi-scale decoding unit in the decoder of the image coding and decoding model is used to calculate multi-scale decoding features based on multi-scale coding features and fusion features, and a first predicted image is decoded based on the multi-scale decoding features.

[0112] Exemplarily, the encoder of the VAEGAN model includes 7 encoding units of different scales, and the encoding features output by the 7 encoding units of different scales are obtained to obtain the multi-scale encoding features of the first sample image Lr_1 Using the 7 different scales of decoding units in the decoder of the VAEGAN model, based on the multi-scale encoding features and fusion features, the 7 scales of decoding features are calculated Based on the multi-scale decoding features, the first predicted image Preditct.Hr_1 is decoded.

[0113] In the embodiments of this specification, the multi-scale coding features obtained by the encoder are introduced into the decoding process of the decoder, which makes up for the information loss in the encoding process, realizes the joint adjustment of the model parameters of the encoder and decoder, solves the image distortion problem caused by information loss, and further improves the training effect of the image encoding and decoding model.

[0114] In an optional embodiment of the present specification, a decoder of an image coding and decoding model is used to calculate a multi-scale decoding feature based on the multi-scale coding feature and the fusion feature, and a first predicted image is decoded based on the multi-scale decoding feature, including the following specific steps:

[0115] Using the first decoding unit, based on the encoding features and the fusion features of the first scale, a decoding feature of the first scale is calculated;

[0116] For any decoding unit except the first decoding unit, based on the encoding feature of the corresponding scale of any decoding unit and the decoding feature output by the previous decoding unit, calculate the decoding feature of the corresponding scale of any decoding unit;

[0117] Until the decoding features of the target scale are calculated, the first predicted image is decoded based on the decoding features of the target scale.

[0118] The scale of the encoding unit in the encoder decreases in sequence, and correspondingly, the scale of the decoding unit in the decoder increases in sequence.

[0119] The first decoding unit is the first-order decoding unit in the decoder, and is the decoding unit with the smallest scale among multiple decoding units of different scales. The first decoding unit directly calculates the decoding features of the fused features.

[0120] Any decoding unit except the first decoding unit is a decoding unit of other orders, which is a decoding unit of other scales except the smallest scale among the decoding units of multiple different scales, and does not directly perform decoding feature calculation on the fusion feature.

[0121] The coding feature of the first scale is the coding feature output by the coding unit of the first scale, is the coding feature output by the coding unit with the smallest scale among the coding units of multiple different scales, and is the coding feature output by the last coding unit in the encoder.

[0122] The decoded features of the first scale are the decoded features output by the first decoding unit.

[0123] The coding feature corresponding to the scale of any decoding unit is the coding feature output by the coding unit corresponding to the scale of any decoding unit, and is the coding feature output by the coding units of other scales except the smallest scale among the coding units of multiple different scales.

[0124] The target scale is the largest scale among multiple different scales, that is, the original scale of the image. The decoded features of the target scale are the decoded features of the original scale of the image, that is, the pixel space features.

[0125] In the embodiment of this specification, the multi-scale coding features of the first sample image Lr_1 are obtained. Correspondingly, the multi-scale decoding unit in the decoder calculates the decoding features of the corresponding scale And the input of the decoding unit of the i-th scale includes the decoding features of the i-1-th scale And the encoding features output by the encoding unit of the corresponding scale Therefore, the embodiment of this specification can be specifically expressed by Formula 1:

[0126]

[0127]

[0128] in, is the decoding feature of the first scale, Net() is the calculation process, F is the fusion feature (Fusion), is the encoding feature of the first scale, is the decoding feature of the corresponding scale of any decoding unit, is the decoding feature output by the previous decoding unit, is the encoding feature of the corresponding scale of any decoding unit, and θ is the learnable model parameter. and After feature fusion in advance, the corresponding decoding features are calculated.

[0129] Exemplarily, using the first decoding unit, based on the encoding features of the first scale And fusion feature Fusion, calculate the decoding features of the first scale For any decoding unit except the first decoding unit, based on the encoding features of the corresponding scale of any decoding unit And the decoding features output by the previous decoding unit Calculate the decoding features of any decoding unit corresponding to the scale Until the decoding features of the 7th scale are calculated Based on the decoded features of the 7th scale, the first predicted image Preditct.Hr_1 is decoded.

[0130] In the embodiments of this specification, the multi-scale coding features obtained by the encoder are introduced into the decoding process of the decoder scale by scale, which compensates for the information loss at each scale in the encoding process, realizes the joint adjustment of the model parameters of the encoder and decoder, solves the image distortion problem caused by information loss, and further improves the training effect of the image encoding and decoding model.

[0131] In an optional embodiment of the present specification, performing feature fusion on the sample image feature of the first sample image and the label image feature of the first label image to obtain the fused feature includes the following specific steps:

[0132] A pre-fusion process is performed on the sample image feature of the first sample image and the label image feature of the first label image to obtain a fusion feature.

[0133] The pre-fusion processing is a method of directly fusing multiple latent space features in the latent space, including space fusion, channel fusion, time domain fusion, etc. The specific fusion processing includes but is not limited to: Concat processing and Add processing.

[0134] Exemplarily, a concat process is performed on the sample image feature Z of the first sample image and the label image feature Z′ of the first label image, and the convolution layer is input to obtain a fusion feature Fusion=Conv(Concat(Z, Z′)).

[0135] In the embodiments of the present specification, the fusion features are obtained by using the pre-fusion process, which improves the accuracy of the fusion features and further improves the training effect of the image encoding and decoding model.

[0136] In an optional embodiment of the present specification, adjusting the model parameters of the image coding and decoding model based on the first predicted image and the first label image includes the following specific steps:

[0137] Based on the first predicted image and the first label image, a loss value is calculated;

[0138] Based on the loss value, the model parameters of the encoder and decoder are jointly adjusted.

[0139] The loss value measures the difference between the predicted image and the labeled image and is used to evaluate the model performance of the image encoding and decoding model, including but not limited to: cross entropy loss, mean square error loss, average error loss, and logarithmic loss.

[0140] Based on the loss value, the model parameters of the encoder and decoder are jointly adjusted. The specific method is: based on the loss value, the model parameters of the encoder and decoder are jointly adjusted using the gradient descent method. Among them, the gradient descent method is a commonly used model parameter adjustment algorithm, which is used to solve the minimum value of the loss value and realize the optimization path of model performance. The idea of ​​the gradient descent method is: starting from the current loss value, iterate in the opposite direction of the gradient to find the minimum point of the loss value function. The iterative process of the gradient descent method can be expressed as: X(k+1)=X(k)-α*g(X(k)), where X(k) represents the current loss value, α represents the learning rate, and g() represents the opposite direction of the gradient.

[0141] Exemplarily, based on the first predicted image and the first label image, the loss value Loss is calculated. Based on the loss value, the model parameters of the encoder Encoder_A and the decoder Decoder in the VAEGAN model are jointly adjusted using the gradient descent method until the input and output of all 4,000,000 sample pairs are completed, thereby obtaining a trained VAEGAN model.

[0142] In the embodiment of the present specification, the loss value is calculated based on the first predicted image and the first label image, and then the gradient descent method is used to jointly adjust the model parameters of the encoder and decoder, which improves the efficiency of model training, reduces the complexity of model training, and further improves the training effect.

[0143] In an optional embodiment of the present specification, before step 106, the following specific steps are also included:

[0144] Acquire a second sample set, wherein the second sample set includes a plurality of sample pairs, and the sample pairs include a sample image and a label image;

[0145] Extracting a second sample pair from the second sample set, wherein the second sample pair includes a second sample image and a second label image, and the second sample pair is any sample pair in the second sample set;

[0146] Inputting the second sample image into the image processing model to obtain a second predicted image;

[0147] Based on the second predicted image and the second label image, the model parameters of the image processing model are adjusted until the training stop condition is reached, thereby obtaining a trained image processing model.

[0148] Because the training of the image processing model and the image encoding and decoding model are decoupled, the image processing model can be trained separately. The essence of the image processing model's inference and prediction is forward denoising and backward denoising, so it can be trained directly with sample images and label images.

[0149] The second sample set is a set of training samples for training the image processing model, and the second sample set includes multiple sample pairs, and any sample pair includes a low-quality sample image and a high-quality label image. For example, the second sample set includes 5,000 sample pairs, and the 5,000 sample pairs include 5,000 face photos with occluded faces and 5,000 face photos without occluded faces. The second sample set can be obtained from a sample database, such as an open source sample database, or can be artificially constructed, such as generated using an image processing model represented by an image processing model, or can be obtained from a historical database, such as obtaining visual images before processing and visual images after processing from a historical database, which is not limited here.

[0150] The sample image is a training sample image used to train the image processing model, which is a visual image to be processed and a low-quality visual image. The label image is a training label image used to train the image processing model, which is a visual image used as a reference for image processing and a high-quality visual image. The predicted image is an image output by the image processing model during the training of the image processing model and is a visual image that has been processed.

[0151] The training stop condition is a pre-set judgment condition for stopping training, including but not limited to: a preset number of iterations, a preset loss value threshold, a preset training time and a preset model convergence condition.

[0152] The second sample image is input into the image processing model to obtain a second predicted image. Specifically, the second sample image is input into the image processing model, and image processing is performed based on the second sample image to obtain the second predicted image.

[0153] Based on the second predicted image and the second label image, the model parameters of the image processing model are adjusted, specifically, based on the second predicted image and the second label image, a loss value is calculated, and based on the loss value, the model parameters of the image processing model are adjusted. The loss value is a measure of the difference between the predicted image and the label image, and is used to evaluate the model performance of the image processing model, including but not limited to: cross entropy loss, mean square error loss, average error loss, and logarithmic loss.

[0154] Exemplarily, a sample set is obtained from an open source sample database, the sample set includes 200,000 images of different faces, face occlusion is added to the 200,000 images of different faces, and a second sample set is constructed, the second sample set includes 200,000 sample pairs, and any sample pair includes a sample image and a label image. A second sample pair is extracted from the 200,000 sample pairs for training the current diffusion model Stable Diffusion, the second sample pair includes a second sample image Lr_2 with face occlusion and a second label image Hr_2 without face occlusion, the second sample image is input into the diffusion model, inference prediction is performed based on the second sample image, and a second predicted image Preditct.Hr_2 is obtained, and a loss value Loss is calculated based on the second predicted image and the second label image, and the model parameters of the diffusion model are adjusted based on the loss value until the input and output of all 200,000 sample pairs are completed, and the trained diffusion model Stable Diffusion is obtained.

[0155] In the embodiments of the present specification, the image processing function of the image processing model is improved through supervised training of the image processing model, which is independent of the training of the image encoding and decoding model, and the image processing effect is improved.

[0156] In an optional embodiment of the present specification, step 102 includes the following specific steps:

[0157] Receive the video to be processed sent by the front-end user;

[0158] Extracting a video frame to be processed from the video to be processed;

[0159] Correspondingly, after step 108, the following specific steps are also included:

[0160] synthesizing a target video based on the processed target video frame;

[0161] Feedback the target video to the front-end user.

[0162] At present, low-quality videos are transcoded into high-quality videos, including video processing enhancement pre-processing and video compression encoding. Improving video quality mainly relies on pre-video processing enhancement pre-processing, which is implemented by the embodiments of this specification, effectively improving the quality of the transcoded video and improving the user experience of front-end users.

[0163] The video to be processed is a video with damaged image quality or missing information, which is a low-quality video. This is caused by problems in the process of video acquisition, transmission, storage, or video editing and synthesis. For example, a camera may capture a blurry video due to weather conditions. The video frame to be processed is a video frame with damaged image quality or missing information, which is a low-quality visual image.

[0164] The target video is a video with no image quality damage or information supplementation, and is a high-quality video. The target video frame is a video frame that has been processed, and is obtained through the processing from step 104 to step 108, and is a high-quality visual image.

[0165] Exemplarily, receive the video to be processed video.mp4 (IPBIPPBIPBPB...) sent by the front-end user on the client of the application, extract 120 video frames to be processed from the video to be processed: I, P, B, I..., and obtain 120 processed target video frames through processing from step 104 to step 108. Based on the processed target video frames, synthesize the target video video (restored).mp4, and feed back the target video to the front-end user.

[0166] In the embodiment of the present specification, the video to be processed sent by the front-end user is processed by interactive means using steps 104 to 108, thereby effectively improving the video quality and user experience.

[0167] In an optional embodiment of the present specification, after feeding back the target video to the front-end user, the following specific steps are also included:

[0168] Receive adjustment information sent by a front-end user for a target video;

[0169] Based on the adjustment information, model parameters of the image encoding and decoding model are adjusted.

[0170] The adjustment information is feedback information of the front-end user generated and sent for the target video, including but not limited to: information loss and image quality damage.

[0171] Based on the adjustment information, the model parameters of the image coding and decoding model are adjusted, specifically in the following manner: based on the adjustment information, the sample image and the label image are updated, and based on the updated sample image and the label image, the model parameters of the image coding and decoding model are adjusted. Wherein, based on the updated sample image and the label image, the model parameters of the image coding and decoding model are adjusted, specifically in the following manner: based on the updated sample image and the label image, positive and negative samples are determined, and the model parameters of the image coding and decoding model are adjusted using the positive and negative samples.

[0172] Exemplarily, feedback information generated and sent by a front-end user for a target video video(restored).mp4 is received. There is still information loss in the first frame, and image quality is damaged in the third frame. Based on the adjustment information, the sample image and the label image are updated. Based on the updated sample image and the label image, positive and negative samples are determined. The model parameters of the VAEGAN model are adjusted using the positive and negative samples.

[0173] In the embodiments of this specification, further adjustment of the image encoding and decoding model parameters is completed through an interactive manner, thereby specifically improving the subsequent video processing effect.

[0174] Figure 3 A schematic diagram of training an image encoding and decoding model in an image processing solution provided by an embodiment of this specification is shown. Figure 3 As shown:

[0175] The sample image and the label image are respectively input into the encoder of the image coding and decoding model, and the corresponding sample image features of the input sample image and the label image features of the label image are used. Based on the sample image features of the sample image and the label image features of the label image, feature fusion is performed to obtain fused features, and the fused features are input into the encoder of the image coding and decoding model. At the same time, the multi-scale features of the sample image obtained in the encoding process are introduced into the decoding process of the decoder of the image coding and decoding model by using the feature fusion module, and the decoding features of the target scale are calculated. Based on the decoding features of the target scale, a predicted image is obtained by decoding, and based on the predicted image and the label image, the parameters of the encoder and decoder of the image coding and decoding model are adjusted.

[0176] See also Figure 4 , Figure 4 A flowchart of an image restoration method provided by an embodiment of the present specification is shown, including the following specific steps:

[0177] Step 402: Acquire the image to be restored.

[0178] Step 404: Encode the image to be repaired using the image encoding and decoding model to obtain a first image feature.

[0179] Step 406: Use the image restoration model to perform image feature restoration on the first image feature to obtain a second image feature.

[0180] Step 408: Use the image coding and decoding model to decode the second image features to obtain a restored target image, wherein the image coding and decoding model is trained based on the predicted image and the label image, the predicted image is obtained by decoding the fusion features using the image coding and decoding model, and the fusion features are obtained by fusing the sample image features of the sample image and the label image features of the label image.

[0181] The embodiments of this specification are applied to the client or server of an application, website or applet with an image restoration function, and an image encoding and decoding model and an image processing model are deployed on the client or server to implement the image restoration function.

[0182] The embodiments of this specification are similar to the above Figure 1 The embodiments of the specification are based on the same inventive concept. For the specific methods of steps 402 to 408, please refer to Figure 1 The contents of step 102 to step 108 in the embodiment of the specification.

[0183] In the embodiment of the present specification, an image to be repaired is obtained; an image coding and decoding model is used to encode the image to be repaired to obtain a first image feature; an image repair model is used to repair the first image feature to obtain a second image feature; and an image coding and decoding model is used to decode the second image feature to obtain a repaired target image, wherein the image coding and decoding model is obtained by training based on a predicted image and a label image, the predicted image is obtained by decoding a fusion feature using the image coding and decoding model, and the fusion feature is obtained by fusion of a sample image feature of a sample image and a label image feature of a label image. The image coding and decoding model is used to decode a fusion feature obtained by fusion of the sample image feature of the sample image and the label image feature of the label image, so as to pre-train the image coding and decoding model, directly fuse the label image feature of the label image as the image feature after image repair, avoid introducing the image repair model to participate in the pre-training of the image coding and decoding model, realize the decoupling of the image repair model and the image coding and decoding model in training, effectively improve the training effect and the image repair effect, and at the same time, make the selection of the image repair model more flexible, and the image repair has higher generalization.

[0184] See also Figure 5 , Figure 5 A flowchart of another image processing method provided by an embodiment of the present specification is shown, which is applied to a cloud-side device and includes the following specific steps:

[0185] Step 502: receiving an image to be processed sent by a device on the receiving end side.

[0186] Step 504: Encode the image to be processed using the image encoding and decoding model to obtain a first image feature.

[0187] Step 506: Using the image processing model, perform image feature processing on the first image feature to obtain a second image feature.

[0188] Step 508: Use the image coding and decoding model to decode the second image features to obtain a target image, wherein the image coding and decoding model is trained based on the predicted image and the label image, the predicted image is obtained by decoding the fusion features using the image coding and decoding model, and the fusion features are obtained by fusing the sample image features of the sample image and the label image features of the label image.

[0189] Step 510: Feedback the target image to the terminal device.

[0190] The embodiments of this specification are applied to the network cloud device where the server of a web page, application or applet with image processing function is located. It is a virtual device, and an image encoding and decoding model and an image processing model with image processing function are deployed on the cloud-side device. The end-side device is a physical device where the client of the web page, application or applet with image processing function logged in by the user is located. The cloud-side device and the end-side device are connected through a network transmission channel for data transmission. The computing power performance and storage performance of the cloud-side device are higher than those of the end-side device.

[0191] The embodiments of this specification are similar to the above Figure 1 The embodiments of the specification are based on the same inventive concept. For the specific methods of steps 504 to 508, please refer to Figure 1 The contents of step 104 to step 108 in the embodiment of the specification.

[0192] In an embodiment of the present specification, an image processing method is provided, which is applied to a cloud-side device, including: receiving an image to be processed sent by a terminal-side device; encoding the image to be processed using an image coding and decoding model to obtain a first image feature; performing image feature processing on the first image feature using the image processing model to obtain a second image feature; decoding the second image feature using the image coding and decoding model to obtain a target image, wherein the image coding and decoding model is obtained based on training of a predicted image and a label image, the predicted image is obtained by decoding a fusion feature using the image coding and decoding model, and the fusion feature is obtained based on a fusion of a sample image feature of a sample image and a label image feature of a label image; and feeding back the target image to the terminal-side device. The image encoding and decoding model is used to decode the fused features obtained by fusing the sample image features of the sample image and the label image features of the label image, so as to pre-train the image encoding and decoding model. The label image features of the label image are directly fused as the image features after image processing, thereby avoiding the introduction of the image processing model to participate in the pre-training of the image encoding and decoding model, realizing the decoupling of the image processing model and the image encoding and decoding model in training, effectively improving the training effect and image processing effect, and at the same time, making the selection of the image processing model more flexible, and the image processing more generalizable. In addition, the high computing performance and high storage performance of the cloud-side devices are utilized to improve the efficiency and stability of image processing.

[0193] In an optional embodiment of the present specification, after step 510, the following specific steps are also included:

[0194] The adjustment information for the target image sent by the receiving end device;

[0195] Based on the adjustment information, model parameters of the image encoding and decoding model are adjusted.

[0196] The embodiments of this specification and Figure 1 The embodiments of the specification are based on the same inventive concept. The specific methods of the embodiments of this specification are consistent with the above-mentioned scheme of further adjusting the image coding and decoding model parameters through the interactive method, and will not be repeated here.

[0197] In the embodiments of this specification, further adjustment of the image encoding and decoding model parameters is completed through an interactive manner, thereby specifically improving the subsequent image processing effect.

[0198] See also Figure 6 , Figure 6 A flowchart of a data processing method for image processing provided by an embodiment of the present specification is shown, which is applied to a cloud-side device and includes the following specific steps:

[0199] Step 602: Acquire a sample set, wherein the sample set includes a plurality of sample pairs, and the sample pairs include a sample image and a label image.

[0200] Step 604: extract a first sample pair from the sample set, wherein the first sample pair includes a first sample image and a first label image, and the first sample pair is any sample pair in the sample set.

[0201] Step 606: input the first sample image and the first label image into the encoder of the image coding and decoding model respectively, and encode to obtain the sample image features of the first sample image and the label image features of the first label image.

[0202] Step 608: Fusing the sample image features of the first sample image and the label image features of the first label image to obtain fused features.

[0203] Step 610: Input the fused features into the decoder of the image encoding and decoding model, and decode to obtain a first predicted image.

[0204] Step 612: Based on the first predicted image and the first label image, adjust the model parameters of the image encoding and decoding model until the training stop condition is reached, thereby obtaining a trained image encoding and decoding model.

[0205] Step 614: Feedback the model parameters of the image encoding and decoding model to the terminal device.

[0206] The embodiments of this specification are applied to network cloud devices with model training functions, which are virtual devices. The end-side device is a terminal where the client of the web page, application or applet with model training function logged in by the user is located, which is a physical device. The cloud-side device and the end-side device are connected through a network transmission channel for data transmission. The computing power performance and storage performance of the cloud-side device are higher than those of the end-side device.

[0207] The embodiments of this specification are similar to the above Figure 1 The embodiments of the specification are based on the same inventive concept. For the specific methods of steps 602 to 612, please refer to Figure 1 The contents of the image encoding and decoding model pre-training embodiment in the specification embodiment.

[0208] In an embodiment of the present specification, a data processing method for image processing is provided, which is applied to a cloud-side device, including: obtaining a sample set, wherein the sample set includes multiple sample pairs, and the sample pair includes a sample image and a label image; extracting a first sample pair from the sample set, wherein the first sample pair includes a first sample image and a first label image, and the first sample pair is any sample pair in the sample set; inputting the first sample image and the first label image into an encoder of an image coding and decoding model respectively, encoding to obtain a sample image feature of the first sample image and a label image feature of the first label image; performing feature fusion on the sample image feature of the first sample image and the label image feature of the first label image to obtain a fused feature; inputting the fused feature into a decoder of the image coding and decoding model, decoding to obtain a first predicted image; adjusting the model parameters of the image coding and decoding model based on the first predicted image and the first label image until a training stop condition is reached, and obtaining a trained image coding and decoding model; and feeding back the model parameters of the image coding and decoding model to the end-side device. The decoder of the image coding and decoding model is used to decode the fused features obtained by fusing the sample image features of the sample image and the label image features of the label image to pre-train the image coding and decoding model. The label image features of the label image are directly fused as the latent space features after image processing, avoiding the introduction of latent space features predicted by the image processing model to participate in the pre-training of the image coding and decoding model, realizing the decoupling of the image processing model and the image coding and decoding model in training, effectively improving the training effect and image processing effect, and at the same time, making the selection of the image processing model more flexible and the image processing more generalizable. In addition, the high computing performance and high storage performance of the cloud-side devices are utilized to improve the efficiency and training effect of the model training.

[0209] The following combination Figure 7 , taking the application of the image processing method provided in this specification in improving video quality as an example, the image processing method is further described. Figure 7A flowchart of a processing process of an image processing method for improving video quality provided by an embodiment of the present specification is shown, including the following specific steps:

[0210] Step 702: Obtain a sample set, wherein the sample set includes 800,000 sample pairs, and the sample pairs include standard definition sample images and high definition label images.

[0211] Step 704: extract a first sample pair from the sample set, wherein the first sample pair includes a first standard definition sample image and a first high definition label image, and the first sample pair is any sample pair in the sample set.

[0212] Step 706: Input the first SD sample image and the first HD label image into the encoder of the VAEGAN model respectively, and encode to obtain the latent space features of the first SD sample image, the latent space features of the first HD label image, and the multi-scale coding features of the first SD sample image.

[0213] Step 708: Perform pre-fusion processing on the latent space features of the first standard definition sample image and the latent space features of the first high definition label image to obtain fusion features.

[0214] Step 710: using the decoder of the VAEGAN model, based on the multi-scale coding features and the fusion features, calculate the multi-scale decoding features, and decode to obtain a first high-definition predicted image based on the multi-scale decoding features.

[0215] Step 712: Based on the first high-definition predicted image and the first high-definition label image, a loss value is calculated, and based on the loss value, the model parameters of the encoder and the decoder are jointly adjusted using the gradient descent method until the training stop condition is reached, thereby obtaining a trained VAEGAN model.

[0216] Step 714: Obtain the standard definition movie video to be improved sent by the front-end user, wherein the standard definition movie video includes 260,000 standard definition video frames.

[0217] Step 716: Input the standard definition video frame into the encoder of the VAEGAN model and encode it to obtain the first latent space feature.

[0218] Step 718: Input the first latent space feature into the diffusion model to predict the second latent space feature.

[0219] Step 720: Input the second latent space features into the decoder of the VAEGAN model, and decode to obtain the transcoded high-definition video frame.

[0220] Step 722: synthesize a high-definition movie video based on the transcoded 260,000 high-definition video frames, and feed the high-definition movie video back to the front-end user.

[0221] In the embodiments of this specification, the multi-scale coding features obtained by the encoder are introduced into the decoding process of the decoder, which makes up for the information loss in the encoding process, realizes the joint adjustment of the model parameters of the encoder and the decoder, solves the image distortion problem caused by information loss, and improves the training effect of the VAEGAN model. The decoder of the VAEGAN model is used to decode the fusion features obtained by fusing the latent space features of the sample image and the latent space features of the label image to pre-train the VAEGAN model, directly fusing the latent space features of the label image as the latent space features after image processing, avoiding the introduction of the latent space features predicted by the diffusion model to participate in the pre-training of the VAEGAN model, realizing the decoupling of the diffusion model and the VAEGAN model in training, effectively improving the training effect and image processing effect, and at the same time, making the selection of the diffusion model more flexible and the image processing more generalizable.

[0222] Corresponding to the above method embodiment, this specification also provides an image processing device embodiment, Figure 8 FIG. 2 shows a schematic diagram of the structure of an image processing device provided by an embodiment of the present specification. Figure 8 As shown, the device comprises:

[0223] A first acquisition module 802 is configured to acquire an image to be processed;

[0224] The first encoding module 804 is configured to encode the image to be processed by using an image encoding and decoding model to obtain a first image feature;

[0225] The first prediction module 806 is configured to perform image feature processing on the first image feature using an image processing model to obtain a second image feature;

[0226] The first decoding module 808 is configured to use an image coding and decoding model to decode the second image feature to obtain a target image, wherein the image coding and decoding model is trained based on a predicted image and a label image, the predicted image is obtained by decoding a fusion feature using the image coding and decoding model, and the fusion feature is obtained by fusing a sample image feature of the sample image and a label image feature of the label image.

[0227] Optionally, the image encoding and decoding model includes an encoder and a decoder;

[0228] Correspondingly, the first encoding module 804 is further configured as follows:

[0229] Inputting the image to be processed into an encoder and encoding it to obtain a first image feature;

[0230] Correspondingly, the first decoding module 808 is further configured to:

[0231] The second image feature is input into the decoder and decoded to obtain the target image.

[0232] Optionally, the device further comprises:

[0233] The first training module is configured to obtain a first sample set, wherein the first sample set includes multiple sample pairs, and the sample pairs include sample images and label images; extract a first sample pair from the first sample set, wherein the first sample pair includes a first sample image and a first label image, and the first sample pair is any sample pair in the sample set; input the first sample image and the first label image into the encoder of the image coding and decoding model respectively, and encode to obtain sample image features of the first sample image and label image features of the first label image; perform feature fusion on the sample image features of the first sample image and the label image features of the first label image to obtain fused features; input the fused features into the decoder of the image coding and decoding model, and decode to obtain a first predicted image; based on the first predicted image and the first label image, adjust the model parameters of the image coding and decoding model until the training stop condition is reached, and obtain a trained image coding and decoding model.

[0234] Optionally, the first training module is further configured as

[0235] A multi-scale coding feature of a first sample image is obtained; a multi-scale decoding feature is calculated based on the multi-scale coding feature and the fusion feature using a decoder of the image coding and decoding model; and a first predicted image is decoded based on the multi-scale decoding feature.

[0236] Optionally, the first training module is further configured to:

[0237] The first decoding unit is used to calculate the decoding features of the first scale based on the coding features and fusion features of the first scale; for any decoding unit except the first decoding unit, the decoding features of the corresponding scale of any decoding unit are calculated based on the coding features of the corresponding scale of any decoding unit and the decoding features output by the previous decoding unit; until the decoding features of the target scale are calculated, and the first predicted image is decoded based on the decoding features of the target scale.

[0238] Optionally, the first training module is further configured to:

[0239] A pre-fusion process is performed on the sample image feature of the first sample image and the label image feature of the first label image to obtain a fusion feature.

[0240] Optionally, the first training module is further configured to:

[0241] A loss value is calculated based on the first predicted image and the first label image; and model parameters of the encoder and the decoder are jointly adjusted based on the loss value.

[0242] Optionally, the device further comprises:

[0243] The second training module is configured to obtain a second sample set, wherein the second sample set includes multiple sample pairs, and the sample pairs include sample images and label images; extract second sample pairs from the second sample set, wherein the second sample pairs include second sample images and second label images, and the second sample pairs are any sample pairs in the second sample set; input the second sample images into the image processing model to obtain a second predicted image; based on the second predicted image and the second label image, adjust the model parameters of the image processing model until the training stop condition is reached, and obtain a trained image processing model.

[0244] Optionally, the first acquisition module 802 is further configured to:

[0245] Receive the video to be processed sent by the front-end user; extract the video frame to be processed from the video to be processed;

[0246] Correspondingly, the device also includes:

[0247] The first feedback module is configured to synthesize a target video based on the processed target video frame; and feed back the target video to a front-end user.

[0248] Optionally, the device further comprises:

[0249] The first adjustment module is configured to receive adjustment information sent by a front-end user for a target video; and adjust model parameters of an image encoding and decoding model based on the adjustment information.

[0250] In the embodiments of the present specification, the image coding and decoding model is pre-trained by decoding the fused features obtained by fusing the sample image features of the sample image and the label image features of the label image using the image coding and decoding model, and the label image features of the label image are directly fused as the image features after image processing, thereby avoiding the introduction of the image processing model to participate in the pre-training of the image coding and decoding model, realizing the decoupling of the image processing model and the image coding and decoding model in training, effectively improving the training effect and the image processing effect, and at the same time, making the selection of the image processing model more flexible and the image processing more generalizable.

[0251] The above is a schematic scheme of an image processing device of this embodiment. It should be noted that the technical scheme of the image processing device and the technical scheme of the above-mentioned image processing method belong to the same concept, and the details not described in detail in the technical scheme of the image processing device can be referred to the description of the technical scheme of the above-mentioned image processing method.

[0252] Corresponding to the above method embodiment, this specification also provides an image processing device embodiment, Fig. 9FIG. 2 shows a schematic diagram of the structure of an image restoration device provided by an embodiment of the present specification. Fig. 9 As shown, including:

[0253] The second acquisition module 902 is configured to acquire the image to be restored.

[0254] The second encoding module 904 is configured to encode the image to be repaired by using an image encoding and decoding model to obtain a first image feature.

[0255] The second prediction module 906 is configured to perform image feature restoration on the first image feature using an image restoration model to obtain a second image feature.

[0256] The second decoding module 908 is configured to use an image encoding and decoding model to decode the second image features to obtain a restored target image, wherein the image encoding and decoding model is trained based on a predicted image and a label image, the predicted image is obtained by decoding a fusion feature using the image encoding and decoding model, and the fusion feature is obtained by fusing a sample image feature of the sample image and a label image feature of the label image.

[0257] In the embodiments of the present specification, the image coding and decoding model is pre-trained by decoding the fused features obtained by fusing the sample image features of the sample image and the label image features of the label image using the image coding and decoding model, and the label image features of the label image are directly fused as the image features after image restoration, thereby avoiding the introduction of the image restoration model to participate in the pre-training of the image coding and decoding model, achieving the decoupling of the image restoration model and the image coding and decoding model in training, effectively improving the training effect and the image restoration effect, and at the same time, making the selection of the image restoration model more flexible and the image restoration more generalizable.

[0258] The above is a schematic scheme of an image restoration device of this embodiment. It should be noted that the technical scheme of the image restoration device and the technical scheme of the above-mentioned image restoration method belong to the same concept, and the details not described in detail in the technical scheme of the image restoration device can be referred to the description of the technical scheme of the above-mentioned image restoration method.

[0259] Corresponding to the above method embodiment, this specification also provides an image processing device embodiment, Fig.10 FIG. 2 shows a schematic diagram of the structure of another image processing device provided by an embodiment of the present specification. Fig.10 As shown, the device is applied to a cloud-side device, including:

[0260] The receiving module 1002 is configured to receive an image to be processed sent by a terminal device;

[0261] The third encoding module 1004 is configured to encode the image to be processed by using the image encoding and decoding model to obtain a first image feature;

[0262] The third prediction module 1006 is configured to perform image feature processing on the first image feature using an image processing model to obtain a second image feature;

[0263] The third decoding module 1008 is configured to decode the second image feature using an image coding model to obtain a target image, wherein the image coding model is trained based on the predicted image and the label image, the predicted image is obtained by decoding the fusion feature using the image coding model, and the fusion feature is obtained by fusing the sample image feature of the sample image and the label image feature of the label image;

[0264] The third feedback module 1010 is configured to feed back the target image to the terminal device.

[0265] Optionally, the device further comprises:

[0266] The third adjustment module is configured to receive adjustment information for the target image sent by the terminal-side device; and adjust model parameters of the image coding and decoding model based on the adjustment information.

[0267] In the embodiments of the present specification, the image coding and decoding model is pre-trained by decoding the fused features obtained by fusing the sample image features of the sample image and the label image features of the label image using the image coding and decoding model, and the label image features of the label image are directly fused as the image features after image processing, thereby avoiding the introduction of the image processing model to participate in the pre-training of the image coding and decoding model, realizing the decoupling of the image processing model and the image coding and decoding model in training, effectively improving the training effect and the image processing effect, and at the same time, making the selection of the image processing model more flexible, the image processing more generalizable, and utilizing the high computing performance and high storage performance of the cloud-side device to improve the efficiency and stability of image processing.

[0268] The above is a schematic scheme of another image processing device of this embodiment. It should be noted that the technical scheme of the image processing device and the technical scheme of the above-mentioned image processing method belong to the same concept, and the details not described in detail in the technical scheme of the image processing device can be referred to the description of the technical scheme of the above-mentioned image processing method.

[0269] Corresponding to the above method embodiment, this specification also provides an embodiment of a data processing device for image processing, Fig.11 FIG. 1 is a schematic diagram showing a data processing device for image processing provided by an embodiment of the present specification. Fig.11 As shown, the device is applied to a cloud-side device, including:

[0270] A fourth acquisition module 1102 is configured to acquire a sample set, wherein the sample set includes a plurality of sample pairs, and the sample pair includes a sample image and a label image;

[0271] An extraction module 1104 is configured to extract a first sample pair from the first sample set, wherein the first sample pair includes a first sample image and a first label image, and the first sample pair is any sample pair in the sample set;

[0272] The fourth encoding module 1106 is configured to input the first sample image and the first label image into the encoder of the image encoding and decoding model respectively, and encode to obtain the sample image feature of the first sample image and the label image feature of the first label image;

[0273] A fusion module 1108 is configured to perform feature fusion on the sample image feature of the first sample image and the label image feature of the first label image to obtain a fusion feature;

[0274] The fourth decoding module 1110 is configured to input the fusion feature into a decoder of the image encoding and decoding model, and decode to obtain a first predicted image;

[0275] The adjustment module 1112 is configured to adjust the model parameters of the image coding model based on the first predicted image and the first label image until the training stop condition is reached to obtain the trained image coding model;

[0276] The fourth feedback module 1114 is configured to feed back the model parameters of the image coding and decoding model to the terminal side device.

[0277] In the embodiments of the present specification, the decoder of the image coding and decoding model is used to decode the fused features obtained by fusing the sample image features of the sample image and the label image features of the label image to pre-train the image coding and decoding model, and the label image features of the label image are directly fused as the latent space features after image processing, thereby avoiding the introduction of the latent space features predicted by the image processing model to participate in the pre-training of the image coding and decoding model, realizing the decoupling of the image processing model and the image coding and decoding model in training, effectively improving the training effect and image processing effect, and at the same time, making the selection of the image processing model more flexible, and the image processing more generalizable, and utilizing the high computing performance and high storage performance of the cloud-side device to improve the efficiency and training effect of the model training.

[0278] The above is a schematic scheme of a data processing device for image processing of this embodiment. It should be noted that the technical scheme of the data processing device for image processing and the technical scheme of the data processing method for image processing described above belong to the same concept, and the details of the technical scheme of the data processing device for image processing that are not described in detail can all be referred to the description of the technical scheme of the data processing method for image processing described above.

[0279] Fig.12 The block diagram of a computing device provided by an embodiment of the present specification is shown. The components of the computing device 1200 include but are not limited to a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and the database 1250 is used to store data.

[0280] The computing device 1200 also includes an access device 1240, which enables the computing device 1200 to communicate via one or more networks 1260. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1240 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0281] In one embodiment of the present specification, the above components of the computing device 1200 and Fig.12 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.12 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0282] The computing device 1200 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1200 may also be a mobile or stationary server.

[0283] The processor 1220 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned image processing method or image processing data processing method.

[0284] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned image processing method and image processing data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned image processing method or image processing data processing method.

[0285] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned image processing method or image processing data processing method.

[0286] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned image processing method and image processing data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned image processing method or image processing data processing method.

[0287] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned image processing method or the image processing data processing method.

[0288] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above-mentioned image processing method and image processing data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above-mentioned image processing method or image processing data processing method.

[0289] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0290] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0291] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0292] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0293] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. An image processing method, comprising: Get the image to be processed; Encoding the image to be processed by using an image encoding and decoding model to obtain a first image feature; Using an image processing model, performing image feature processing on the first image feature to obtain a second image feature; The second image feature is decoded using the image encoding and decoding model to obtain a target image, wherein the image encoding and decoding model is trained based on a predicted image and a label image, the predicted image is obtained by decoding a fusion feature using the image encoding and decoding model, and the fusion feature is obtained by fusing a sample image feature of a sample image and a label image feature of the label image.

2. According to the method of claim 1, the image coding and decoding model comprises an encoder and a decoder; The method of encoding the image to be processed by using the image encoding and decoding model to obtain the first image feature includes: Inputting the image to be processed into the encoder and encoding it to obtain a first image feature; The step of decoding the second image feature by using the image encoding and decoding model to obtain a target image includes: The second image feature is input into the decoder and decoded to obtain a target image.

3. The method according to claim 2, before inputting the image to be processed into the encoder and encoding to obtain the first image feature, further comprising: Acquire a first sample set, wherein the first sample set includes a plurality of sample pairs, and the sample pairs include a sample image and a label image; Extracting a first sample pair from the first sample set, wherein the first sample pair includes a first sample image and a first label image, and the first sample pair is any sample pair in the sample set; Inputting the first sample image and the first label image into an encoder of an image coding and decoding model respectively, and encoding to obtain a sample image feature of the first sample image and a label image feature of the first label image; Performing feature fusion on the sample image feature of the first sample image and the label image feature of the first label image to obtain a fused feature; Inputting the fused features into a decoder of the image encoding and decoding model, and decoding to obtain a first predicted image; Based on the first predicted image and the first label image, the model parameters of the image encoding and decoding model are adjusted until a training stop condition is reached, thereby obtaining the trained image encoding and decoding model.

4. The method according to claim 3, after inputting the first sample image into the encoder of the image coding and decoding model, further comprising: Obtaining a multi-scale coding feature of the first sample image; The step of inputting the fusion feature into a decoder of the image encoding and decoding model and decoding to obtain a first predicted image comprises: A decoder of the image encoding and decoding model is used to calculate multi-scale decoding features based on the multi-scale encoding features and the fusion features, and a first predicted image is decoded based on the multi-scale decoding features.

5. The method according to claim 4, wherein the decoder using the image encoding and decoding model calculates multi-scale decoding features based on the multi-scale encoding features and the fusion features, and decodes to obtain a first predicted image based on the multi-scale decoding features, comprising: Using a first decoding unit, based on the encoding features at the first scale and the fusion features, calculate the decoding features at the first scale; For any decoding unit except the first decoding unit, based on the encoding feature of the scale corresponding to the any decoding unit and the decoding feature output by the previous decoding unit, calculate the decoding feature of the scale corresponding to the any decoding unit; Until the decoding features of the target scale are calculated, the first predicted image is decoded based on the decoding features of the target scale.

6. The method according to claim 3, wherein adjusting the model parameters of the image coding model based on the first predicted image and the first label image comprises: Based on the first predicted image and the first label image, a loss value is calculated; Based on the loss value, model parameters of the encoder and the decoder are jointly adjusted.

7. The method according to claim 1, before using the image processing model to perform image feature processing on the first image feature to obtain the second image feature, further comprising: Acquire a second sample set, wherein the second sample set includes a plurality of sample pairs, and the sample pairs include a sample image and a label image; Extracting a second sample pair from the second sample set, wherein the second sample pair includes a second sample image and a second label image, and the second sample pair is any sample pair in the second sample set; Inputting the second sample image into an image processing model to obtain a second predicted image; Based on the second predicted image and the second label image, the model parameters of the image processing model are adjusted until the training stop condition is reached, thereby obtaining the trained image processing model.

8. The method according to claim 1, wherein obtaining the image to be processed comprises: Receive the video to be processed sent by the front-end user; Extracting a video frame to be processed from the video to be processed; After the second image feature is decoded by using the image encoding and decoding model to obtain a target image, the method further includes: synthesizing a target video based on the processed target video frame; The target video is fed back to the front-end user.

9. The method according to claim 8, after feeding back the target video to the front-end user, further comprises: Receiving adjustment information sent by the front-end user for the target video; Based on the adjustment information, model parameters of the image coding model are adjusted.

10. An image restoration method, comprising: Obtain the image to be repaired; Encoding the image to be repaired by using an image encoding and decoding model to obtain a first image feature; Using an image restoration model, performing image feature restoration on the first image feature to obtain a second image feature; The second image feature is decoded using the image encoding and decoding model to obtain a restored target image, wherein the image encoding and decoding model is trained based on a predicted image and a label image, the predicted image is obtained by decoding a fusion feature using the image encoding and decoding model, and the fusion feature is obtained by fusing a sample image feature of the sample image and a label image feature of the label image.

11. An image processing method, applied to a cloud-side device, comprising: An image to be processed sent by a receiving device; Encoding the image to be processed by using an image encoding and decoding model to obtain a first image feature; Using an image processing model, performing image feature processing on the first image feature to obtain a second image feature; Using the image coding and decoding model, decoding the second image feature to obtain a target image, wherein the image coding and decoding model is trained based on a predicted image and a label image, the predicted image is obtained by decoding a fusion feature using the image coding and decoding model, and the fusion feature is obtained by fusing a sample image feature of a sample image and a label image feature of the label image; The target image is fed back to the terminal device.

12. A data processing method for image processing, applied to a cloud-side device, comprising: Acquire a sample set, wherein the sample set includes a plurality of sample pairs, and the sample pairs include a sample image and a label image; Extracting a first sample pair from the sample set, wherein the first sample pair includes a first sample image and a first label image, and the first sample pair is any sample pair in the sample set; Inputting the first sample image and the first label image into an encoder of an image coding and decoding model respectively, and encoding to obtain a sample image feature of the first sample image and a label image feature of the first label image; Performing feature fusion on the sample image feature of the first sample image and the label image feature of the first label image to obtain a fused feature; Inputting the fused features into a decoder of the image encoding and decoding model, and decoding to obtain a first predicted image; Based on the first predicted image and the first label image, adjusting the model parameters of the image encoding and decoding model until a training stop condition is reached, thereby obtaining the trained image encoding and decoding model; The model parameters of the image encoding and decoding model are fed back to the terminal side device.

13. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 12 are implemented.

14. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

Images