Image generation model training method and device, and electronic device

Abstract

The present disclosure provides a method and device for training an image generation model, and an electronic device. The method comprises: obtaining an original image and a corresponding incomplete image, inputting the incomplete image into the image generation model to correct a local area of the incomplete image, and obtaining a corrected image. After feature extraction is performed on the original image to obtain a first feature matrix and feature extraction is performed on the corrected image to obtain a second feature matrix, feature decomposition is performed on the first feature matrix to obtain first decomposed feature information and feature decomposition is performed on the second feature matrix to obtain second decomposed feature information. Then, the image generation model is trained according to the similarity between the first decomposed feature information and the second decomposed feature information, and a trained image generation model is obtained. Thus, the image generation model is trained based on the decomposed feature information, the image generation model focuses on different decomposed features, and the overall image remains coordinated.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image processing, and in particular, to a training method of an image generation model, an image processing method, a training device of an image generation model, an image processing device, an electronic device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] In the field of image processing, a generative adversarial network (GAN) is a very popular unsupervised deep learning model in recent years. In the related art, due to the instability of the Nash equilibrium point in the training process of the generative adversarial network, the image generated by the inpainting algorithm depending on the generative adversarial network has uncertainty, which destroys the coordination of the whole picture. The image generated by the inpainting algorithm not depending on the generative adversarial network often lacks texture information and is difficult to connect with the background. SUMMARY

[0003] The present disclosure provides a training method of an image generation model, an image processing method, a training device of an image generation model, an image processing device, an electronic device, a computer readable storage medium, and a computer program product, aiming to at least solve one of the technical problems in the related art to some extent.

[0004] According to a first aspect of the present disclosure, a training method of an image generation model is provided, comprising the following steps:

[0005] obtaining an original image and a corresponding incomplete image;

[0006] inputting the incomplete image into the image generation model to correct the local area missing in the incomplete image, and obtaining a corrected image;

[0007] performing feature extraction on the original image to obtain a first feature matrix, and performing feature extraction on the corrected image to obtain a second feature matrix;

[0008] performing feature decomposition on the first feature matrix to obtain first decomposition feature information, and performing feature decomposition on the second feature matrix to obtain second decomposition feature information;

[0009] training the image generation model according to the similarity between the first decomposition feature information and the second decomposition feature information, to obtain a trained image generation model.

[0010] In some possible implementation manners, the feature decomposition on the first feature matrix to obtain the first decomposition feature information, and the feature decomposition on the second feature matrix to obtain the second decomposition feature information, comprise:

[0011] performing eigen decomposition on the first feature matrix to obtain first eigen decomposition feature information, and performing eigen decomposition on the second feature matrix to obtain second eigen decomposition feature information.

[0012] In some possible implementation manners, the first eigen decomposition feature information and the second eigen decomposition feature information each include a left singular matrix, a right singular matrix, and a singular value matrix, and the training of the image generation model according to the similarity between the first eigen decomposition feature information and the second eigen decomposition feature information to obtain the trained image generation model includes:

[0013] determining a first loss term according to the similarity between the product of the left singular matrix and the singular value matrix in the first eigen decomposition feature information and the product of the left singular matrix and the singular value matrix in the second eigen decomposition feature information;

[0014] determining a second loss term according to the similarity between the product of the singular value matrix and the right singular matrix in the first eigen decomposition feature information and the product of the singular value matrix and the right singular matrix in the second eigen decomposition feature information;

[0015] weighting the first loss term and the second loss term to obtain a loss function value;

[0016] determining a model adjustment parameter of the image generation model according to the loss function value;

[0017] updating the image generation model according to the model adjustment parameter to obtain the trained image generation model.

[0018] In some possible implementation manners, the inputting of the incomplete image into the image generation model to correct the incomplete area in the incomplete image to obtain a corrected image includes:

[0019] inputting the incomplete image into an encoder of the image generation model to obtain a context feature matrix, wherein each element in the context feature matrix corresponds to at least one pixel unit in the original image and is used to indicate an association relationship between the corresponding pixel unit and a neighboring pixel unit;

[0020] inputting the context feature matrix into a decoder of the image generation model to correct the incomplete area in the incomplete image according to the association relationship to obtain a corrected image.

[0021] According to a second aspect of the present disclosure, an image processing method is provided, and the method includes:

[0022] displaying an original image;

[0023] In response to a local deletion operation on the original image, a local region of the original image is deleted to obtain a defective image;

[0024] The defective image is input into the trained image generation model obtained in the first aspect to correct the defective local region in the defective image to obtain a corrected image.

[0025] According to a third aspect of the present disclosure, a training device of an image generation model is provided, comprising:

[0026] An acquisition module is configured to acquire an original image and a corresponding defective image;

[0027] A correction module is configured to input the defective image into an image generation model to correct a defective local region in the defective image to obtain a corrected image.

[0028] An extraction module is configured to perform feature extraction on the original image to obtain a first feature matrix and perform feature extraction on the corrected image to obtain a second feature matrix.

[0029] A decomposition module is configured to perform feature decomposition on the first feature matrix to obtain first decomposition feature information and perform feature decomposition on the second feature matrix to obtain second decomposition feature information.

[0030] A training module is configured to train the image generation model according to the similarity between the first decomposition feature information and the second decomposition feature information to obtain a trained image generation model.

[0031] In some possible implementation manners, the decomposition module is configured to:

[0032] The singular value decomposition algorithm is used to perform feature decomposition on the first feature matrix to obtain first decomposition feature information and perform feature decomposition on the second feature matrix to obtain second decomposition feature information.

[0033] In some possible implementation manners, the first decomposition feature information and the second decomposition feature information each include a left singular matrix, a right singular matrix, and a singular value matrix, and the training module includes:

[0034] A first determination unit is configured to determine a first loss term according to the similarity between the product of the left singular matrix and the singular value matrix in the first decomposition feature information and the product of the left singular matrix and the singular value matrix in the second decomposition feature information.

[0035] a second determining unit, configured to determine a second loss term according to a similarity between a product of a singular value matrix and a right singular matrix in the first decomposition feature information and a product of a singular value matrix and a right singular matrix in the second decomposition feature information;

[0036] a weighting unit, configured to weight the first loss term and the second loss term to obtain a loss function value;

[0037] a third determining unit, configured to determine a model adjustment parameter of the image generation model according to the loss function value.

[0038] an updating unit, configured to update the image generation model according to the model adjustment parameter to obtain a trained image generation model.

[0039] In some possible implementation manners, the correction module comprises:

[0040] an encoding unit, configured to input the incomplete image into an encoder of the image generation model to obtain a context feature matrix; wherein each element in the context feature matrix corresponds to at least one pixel unit in the original image and is used to indicate an association relationship between the corresponding pixel unit and a neighboring pixel unit;

[0041] a decoding unit, configured to input the context feature matrix into a decoder of the image generation model to correct the incomplete region in the incomplete image according to the association relationship to obtain a corrected image.

[0042] According to a fourth aspect of the present disclosure, an image processing apparatus is provided, comprising:

[0043] a display module, configured to display an original image;

[0044] a deletion module, configured to delete a partial region of the original image to obtain an incomplete image in response to a partial deletion operation on the original image;

[0045] an input module, configured to input the incomplete image into the trained image generation model obtained according to the first aspect to correct the incomplete partial region in the incomplete image to obtain a corrected image.

[0046] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising:

[0047] at least one processor; and

[0048] a memory connected with the at least one processor in communication; wherein

[0049] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the first aspect, or perform the method of the second aspect.

[0050] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the method of the first aspect, or perform the method of the second aspect.

[0051] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method of the first aspect, or performs the method of the second aspect.

[0052] In the training method of the image generation model and the image processing method of the embodiments of the present disclosure, the image generation model is trained based on the decomposed feature information, instead of using the generative adversarial network for training, thereby breaking away from the dependence on the generative adversarial network. At the same time, the image generation model is prompted to pay attention to different decomposed features, so that the overall image remains coordinated, and the technical problem of insufficient natural connection between the generated image and the background in the related art is solved.

[0053] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0054] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure. It is clear that the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings. In the drawings:

[0055] Figure 1 A flowchart of a training method of an image generation model according to an exemplary embodiment of the present disclosure is shown;

[0056] Figure 2 A flowchart of a training method of an image generation model according to another exemplary embodiment of the present disclosure is shown;

[0057] Figure 3 A flowchart of a training method of an image generation model according to another exemplary embodiment of the present disclosure is shown;

[0058] Figure 4A flowchart of a training method of an image generation model according to another example embodiment of the present disclosure is shown in FIG. 2;

[0059] Figure 5 A training framework of an image generation model according to an example embodiment of the present disclosure is shown in FIG. 3;

[0060] Figure 6 A flowchart of an image processing method according to an example embodiment of the present disclosure is shown in FIG. 4;

[0061] Figure 7 A structure diagram of a training device of an image generation model according to an example embodiment of the present disclosure is shown in FIG. 5;

[0062] Figure 8 A structure diagram of an image processing device according to an example embodiment of the present disclosure is shown in FIG. 6; and

[0063] Figure 9 A structure diagram of an electronic device according to an example embodiment of the present disclosure is shown in FIG. 7. DETAILED DESCRIPTION

[0064] The technical solutions of the present disclosure will be described clearly and completely below with reference to the drawings, obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present disclosure. In addition, the technical features involved in the different embodiments of the present disclosure described below can be combined with each other as long as they do not conflict with each other.

[0065] The examples of the embodiments described below are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present disclosure, and cannot be understood as a limitation of the present disclosure.

[0066] In the example embodiments of the present disclosure, the training method of the image generation model described below can be implemented by a server, that is, each step of the training method of the image generation model can be performed by the server, in which case the training device of the image generation model can be configured in the server. Alternatively, the training method of the image generation model can also be implemented by a terminal device (such as a smart phone, a tablet computer, a personal computer, etc.).

[0067] In addition, the image processing method described below can be implemented by a terminal device, so that the image captured by the terminal device can be processed in real time. In this case, the image processing apparatus can be configured in the terminal device. Alternatively, the image processing method can also be implemented by a server.

[0068] That is, in the exemplary embodiments of the present disclosure, the training method of the image generation model and / or the image processing method of the present disclosure can be implemented by the terminal device and the server based on the interactive manner. Or the training method of the image generation model of the present disclosure is implemented by the server, and after the terminal device obtains the trained image generation model from the server, the terminal device can implement the image processing method of the present disclosure based on the trained image generation model.

[0069] It should be understood that the training method of the image generation model and / or the image processing method of the present disclosure can be implemented by the terminal device alone. Alternatively, the training method of the image generation model and / or the image processing method of the present disclosure can be implemented by the server alone.

[0070] The training method and apparatus of the image generation model according to the embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0071] Figure 1 A flowchart of a training method of an image generation model according to an exemplary embodiment of the present disclosure is shown.

[0072] In the related art, the image generation algorithm based on deep learning is divided into two categories, which are the repair algorithm dependent on the generative adversarial network and the repair algorithm independent of the generative adversarial network. Since the Nash equilibrium point is unstable in the training process of the generative adversarial network, it will cause the image generated by the repair algorithm dependent on the generative adversarial network to have uncertainty, which destroys the coordination of the whole picture. And the image generated by the repair algorithm independent of the generative adversarial network often lacks texture information, which is difficult to connect with the background.

[0073] To solve the above problems, the embodiments of the present disclosure provide a training method of an image generation model to train the image generation model based on decomposed feature information, rather than using the generative adversarial network for training, thereby getting rid of the dependence on the generative adversarial network. At the same time, the image generation model is prompted to pay attention to different decomposed features, so that the whole image remains coordinated, solving the technical problem that the generated image is not natural enough to connect with the background in the related art. As shown in Figure 1 The training method of the image generation model includes the following steps:

[0074] Step 101, obtaining an original image and a corresponding incomplete image.

[0075] In some possible implementation manners, the original image can be an image captured by a user using an electronic device, and the corresponding incomplete image can be an image obtained by determining a local region to be removed by the user in response to a user operation and performing a removing operation on the local region in the original image. The user operation can be a selection shape operation in an image editing operation, such as a rectangular selection, a freeform selection, or the like, and the local region to be removed by the user can be some foreign matter in the image, such as a stranger, a passing vehicle, a small animal, or the like.

[0076] The electronic device can be any device having a photographing capability, for example, a camera, a mobile terminal, or the like, and the mobile terminal can be a mobile phone, a tablet computer, or the like, which has various operating systems, touch screens, and / or display screens.

[0077] For example, when a user uses a mobile phone to take a photo outdoors, the obtained photo is the original image. Some foreign matter, such as a stranger, a passing vehicle, a small animal, or the like, can appear in the background of the photo. At this time, in order to remove the foreign matter, the user can wipe out the region to be removed on the screen of the mobile phone, thereby obtaining the corresponding incomplete image, and then the region can be automatically removed, which greatly saves the user's manpower and time.

[0078] In some possible implementation manners, the original image can be a complete image obtained from an image database, and the corresponding incomplete image can be an image obtained by deleting a local region from the original image in a random manner or according to a set rule. The local region deleted in the random manner is not fixed, and the local region deleted according to the set rule can be a region of a certain fixed position and fixed size in the original image. For example, the set rule can set the local region to be deleted from the original image as a region of 30% of the size of the original image located at the center position of the original image.

[0079] In step 102, the incomplete image is input into the image generation model to correct the incomplete local region in the incomplete image, to obtain a corrected image.

[0080] In some possible implementation manners, after receiving the incomplete image, the image generation model can predict the content shown by the incomplete local region according to the image content in the incomplete image, correct the incomplete image based on the predicted content, and obtain a corrected image.

[0081] As a possible implementation, the image generation model comprises an encoder and a decoder. The incomplete image is input into the encoder for image encoding to obtain the image encoding corresponding to the incomplete image output by the encoder, and then the image encoding is input into the decoder for image decoding, so that the decoder can predict the corrected image output by the decoder based on the image encoding. However, at this time, the image generation model is in an untrained state and has not completely learned the mapping relationship between the input original image and the output corrected image, which may cause the corrected image predicted by the image generation model to have image content defects, such as mismatch between foreground and background, or unnatural connection, and the like, and thus subsequent steps are needed to train the image generation model, so that the trained image generation model can completely learn the mapping relationship between the input original image and the output corrected image, avoiding the occurrence of image content defects in the predicted corrected image.

[0082] In step 103, feature extraction is performed on the original image to obtain a first feature matrix, and feature extraction is performed on the corrected image to obtain a second feature matrix.

[0083] In some possible implementations, an image feature extractor, such as a common convolutional neural network (CNN) such as VGG (Visual Geometry Group Network), RNN (Recurrent Neural Network), ResNet (Residual Network), etc., can be used to perform feature extraction on the original image to obtain a first feature matrix, and perform feature extraction on the corrected image to obtain a second feature matrix.

[0084] It should be noted that the feature extractor used for feature extraction on the original image and the feature extractor used for feature extraction on the corrected image can be the same, and the matrix size of the first feature matrix obtained by feature extraction on the original image and the matrix size of the second feature matrix obtained by feature extraction on the corrected image can also be the same.

[0085] In some implementations, as a possible implementation, VGG16 pre-trained on the ImageNet dataset can be used for feature extraction. The original image and the corrected image are input into the VGG16 feature extractor pre-trained on the ImageNet dataset, respectively, to obtain the feature matrix of the original image and the feature matrix of the corrected image. The feature matrix of the original image is the first feature matrix, and the feature matrix of the corrected image is the second feature matrix.

[0086] ImageNet is a large visual database project for visual object recognition software research, is the world's largest image recognition database, more than 14 million image URL (uniform resource locator) is manually annotated by ImageNet to indicate the object in the picture; in at least one million images, also provides the boundary box.

[0087] VGG16 is a convolutional neural network model, including 13 convolutional layers (Convolutional Layer), respectively, conv3-XXX, 3 fully connected layers (Fully connected Layer), respectively, FC-XXXX and 5 pool layers (Pool layer), respectively, maxpool. Among them, 13 convolutional layers and 5 pool layers are responsible for feature extraction, and 3 fully connected layers are responsible for completing the classification task. Different convolutional layers have different channel numbers, and the increase of channel number will make more information be extracted. Therefore, in this embodiment, the convolutional layer with more channel number can be selected for feature extraction, for example, the last conv3-512 convolutional layer of VGG16 can be selected for feature extraction.

[0088] Here, VGG16 pre-trained on ImageNet dataset is used for feature extraction, and since the training dataset is large enough, the obtained feature matrix has high accuracy.

[0089] Step 104, the first feature matrix is decomposed to obtain the first decomposition feature information, and the second feature matrix is decomposed to obtain the second decomposition feature information.

[0090] The elements in the first feature matrix obtained by feature extraction of the original image represent the feature vectors of multiple dimensions of the original image, and similarly, the elements in the second feature matrix obtained by feature extraction of the corrected image represent the feature vectors of multiple dimensions of the corrected image. Among them, the multiple dimension feature vectors can be understood as the feature information implied in the image, including but not limited to color features, texture features, shape features and spatial relationship features of the image.

[0091] It should be noted that dimensionality, also known as dimension, is the number of independent parameters in mathematics. For images, the dimension is the number of feature vectors in the image.

[0092] In some possible implementation, the first feature matrix can be subjected to eigen decomposition by an eigen decomposition algorithm to obtain first decomposed feature information with a dimension smaller than the first feature matrix, and the second feature matrix can be subjected to eigen decomposition to obtain second decomposed feature information with a dimension smaller than the second feature matrix, where the decomposed feature information can be in a matrix form. In the process of eigen decomposition, the high-dimensional space is mapped to a low-dimensional space. In this step, the high-dimensional first feature matrix is decomposed into a plurality of low-dimensional first decomposed feature information, so that the features affecting the image content of the original image are dispersed into the plurality of first decomposed feature information. Similarly, the high-dimensional second feature matrix is decomposed into a plurality of low-dimensional second decomposed feature information, so that the features affecting the image content of the modified image are dispersed into the plurality of second decomposed feature information.

[0093] In step 105, the image generation model is trained according to the similarity between the first decomposed feature information and the second decomposed feature information, to obtain a trained image generation model.

[0094] In some possible implementation, the loss function value can be calculated according to the similarity between the first decomposed feature information and the second decomposed feature information obtained in the previous step, and the model parameters of the image generation model can be adjusted according to the loss function value, to train the model and obtain the trained image generation model.

[0095] Since the features affecting the image content of the original image are dispersed into the plurality of first decomposed feature information, and the features affecting the image content of the modified image are dispersed into the plurality of second decomposed feature information, the similarity between the first decomposed feature information and the second decomposed feature information can reflect the detailed differences in image content between the original image and the modified image. For example, the differences can be represented by the Euclidean space distance between the decomposed feature information in the matrix form. According to the training of the image generation model, the trained image generation model can focus on the differences between the decomposed feature information. Compared with the overall comparison between the high-dimensional features in the related art, the comparison based on the decomposed features is more detailed, so that the image generation model learns more detailed mapping relationship between the texture, contour and color features of the original image and the corresponding features of the modified image, and the foreground and background of the generated modified image can be more coordinated.

[0096] It should be noted that the model can be trained by using multiple sets of data to achieve better training effect. Therefore, the image generation model should obtain multiple original images and corresponding incomplete images, so as to obtain multiple loss function values, and adjust the model parameters of the image generation model according to the loss function values for iterative training. When the loss function value converges, the model has reached a relatively ideal training effect, at which point the training of the model can be stopped. That is, the image generation model obtained at this time is the trained image generation model.

[0097] The training method of the image generation model disclosed by the present disclosure comprises the following steps: obtaining an original image and a corresponding incomplete image, inputting the incomplete image into the image generation model to correct the local area of the incomplete image, and obtaining a corrected image. After feature extraction is performed on the original image to obtain a first feature matrix and feature extraction is performed on the corrected image to obtain a second feature matrix, feature decomposition is performed on the first feature matrix to obtain first decomposition feature information, and feature decomposition is performed on the second feature matrix to obtain second decomposition feature information. Then, the image generation model is trained according to the similarity between the first decomposition feature information and the second decomposition feature information, and a trained image generation model is obtained. Thus, the image generation model is trained based on the decomposition feature information, instead of using a generative adversarial network for training, thereby breaking the dependence on the generative adversarial network. At the same time, the image generation model focuses on different decomposition features, so that the overall image remains coordinated, and the technical problem of insufficient natural connection between the generated image and the background in the related art is solved.

[0098] In some other embodiments, the present embodiment provides another possible implementation of the training method of the image generation model, Figure 2 A flowchart of a training method of an image generation model according to another example embodiment of the present disclosure is shown.

[0099] As Figure 2 shown, the method can comprise the following steps:

[0100] Step 201, obtaining an original image and a corresponding incomplete image.

[0101] It should be noted that this step can refer to step 101 in the previous embodiment for details, and the principle is the same, which will not be repeated here.

[0102] Step 202, inputting the incomplete image into the encoder of the image generation model for encoding to obtain a context feature matrix.

[0103] In some embodiments, as one possible implementation, the encoder of the image generation model comprises a plurality of convolutional layers for extracting context feature information of the image. Here, after the encoder of the image generation model receives the input incomplete image, the context feature information of the image can be extracted by convolution operation, thereby obtaining a context feature matrix, wherein each element in the context feature matrix corresponds to at least one pixel unit in the original image, and is used to indicate the association relationship between the corresponding pixel unit and the adjacent pixel unit.

[0104] Step 203, input the context feature matrix into the decoder of the image generation model for decoding, so as to correct the incomplete area in the incomplete image according to the association relationship, thereby obtaining a corrected image.

[0105] In this embodiment, the decoder of the image generation model also comprises a plurality of convolutional layers for predicting the filling area and generating the corrected image. Here, after the decoder of the image generation model receives the input context feature matrix, the incomplete area in the incomplete image can be predicted according to the association relationship indicated by the context feature matrix, thereby correcting the incomplete area in the incomplete image to obtain the corrected image.

[0106] Step 204, feature extraction is performed on the original image to obtain a first feature matrix, and feature extraction is performed on the corrected image to obtain a second feature matrix.

[0107] Step 205, feature decomposition is performed on the first feature matrix to obtain first decomposition feature information, and feature decomposition is performed on the second feature matrix to obtain second decomposition feature information.

[0108] Step 206, according to the similarity between the first decomposition feature information and the second decomposition feature information, the image generation model is trained to obtain a trained image generation model.

[0109] It should be noted that steps 204-206 can specifically refer to steps 103-105 in the previous embodiment, and the principle is the same, which will not be repeated here.

[0110] The training method of the image generation model disclosed in the present disclosure comprises the following steps: obtaining an original image and a corresponding incomplete image; inputting the incomplete image into the image generation model to correct a local area with defects in the incomplete image and obtain a corrected image; performing feature extraction on the original image to obtain a first feature matrix and performing feature extraction on the corrected image to obtain a second feature matrix; performing feature decomposition on the first feature matrix to obtain first decomposition feature information and performing feature decomposition on the second feature matrix to obtain second decomposition feature information; and training the image generation model according to the similarity between the first decomposition feature information and the second decomposition feature information to obtain a trained image generation model. Thus, the image generation model is trained based on the decomposition feature information instead of the generative adversarial network, thereby breaking the dependence on the generative adversarial network. Meanwhile, the image generation model focuses on different decomposition features, so that the overall image remains coordinated and the technical problem of insufficient natural connection between the generated image and the background in the related art is solved.

[0111] In some embodiments, the present embodiment provides another possible implementation of the training method of the image generation model, Figure 3 FIG. 1 shows a flowchart of a training method of an image generation model according to another example embodiment of the present disclosure.

[0112] In step 301, an original image and a corresponding incomplete image are obtained.

[0113] In step 302, the incomplete image is input into the image generation model to correct a local area with defects in the incomplete image and obtain a corrected image.

[0114] In step 303, feature extraction is performed on the original image to obtain a first feature matrix and feature extraction is performed on the corrected image to obtain a second feature matrix.

[0115] It should be noted that steps 301-303 can refer to steps 101-103 in the foregoing embodiments for the same principle, which will not be described here.

[0116] In step 304, a singular value decomposition algorithm is used to perform feature decomposition on the first feature matrix to obtain first decomposition feature information and perform feature decomposition on the second feature matrix to obtain second decomposition feature information.

[0117] In some embodiments, as one possible implementation, an eigenvalue decomposition algorithm can be used to perform feature decomposition on the first feature matrix to obtain first decomposition feature information and perform feature decomposition on the second feature matrix to obtain second decomposition feature information.

[0118] It should be noted that eigenvalue decomposition is to decompose a matrix A into the following form: A = Q∑Q-1 .

[0119] Wherein, Q is a matrix composed of eigenvectors of the matrix A, and ∑ is a diagonal matrix, and each diagonal element is an eigenvalue.

[0120] Since the eigenvalue decomposition algorithm can only be applied to square matrix, i.e. n x n matrix, in the embodiment of the present disclosure, the singular value decomposition (SVD) algorithm can be used to perform eigenvalue decomposition on the first eigenvalue matrix to obtain the first decomposition eigenvalue information, and perform eigenvalue decomposition on the second eigenvalue matrix to obtain the second decomposition eigenvalue information. Wherein, singular value decomposition is also a decomposition of matrix, but unlike eigenvalue decomposition, singular value decomposition does not require the matrix to be decomposed to be a square matrix, and is a decomposition algorithm applicable to any matrix. For any matrix A, there always exists a singular value decomposition: A = U∑V T .

[0121] Suppose A is an m x n matrix, then the matrix product of the matrix A and the transpose matrix A T of the matrix A will obtain an m x m square matrix AA T , and then the m eigenvalues and m eigenvectors of the square matrix AA T can be obtained, and an m x m matrix U is composed of the m eigenvectors, and each eigenvector of U is called a left singular vector. ∑ is an m x n matrix, and all elements of ∑ except the elements on the main diagonal are 0, and each element on the main diagonal is called a singular value. The matrix product of the transpose matrix A T and the matrix A will obtain an n x n matrix A T A, and then the n eigenvalues and n eigenvectors of the square matrix A T A can be obtained, and an n x n matrix V is composed of the n eigenvectors, and each eigenvector of V is called a right singular vector. Wherein, U and V are both unitary matrices, and satisfy U T U = I, V T V = I.

[0122] Step 305, training the image generation model according to the similarity between the first decomposition eigenvalue information and the second decomposition eigenvalue information, to obtain the trained image generation model.

[0123] It should be noted that the present step can refer to the step 105 in the foregoing embodiment, and the principle is the same, which will not be repeated here.

[0124] The training method of the image generation model disclosed in the present disclosure comprises the following steps: obtaining an original image and a corresponding incomplete image; inputting the incomplete image into the image generation model to correct a local area with defects in the incomplete image and obtain a corrected image; performing feature extraction on the original image to obtain a first feature matrix and performing feature extraction on the corrected image to obtain a second feature matrix; performing feature decomposition on the first feature matrix to obtain first decomposition feature information and performing feature decomposition on the second feature matrix to obtain second decomposition feature information; and training the image generation model according to the similarity between the first decomposition feature information and the second decomposition feature information to obtain a trained image generation model. Thus, the image generation model is trained based on the decomposition feature information instead of the generative adversarial network, thereby breaking the dependence on the generative adversarial network. Meanwhile, the image generation model focuses on different decomposition features, so that the overall image remains coordinated and the technical problem of insufficient natural connection between the generated image and the background in the related art is solved.

[0125] In some other embodiments, the present embodiment provides another possible implementation of the training method of the image generation model, Figure 4 FIG. 1 shows a flowchart of a training method of an image generation model according to another example embodiment of the present disclosure.

[0126] In step 401, an original image and a corresponding incomplete image are obtained.

[0127] In step 402, the incomplete image is input into the image generation model to correct a local area with defects in the incomplete image and obtain a corrected image.

[0128] In step 403, feature extraction is performed on the original image to obtain a first feature matrix and feature extraction is performed on the corrected image to obtain a second feature matrix.

[0129] It should be noted that steps 401-403 can specifically refer to steps 101-103 in the foregoing embodiments, and the principles are the same, which will not be described here again.

[0130] In step 404, a singular value decomposition algorithm is used to perform feature decomposition on the first feature matrix to obtain first decomposition feature information and perform feature decomposition on the second feature matrix to obtain second decomposition feature information.

[0131] It should be noted that this step can specifically refer to step 304 in the previous embodiment, and the principles are the same, which will not be described here again.

[0132] In step 405, a first loss term is determined according to the similarity between the product of the left singular matrix and the singular value matrix in the first decomposition feature information and the product of the left singular matrix and the singular value matrix in the second decomposition feature information.

[0133] It should be noted that the first decomposition feature information and the second decomposition feature information obtained in step 404 can include left singular matrices, right singular matrices and singular value matrices, wherein the left singular matrix in the first decomposition feature information is U, the right singular matrix is V, and the singular value matrix is ∑, and similarly, the left singular matrix in the second decomposition feature information is U', the right singular matrix is V', and the singular value matrix is ∑'.

[0134] The similarity between the product of the left singular matrix and the singular value matrix in the first decomposition feature information and the product of the left singular matrix and the singular value matrix in the second decomposition feature information, that is, the similarity between U∑ and U'∑'.

[0135] In some embodiments, as one possible implementation, the first loss term can be determined by calculating the cosine similarity of U∑ and U'∑'. The cosine similarity, also known as cosine similarity, measures the similarity between two vectors by calculating the cosine of the angle between them. Given two attribute vectors, a and b, the cosine similarity cossim(a, b) is calculated by the dot product and the vector length, and the specific calculation formula is as follows:

[0136]

[0137] where a i and b i represent the components of vectors a and b, respectively. In this embodiment, since the product of the left singular matrix and the singular value matrix is a matrix, that is, U∑ is a matrix, and similarly, U'∑' is also a matrix, in order to calculate the cosine similarity between them, the matrix needs to be converted into a vector first, and then the calculation is performed.

[0138] It should be noted that in order to ensure that the greater the cosine similarity, the smaller the value of the SVD loss function, the first loss term can be designed as 1-cossim(U∑, U'∑').

[0139] In step 406, a second loss term is determined according to the similarity between the product of the singular value matrix and the right singular matrix in the first decomposition feature information and the product of the singular value matrix and the right singular matrix in the second decomposition feature information.

[0140] The similarity between the product of the singular value matrix and the right singular matrix in the first decomposition feature information and the product of the singular value matrix and the right singular matrix in the second decomposition feature information, that is, the similarity between ∑V and ∑'V'.

[0141] In some embodiments, as a possible implementation, the second loss term can be determined by calculating the cosine similarity of ΣV and Σ'V'. The specific calculation formula can refer to the previous step, and the principle is the same, which will not be repeated here.

[0142] It should be noted that, in order to ensure that the greater the cosine similarity, the smaller the SVD loss function value, the second loss term can be designed as 1-cossim(ΣV,Σ'V').

[0143] Step 407, weighting the first loss term and the second loss term to obtain a loss function value.

[0144] In some embodiments, as a possible implementation, the loss function value can be obtained by weighting the first loss term and the second loss term. The specific calculation formula can be designed as follows:

[0145]

[0146] Wherein, L SVD represents the SVD loss function value.

[0147] It should be noted that the foregoing formula is only a possible implementation, and those skilled in the art should know that the weights of the first loss term and the second loss term can be configured according to different scenarios.

[0148] Step 408, determining the model adjustment parameter of the image generation model according to the loss function value.

[0149] In some embodiments, as a possible implementation, the model adjustment parameter of the image generation model can be determined according to the feedback of the SVD loss function value. For example, when the image generation model includes an encoder and a decoder, since the encoder and the decoder are both neural networks, and the neural network is composed of convolution, the adjustment parameters of all convolution kernels in the encoder and the decoder can be determined according to the SVD loss function value.

[0150] Step 409, updating the image generation model according to the model adjustment parameter to obtain a trained image generation model.

[0151] In some embodiments, as a possible implementation, the image generation model can be updated according to the model adjustment parameter of the image generation model obtained in step 408, so as to obtain a trained image generation model.

[0152] The training method of the image generation model disclosed in the present disclosure comprises the following steps: obtaining an original image and a corresponding incomplete image, inputting the incomplete image into the image generation model to correct the incomplete local area in the incomplete image and obtain a corrected image. After feature extraction is performed on the original image to obtain a first feature matrix and feature extraction is performed on the corrected image to obtain a second feature matrix, feature decomposition is performed on the first feature matrix to obtain first decomposition feature information and feature decomposition is performed on the second feature matrix to obtain second decomposition feature information. Then, the image generation model is trained according to the similarity between the first decomposition feature information and the second decomposition feature information to obtain a trained image generation model. Thus, the image generation model is trained based on the decomposition feature information instead of the generative adversarial network, thereby breaking the dependence on the generative adversarial network. At the same time, the image generation model focuses on different decomposition features, so that the overall image remains coordinated and the technical problem of insufficient natural connection between the generated image and the background in the related art is solved.

[0153] The training method of the image generation model according to the embodiment of the present disclosure will be described by way of example.

[0154] For example, referring to Figure 5 , Figure 5 The training framework of the image generation model according to an example embodiment of the present disclosure is shown. It should be noted that the embodiment of the present disclosure calculates the loss function value, and then trains the image generation model according to the loss function value to obtain a trained image generation model. The training framework can include an image generation model and a model loss function value calculation module. The image generation model can include an encoder and a decoder, the encoder includes a plurality of convolutional layers, and the decoder also includes a plurality of convolutional layers.

[0155] In response to a user operation, the area that the user needs to remove in the original image is determined to obtain a corresponding incomplete image. After the encoder receives the input incomplete image, the context feature matrix of the image is extracted by convolution operation, wherein each element in the context feature matrix corresponds to at least one pixel unit in the original image and is used to indicate the association relationship between the corresponding pixel unit and the adjacent pixel unit. After the decoder receives the context feature matrix, the incomplete area in the incomplete image is corrected according to the association relationship indicated by the context feature matrix to obtain a corrected image.

[0156] The original image and the modified image are respectively input into a VGG16 feature extractor pre-trained on an ImageNet dataset for feature extraction, and dimensions of original image feature maps output by the VGG16 and modified image feature maps generated by the decoder are transformed into (C, HxW), where C refers to an image color channel, H refers to an image height (a vertical side length of the image), and W refers to an image width (a horizontal side length of the image), so as to obtain a first feature matrix of the original image and a second feature matrix of the modified image. The first feature matrix of the original image is denoted as F real The second feature matrix of the modified image is denoted as F fake , and singular value decomposition is performed on the two matrices to obtain first decomposition feature information of F real and second decomposition feature information of F fake , where the first decomposition feature information and the second decomposition feature information each include a left singular matrix, a right singular matrix, and a singular value matrix, that is, the first decomposition feature information of F real includes a left singular matrix U real , a singular matrix ∑ real , and a right singular value matrix V real of F real , and the second decomposition feature information of F fake includes a left singular matrix U fake , a singular matrix ∑ fake , and a right singular matrix V fake of F fake .

[0157] Cosine similarities of U real ∑ real and U fake ∑ fake are calculated and denoted as cossim(U real ∑ real , U fake ∑ fake ). Meanwhile, cosine similarities of ∑ real V real and ∑ fake V fake are calculated and denoted as cossim(∑ real V real , ∑ fake V fake ). In order to ensure that the greater the cosine similarity is, the smaller the SVD loss function value is, the SVD loss function value L SVD is designed in the following form:

[0158]

[0159] The SVD loss function value is designed in this way, which can ensure that the greater the cosine similarity is, the smaller the SVD loss function value is, so that the image generation model is optimized in the direction where the cosine similarity between U real ∑ real and U fake ∑ fake is greater, and the cosine similarity between V real V real and V fake ∑ fake is greater. When the cosine similarity between U real ∑ real and U fake ∑ fake is greater, and the cosine similarity between V real V real and V fake ∑ fake is greater, it means that the decomposition feature information of F real is closer to the decomposition feature information of F fake , so that the generated image is more realistic.

[0160] Then, the SVD loss function value is fed back to determine the model adjustment parameters of the image generation model, and the parameters of all convolution kernels constituting the encoder and the decoder are adjusted according to the model adjustment parameters, so that the image generation model is updated to obtain a trained image generation model.

[0161] The training method of the image generation model disclosed by the present disclosure comprises the following steps: obtaining an original image and a corresponding incomplete image, inputting the incomplete image into the image generation model to correct the incomplete local area in the incomplete image and obtain a corrected image. After feature extraction is performed on the original image to obtain a first feature matrix and feature extraction is performed on the corrected image to obtain a second feature matrix, feature decomposition is performed on the first feature matrix to obtain first decomposition feature information, and feature decomposition is performed on the second feature matrix to obtain second decomposition feature information. Then, the image generation model is trained according to the similarity between the first decomposition feature information and the second decomposition feature information to obtain a trained image generation model. Thus, the image generation model is trained based on the decomposition feature information instead of the generative adversarial network, so that the dependence on the generative adversarial network is eliminated. At the same time, the image generation model focuses on different decomposition features, so that the overall image remains coordinated, and the technical problem of insufficient natural connection between the generated image and the background in the related art is solved.

[0162] Figure 6 A flowchart of an image processing method according to an exemplary embodiment of the present disclosure is shown.

[0163] Referring to Figure 6 , the image processing method according to the embodiment of the present disclosure comprises the following steps:

[0164] Step 601, display the original image.

[0165] In some embodiments, as one possible implementation, the original image can be displayed to the user in response to a user operation request, wherein the user operation request includes but is not limited to opening the image operation, and the original image represents an image that has not been processed, such as a finished photo, a scanned picture, etc.

[0166] Step 602, in response to a local deletion operation on the original image, delete the local area of the original image to obtain a defective image.

[0167] In some embodiments, as one possible implementation, the user can select a local area in the original image for deletion operation as needed, thereby obtaining a defective image. Wherein the shape of the local area can be one or a combination of circular, elliptical, rectangular, square, triangular, diamond, etc., which is not limited here.

[0168] Step 603, input the defective image into the trained image generation model to correct the defective local area in the defective image to obtain a corrected image.

[0169] In some embodiments, the image generation model is a trained image generation model obtained by any of the above image generation model training methods, which has high correction accuracy and can make the corrected image similar to the real image in color, structure, etc., and naturally connected with the background, with good overall coordination, so that the user can get the ideal corrected image without complicated post-processing.

[0170] It should be noted that the defective local area in the defective image is the local area selected by the user for deletion operation in the previous step. Here, the defective local area in the defective image is determined by the user, thereby ensuring that the corrected image obtained is the ideal image that the user can get without complicated post-processing.

[0171] The image processing method disclosed by the present disclosure displays the original image, deletes the local area of the original image in response to a local deletion operation on the original image to obtain a defective image, and then inputs the defective image into the trained image generation model to correct the defective local area in the defective image to obtain a corrected image. Since the image generation model is trained by any of the above image generation model training methods, it has high correction accuracy and can make the corrected image similar to the real image in color, structure, etc., and naturally connected with the background, with good overall coordination.

[0172] The present disclosure also proposes a training device for an image generation model.

[0173] Figure 7 FIG. 1 is a schematic diagram of a training device for an image generation model according to an example embodiment of the present disclosure.

[0174] As shown in FIG. 1, the training device for the image generation model according to the present disclosure can include an acquisition module 71, a correction module 72, an extraction module 73, a decomposition module 74, and a training module 75. Figure 7 The acquisition module 71 is configured to acquire an original image and a corresponding incomplete image.

[0175] The correction module 72 is configured to input the incomplete image into the image generation model to correct a local area with defects in the incomplete image, thereby obtaining a corrected image.

[0176] The extraction module 73 is configured to perform feature extraction on the original image to obtain a first feature matrix and perform feature extraction on the corrected image to obtain a second feature matrix.

[0177] The decomposition module 74 is configured to perform feature decomposition on the first feature matrix to obtain first decomposition feature information and perform feature decomposition on the second feature matrix to obtain second decomposition feature information.

[0178] The training module 75 is configured to train the image generation model according to a similarity between the first decomposition feature information and the second decomposition feature information, thereby obtaining a trained image generation model.

[0179] In some possible implementation manners, the correction module 72 can include:

[0180] An encoding unit configured to input the incomplete image into an encoder of the image generation model to perform encoding, thereby obtaining a context feature matrix; wherein each element in the context feature matrix corresponds to at least one pixel unit in the original image and is used to indicate a correlation between the corresponding pixel unit and a neighboring pixel unit.

[0181] A decoding unit configured to input the context feature matrix into a decoder of the image generation model to perform decoding, thereby correcting the defective area in the incomplete image according to the correlation, and obtaining a corrected image.

[0182] In some possible implementation manners, the decomposition module 74 can be configured to:

[0183] perform feature decomposition on the first feature matrix by using a singular value decomposition algorithm to obtain first decomposition feature information and perform feature decomposition on the second feature matrix by using the singular value decomposition algorithm to obtain second decomposition feature information.

[0184] In some possible implementation manners, the training module 75 can be configured to:

[0185] In some possible implementation manners, the first decomposition feature information and the second decomposition feature information each comprise a left singular matrix, a right singular matrix and a singular value matrix, and the training module 75 comprises:

[0186] a first determination unit configured to determine a first loss term according to a similarity between a product of the left singular matrix and the singular value matrix in the first decomposition feature information and a product of the left singular matrix and the singular value matrix in the second decomposition feature information;

[0187] a second determination unit configured to determine a second loss term according to a similarity between a product of the singular value matrix and the right singular matrix in the first decomposition feature information and a product of the singular value matrix and the right singular matrix in the second decomposition feature information;

[0188] a weighting unit configured to weight the first loss term and the second loss term to obtain a loss function value;

[0189] a third determination unit configured to determine a model adjustment parameter of the image generation model according to the loss function value;

[0190] an updating unit configured to update the image generation model according to the model adjustment parameter to obtain a trained image generation model.

[0191] It should be noted that the foregoing description of the training method of the image generation model also applies to the device of the present embodiment, and the principle is the same, which is not limited in the present embodiment.

[0192] The training device of the image generation model disclosed in the present disclosure obtains an original image and a corresponding incomplete image, inputs the incomplete image into the image generation model to correct a local area that is incomplete in the incomplete image, and obtains a corrected image. After feature extraction is performed on the original image to obtain a first feature matrix and feature extraction is performed on the corrected image to obtain a second feature matrix, feature decomposition is performed on the first feature matrix to obtain first decomposition feature information, and feature decomposition is performed on the second feature matrix to obtain second decomposition feature information. Then, the image generation model is trained according to a similarity between the first decomposition feature information and the second decomposition feature information, and a trained image generation model is obtained. In this way, the image generation model is trained based on the decomposition feature information instead of using a generative adversarial network, thereby breaking the dependence on the generative adversarial network. At the same time, the image generation model focuses on different decomposition features, so that the overall image remains coordinated, and the technical problem of insufficient natural connection between the generated image and the background in the related art is solved.

[0193] The present disclosure further provides an image processing device.

[0194] Figure 8FIG. 1 is a schematic diagram of an image processing device according to an example embodiment of the present disclosure.

[0195] As shown in Figure 8 , the image processing device comprises a display module 81, a deletion module 82 and an input module 83.

[0196] The display module 81 is configured to display an original image.

[0197] The deletion module 82 is configured to, in response to a local deletion operation on the original image, delete a local region of the original image to obtain a defective image.

[0198] The input module 83 is configured to input the defective image into a trained image generation model to correct the defective local region in the defective image to obtain a corrected image.

[0199] The present disclosure also provides an electronic device, which comprises at least one processor and a memory connected with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the training method of the image generation model according to any one of the above-mentioned embodiments of the present disclosure, or perform the image processing method according to any one of the above-mentioned embodiments of the present disclosure.

[0200] Figure 9 FIG. 2 is a schematic diagram of an electronic device according to an example embodiment of the present disclosure, which can implement the image generation model training method according to any one of the above-mentioned embodiments of the present disclosure. Figures 1-8 As shown in Figure 9 , the electronic device can comprise a housing 91, a processor 92, a memory 93, a circuit board 94 and a power supply circuit 95, wherein the circuit board 94 is arranged inside a space enclosed by the housing 91, and the processor 92 and the memory 93 are arranged on the circuit board 94; the power supply circuit 95 is configured to supply power to each circuit or device of the electronic device; the memory 93 is configured to store executable program codes; and the processor 92 is configured to run programs corresponding to the executable program codes by reading the executable program codes stored in the memory 93, and execute the training method of the image generation model according to any one of the above-mentioned embodiments, or execute the image processing method according to any one of the above-mentioned embodiments.

[0201] The specific execution process of the processor 92 in the above steps and the steps further executed by the processor 92 by running the executable program codes can be understood with reference to the description of the above-mentioned embodiments of the present disclosure. Figures 1-8

[0202] ​To achieve the above-mentioned embodiments, the present disclosure further provides a computer readable storage medium storing computer instructions, wherein the computer instructions are used to make the computer execute the training method of the image generation model according to any one of the above-mentioned embodiments of the present disclosure, or execute the image processing method according to any one of the above-mentioned embodiments of the present disclosure.

[0203] To achieve the above-mentioned embodiments, the present disclosure further provides a computer program product comprising a computer program, which, when executed by a processor, implements the training method of the image generation model according to any one of the above-mentioned embodiments of the present disclosure, or executes the image processing method according to any one of the above-mentioned embodiments of the present disclosure.

[0204] In the description of the present specification, the description referring to the terms “one embodiment”, “some embodiments”, “an example”, “a specific example”, or “some examples” and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples, without contradiction.

[0205] In addition, the terms “first”, “second” are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with “first”, “second” can explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of “a plurality of” is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0206] Any process or method descriptions in flow charts or described elsewhere herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions (or steps) in the process, and the various embodiments of the present disclosure include additional implementations in which the order of steps can differ from those shown or discussed, including a step can occur at similar times, can be executed in a different order, or can be executed at least partially concurrently with other steps, including steps described as sequential. It will also be understood that any process or method described herein can be implemented by a processor, such as the processor 102, executing appropriate code.

[0207] The logic and / or steps represented in flow diagrams or otherwise described herein, for example, can be considered as a sequence of instructions to implement logic functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a machine-readable storage device (e.g., magnetic, optical or other) a machine-readable storage diskette (e.g., floppy disk, optical disk, CD- ROM, etc.), a machine- readable storage card (e.g., PCMCIA card, etc.), a machine-readable storage tape (e.g., magnetic tape, optical tape, etc.), a machine-readable storage medium (e.g., RAM, ROM, etc.), a machine-readable signal (e.g., electrical, optical, etc.), a machine-readable medium (e.g., carrier wave, etc.) or any other suitable medium or means of embodying the program. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a RAM, a ROM, an EPROM, a FLASH memory card, an optical fiber, and a portable compact disc read-only memory (CD-ROM). Additionally, the computer-readable medium can be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and stored in a computer memory.

[0208] It should be understood that portions of the present disclosure can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the various steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. As such, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0209] Those skilled in the art can understand that all or part of the steps carried out by the above-mentioned embodiment methods can be completed by programs instructing related hardware, and the programs can be stored in a computer-readable storage medium. When the programs are executed, they include one of the steps of the method embodiments or a combination thereof.

[0210] In addition, each functional unit in each embodiment of the present disclosure can be integrated in one processing module, or each unit can exist physically separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0211] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A method for training an image generation model, characterized in that, The method comprises the following steps: obtaining an original image and a corresponding incomplete image; inputting the incomplete image into an image generation model to correct a local area with defects in the incomplete image, and obtaining a corrected image; performing feature extraction on the original image to obtain a first feature matrix, and performing feature extraction on the corrected image to obtain a second feature matrix; performing feature decomposition on the first feature matrix to obtain first decomposition feature information, and performing feature decomposition on the second feature matrix to obtain second decomposition feature information; training the image generation model according to the similarity between the first decomposition feature information and the second decomposition feature information, and obtaining a trained image generation model.

2. The method of claim 1, wherein, The feature decomposition on the first feature matrix to obtain the first decomposition feature information and the feature decomposition on the second feature matrix to obtain the second decomposition feature information comprise: using a singular value decomposition algorithm to perform feature decomposition on the first feature matrix to obtain the first decomposition feature information, and to perform feature decomposition on the second feature matrix to obtain the second decomposition feature information.

3. The method of claim 2, wherein, The first decomposition feature information and the second decomposition feature information each comprise a left singular matrix, a right singular matrix and a singular value matrix, and the training of the image generation model according to the similarity between the first decomposition feature information and the second decomposition feature information to obtain the trained image generation model comprises: determining a first loss term according to the similarity between the product of the left singular matrix and the singular value matrix in the first decomposition feature information and the product of the left singular matrix and the singular value matrix in the second decomposition feature information; determining a second loss term according to the similarity between the product of the singular value matrix and the right singular matrix in the first decomposition feature information and the product of the singular value matrix and the right singular matrix in the second decomposition feature information; weighting the first loss term and the second loss term to obtain a loss function value; determining a model adjustment parameter of the image generation model according to the loss function value; updating the image generation model according to the model adjustment parameter to obtain the trained image generation model.

4. The method according to any one of claims 1 to 3, characterized in that, The inputting of the incomplete image into the image generation model to correct the local area with defects in the incomplete image to obtain the corrected image comprises: inputting the incomplete image into an encoder of the image generation model to obtain a context feature matrix; wherein each element in the context feature matrix corresponds to at least one pixel unit in the original image, and is used to indicate the association relationship between the corresponding pixel unit and the adjacent pixel unit; inputting the context feature matrix into a decoder of the image generation model to correct the local area with defects in the incomplete image according to the association relationship, and obtaining the corrected image.

5. An image processing method characterized by, The method comprises: displaying an original image; in response to a local deletion operation on the original image, deleting a local area of the original image to obtain an incomplete image; inputting the incomplete image into the trained image generation model obtained by any one of claims 1-4 to correct the local area of the incomplete image to obtain a corrected image.

6. An apparatus for training an image generation model, comprising: The method comprises: an acquisition module configured to acquire an original image and a corresponding incomplete image; a correction module configured to input the incomplete image into the image generation model to correct the local area of the incomplete image to obtain a corrected image; an extraction module configured to perform feature extraction on the original image to obtain a first feature matrix and perform feature extraction on the corrected image to obtain a second feature matrix; a decomposition module configured to perform feature decomposition on the first feature matrix to obtain first decomposed feature information and perform feature decomposition on the second feature matrix to obtain second decomposed feature information; a training module configured to train the image generation model according to the similarity between the first decomposed feature information and the second decomposed feature information to obtain a trained image generation model.

7. The apparatus of claim 6, wherein, The decomposition module is configured to: perform feature decomposition on the first feature matrix to obtain first decomposed feature information and perform feature decomposition on the second feature matrix to obtain second decomposed feature information by using a singular value decomposition algorithm.

8. The apparatus of claim 7, wherein, The first decomposed feature information and the second decomposed feature information each comprise a left singular matrix, a right singular matrix and a singular value matrix, and the training module comprises: a first determination unit configured to determine a first loss term according to the similarity between the product of the left singular matrix and the singular value matrix in the first decomposed feature information and the product of the left singular matrix and the singular value matrix in the second decomposed feature information; a second determination unit configured to determine a second loss term according to the similarity between the product of the singular value matrix and the right singular matrix in the first decomposed feature information and the product of the singular value matrix and the right singular matrix in the second decomposed feature information; a weighting unit configured to weight the first loss term and the second loss term to obtain a loss function value; a third determination unit configured to determine a model adjustment parameter of the image generation model according to the loss function value; an update unit configured to update the image generation model according to the model adjustment parameter to obtain a trained image generation model.

9. The device of any of claims 6-8, wherein, The correction module comprises: an encoding unit configured to input the incomplete image into an encoder of the image generation model to obtain a context feature matrix; wherein each element in the context feature matrix corresponds to at least one pixel unit in the original image and is used to indicate the association relationship between the corresponding pixel unit and a neighboring pixel unit; a decoding unit configured to input the context feature matrix into a decoder of the image generation model to correct the incomplete area in the incomplete image according to the association relationship to obtain a corrected image.

10. An image processing apparatus characterized by comprising: The method comprises: a display module configured to display an original image; a deletion module configured to perform local deletion on the original image to obtain an incomplete image in response to a local deletion operation on the original image. An input module is configured to input the incomplete image into the trained image generation model to correct the incomplete local region in the incomplete image to obtain a corrected image. 11.An electronic device, comprising: at least one processor; and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the at least one processor to perform the method of any one of claims 1-4, or perform the method of claim 5.

12. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, the computer instructions are configured to cause the computer to perform the method of any one of claims 1-4, or perform the method of claim 5. 13.A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-4, or performs the method of claim 5.

Images