Model training method, image editing method, device, medium and electronic equipment

Abstract

The present disclosure provides a model training method, an image editing method, an apparatus, a medium and an electronic device for image editing, relating to the technical field of artificial intelligence. The model training method comprises: processing a first sample image by using an encoder to obtain a sample original feature corresponding to the first sample image; inputting the sample original feature into an editing model to be trained, editing the sample original feature according to a target text by using the editing model to obtain a sample editing feature; processing the sample editing feature by using a decoder to obtain a sample editing image corresponding to the sample editing feature; determining a first loss function value by matching the sample editing image with the target text; and updating parameters of the editing model according to the first loss function value. The present disclosure reduces the data acquisition cost, is conducive to improving the model training effect and improving the model quality.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a model training method, image editing method, apparatus, medium and electronic device for image editing. Background Technology

[0002] Text-driven image editing refers to altering the local or global properties of an image using semantic information described in text, in order to generate a new image.

[0003] In related technologies, training machine learning models for image editing requires a large amount of image-text data pairs to establish the association between images and text. Data acquisition is costly, and the training effect of the model depends on the quality of the data annotation. Summary of the Invention

[0004] This disclosure provides a model training method for image editing, an image editing method, a model training device for image editing, an image editing device, a computer-readable storage medium, and an electronic device, to at least partially solve the problem of high data acquisition costs in related technologies.

[0005] According to a first aspect of this disclosure, a model training method for image editing is provided, comprising: processing a first sample image using an encoder to obtain sample original features corresponding to the first sample image; inputting the sample original features into an editing model to be trained, and editing the sample original features according to target text using the editing model to obtain sample editing features; processing the sample editing features using a decoder to obtain a sample edited image corresponding to the sample editing features; determining a first loss function value by matching the sample edited image with the target text; and updating the parameters of the editing model according to the first loss function value.

[0006] According to a second aspect of this disclosure, an image editing method is provided, comprising: acquiring an image to be edited; and using a pre-trained global image editing model to edit the image to be edited according to target text to obtain a target image corresponding to the image to be edited; wherein the global image editing model includes an encoder, an editing model, and a decoder, and the editing model is trained by the model training method of the first aspect described above.

[0007] According to a third aspect of this disclosure, a model training apparatus for image editing is provided, comprising: an encoding processing module configured to process a first sample image using an encoder to obtain sample original features corresponding to the first sample image; an editing processing module configured to input the sample original features into an editing model to be trained, and edit the sample original features according to target text through the editing model to obtain sample edit features; a decoding processing module configured to process the sample edit features using a decoder to obtain a sample edit image corresponding to the sample edit features; a loss function processing module configured to determine a first loss function value by matching the sample edit image with the target text; and a parameter update module configured to update the parameters of the editing model according to the first loss function value.

[0008] According to a fourth aspect of this disclosure, an image editing apparatus is provided, comprising: an image acquisition module configured to acquire an image to be edited; and an image editing module configured to use a pre-trained global image editing model to edit the image to be edited according to target text to obtain a target image corresponding to the image to be edited; wherein the global image editing model includes an encoder, an editing model, and a decoder, and the editing model is trained by the model training method of the first aspect described above.

[0009] According to a fifth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the methods of the first or second aspect described above, and possible implementations thereof.

[0010] According to a sixth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the methods of the first or second aspect described above and possible implementations thereof by executing the executable instructions.

[0011] The technical solution disclosed herein has the following beneficial effects:

[0012] On the one hand, the dataset required for model training includes a first sample image and target text. The target text, used to indicate the direction of image editing, is not the text labeled on the first sample image. Therefore, this approach eliminates the need for image-text data pairs and text annotation of images, thus reducing data acquisition costs. Furthermore, the model training effect does not depend on the quality of text annotation, which helps improve model training performance and enhance model quality. On the other hand, image editing is achieved by using an editing model to perform feature editing in the feature space. This not only generates images that conform to the attributes described in the target text but also preserves other image attributes, enabling precise image editing. Attached Figure Description

[0013] Figure 1 A flowchart illustrating a model training method for image editing in this exemplary embodiment is shown;

[0014] Figure 2 A sub-flowchart of the model training method in this exemplary embodiment is shown;

[0015] Figure 3 A schematic diagram of the model training method in this exemplary embodiment is shown;

[0016] Figure 4 A flowchart illustrating an image editing method in this exemplary embodiment is shown;

[0017] Figure 5 An example diagram illustrating image editing in this exemplary embodiment is shown;

[0018] Figure 6 A schematic diagram of the system architecture of the operating environment of this exemplary embodiment is shown;

[0019] Figure 7 This illustration shows a schematic diagram of a model training apparatus for image editing in this exemplary embodiment;

[0020] Figure 8 This diagram illustrates the structure of an image editing apparatus according to this exemplary embodiment.

[0021] Figure 9 A schematic diagram of the structure of an electronic device in this exemplary embodiment is shown. Detailed Implementation

[0022] Exemplary embodiments of this disclosure will be described more fully below with reference to the accompanying drawings.

[0023] The accompanying drawings are schematic illustrations of this disclosure and are not necessarily drawn to scale. Some block diagrams shown in the drawings may be functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in hardware modules or integrated circuits, or in networks, processors, or microcontrollers. Implementations can be carried out in various forms and should not be construed as limited to the examples set forth herein. The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough description of embodiments of this disclosure. However, those skilled in the art will recognize that one or more specific details may be omitted when implementing the technical solutions of this disclosure, or other methods, components, apparatuses, steps, etc., may be used to replace one or more specific details.

[0024] In related technologies, training machine learning models for image editing requires a large number of image-text data pairs to enable the model to learn the relationship between images and text, and then generate images that match the text descriptions during the model's inference phase. This approach places high demands on the dataset, requiring extensive text annotation of numerous images, which is challenging and leads to high data acquisition costs. Furthermore, the model's training performance depends heavily on the quality and coverage of the dataset's annotations. For example, if image information is omitted during text annotation, resulting in an incomplete relationship between images and text, the accuracy of the model's output will decrease; or, the model may struggle to generate high-quality images for text types or scenarios not covered by the dataset.

[0025] In view of one or more of the above-mentioned problems, exemplary embodiments of this disclosure provide a model training method for image editing. Figure 1 The flowchart of the model training method is shown, which may include the following steps S110 to S150:

[0026] Step S110: The first sample image is processed using an encoder to obtain the original features of the sample corresponding to the first sample image;

[0027] Step S120: Input the original features of the sample into the editing model to be trained, and edit the original features of the sample according to the target text through the editing model to obtain the edited features of the sample;

[0028] Step S130: The sample editing features are processed using a decoder to obtain the sample editing image corresponding to the sample editing features;

[0029] Step S140: Determine the value of the first loss function by matching the sample edited image with the target text;

[0030] Step S150: Update the parameters of the edited model based on the first loss function value.

[0031] Based on the above method, on the one hand, the dataset required for model training includes a first sample image and target text. The target text is used to indicate the direction of image editing and is not the text labeled on the first sample image. Therefore, this scheme does not require image-text data pairs or text annotation of images, thereby reducing data acquisition costs. Furthermore, the model training effect does not depend on the quality of text annotation, which is beneficial for improving model training performance and enhancing model quality. On the other hand, image editing is achieved by using an editing model to perform feature editing in the feature space. This not only generates images that conform to the attributes described by the target text but also maintains other image attributes unchanged, achieving precise image editing.

[0032] The following is about Figure 1Each step in the process will be explained in detail.

[0033] refer to Figure 1 In step S110, the encoder is used to process the first sample image to obtain the original features of the sample corresponding to the first sample image.

[0034] The first sample image is the original sample image used to train the model; that is, an unedited sample image. The first sample image can be an image of any type and obtained from any source. For example, the first sample image can be a face image collected from the Internet (such as a social network).

[0035] An encoder is used to extract features from an image and encode them into data in the form of tensors (such as vectors, matrices, etc.). The encoder can be implemented as a machine learning model. For example, the encoder can be a neural network model, including intermediate layers such as convolutional layers, fully connected layers (or global pooling layers), to extract features at different scales from the image and encode them into vectors. The first sample image is input into the encoder, and the encoder outputs the original sample features. Original sample features refer to the unedited features of the sample. In one implementation, the original sample features can be the original sample feature vector. The encoder used in step S110 can be an untrained encoder, a partially trained encoder, or a fully trained encoder. The higher the training level of the encoder, the more accurately the original sample features can represent the content of the first sample image.

[0036] In one implementation, the encoder can extract features from the first sample image and map them into a latent vector space to obtain the original features of the sample. In the latent vector space, information in different dimensions represents different aspects of attributes, which is beneficial for subsequent editing models to edit specific dimensions without changing information in other dimensions. That is, only the attributes related to image editing are changed, without changing other attributes, thereby improving the accuracy and efficiency of image editing.

[0037] Continue to refer to Figure 1 In step S120, the original features of the sample are input into the editing model to be trained. The original features of the sample are then edited according to the target text by the editing model to obtain the edited features of the sample.

[0038] The target text is the text that drives image editing and indicates the direction of image editing. For example, the target text could be "a girl with a smiling face," and the edited image should include the relevant image content. In this exemplary embodiment, multiple sample texts can be prepared for model training, and the target text can be any one of them.

[0039] The original sample features only include information from the first sample image. The editing model is used to edit the original sample features into edited sample features that match the target text, so that the edited sample features include information from the target text. The edited sample features and the original sample features can be data of the same form, such as both being vectors.

[0040] In one implementation, the editing model may include one or more fully connected layers capable of mapping the original features of a sample from the input feature space to the output feature space containing the edited features. The editing model may also include residual structures to enable connections between features at different levels.

[0041] In one implementation, inputting the original features of the samples into the editing model to be trained may include the following steps:

[0042] Among multiple editing models that correspond one-to-one with multiple sample texts, obtain the editing model corresponding to the target text, and input the original features of the sample into the editing model corresponding to the target text.

[0043] In other words, a corresponding editing model can be constructed for each sample text, ensuring a one-to-one correspondence between the sample text and the editing model. During model training, each editing model can be trained separately. For example, in step S120, the editing model corresponding to the target text is determined; in step S140, the determined first loss function value is applied to the editing model corresponding to the target text; and in step S150, the parameters of the editing model corresponding to the target text are updated based on the first loss function value. If it is necessary to train editing models corresponding to other sample texts, the target text is replaced with other sample texts, and steps S110 to S150 are executed again.

[0044] The editing model can have only one input channel for the original features of the input sample, eliminating the need to input the target text into the editing model. By training the editing model, it learns the information from the target text. Therefore, a relatively simple editing model structure can be set up, where each editing model only needs to learn the information of a corresponding sample text, editing the original features of the sample according to the direction indicated by the sample text, which helps to accelerate the training process.

[0045] In one implementation, an editing model can be constructed for different sample texts. This editing model can have two input channels for inputting the original features of the sample and the target text. If the input target text is different, the direction in which the editing model edits the original features of the sample will also be different, resulting in different edited features of the sample.

[0046] In one implementation, the editing model can be configured to output feature increments. For example, 'w' represents the original features of the sample, and the editing model edits 'w' according to the direction indicated by the target text, outputting feature increments. w, then w+ w represents the sample editing features. This simplifies the intermediate data processing in the editing model, reduces the value of intermediate data, and improves efficiency.

[0047] Continue to refer to Figure 1 In step S130, the sample editing features are processed using a decoder to obtain the sample editing image corresponding to the sample editing features.

[0048] The decoder is used to reconstruct the image from the features. The decoder can be implemented as a machine learning model. In one implementation, the decoder can be a generative model in a pre-trained styleGAN (style Generative Adversarial Network), styleGAN2, or styleGAN3 (styleGAN2 and styleGAN3 are improved versions of styleGAN).

[0049] The decoder used in step S130 can be an untrained decoder, a partially trained decoder, or a fully trained decoder. The higher the training level of the decoder, the better the matching between the sample edited image and the sample edited features.

[0050] By inputting the sample editing features into the decoder, the corresponding sample editing image can be output. It can be seen that in steps S110 to S130, the first sample image is finally edited into a sample editing image through the sequential processing of the encoder, editing model, and decoder. The sample editing image is the result of the editing model indirectly editing the first sample image (the editing model directly edits the original features of the sample, which is equivalent to indirectly editing the first sample image).

[0051] Continue to refer to Figure 1 In step S140, the first loss function value is determined by matching the sample edited image with the target text.

[0052] As described above, the sample edited image is the result of the editing model indirectly editing the first sample image. The training objective of the editing model can include enabling the model to accurately edit in the direction indicated by the target text; therefore, the sample edited image should include image content matching the target text. Thus, in this exemplary embodiment, a first loss function can be constructed and its value calculated based on the degree of mismatch between the sample edited image and the target text. The first loss function can be an L1 loss function, an L2 loss function, cross-entropy, etc.

[0053] In one implementation, to facilitate matching the sample edited image with the target text, the above-mentioned method of determining the first loss function value by matching the sample edited image with the sample text may include the following steps:

[0054] The sample edited image and the target text are mapped to information of the same modality. The value of the first loss function is determined by matching the mapped sample edited image and the target text.

[0055] In this context, "modality" refers to the form of information; for example, images and text are two different modalities. To match sample edited images and target text, both are first mapped to information of the same modality. For instance, the sample edited image can be mapped to text modality information for direct matching with the target text. Alternatively, the target text can be mapped to image modality information for direct matching with the sample edited image. Another approach is to map both the sample edited image and the target text into vector form for matching.

[0056] In one implementation, a matching degree can be calculated between the mapped sample edited image and the target text. If the matching degree is a value in the range of 0 to 1, then "1 - matching degree" can be used as the first loss function value. Alternatively, a cosine distance can be calculated between the mapped sample edited image and the target text, and used as the first loss function value.

[0057] In one implementation, both the mapped sample edited image and the target text can include information in multiple dimensions, each representing an attribute. The information in each dimension can be matched separately, with a supervision value of 1 set for the matching degree of information in the same dimension and a supervision value of 0 set for the matching degree of information in different dimensions. Based on the calculated L1 loss, L2 loss, cross-entropy, etc., between the matching degree and the supervision value, a first loss function value is obtained.

[0058] For example, the first loss function can be as follows:

[0059] (1)

[0060] Where Loss1 represents the first loss function, w+ w represents the sample edit feature, decode represents decoding, decode(w+ w) represents the sample edit image obtained by decoding the sample edit features, T represents the target text, and Dis represents the distance calculated between the mapped sample edit image and the target text. This distance can be the cosine distance or Euclidean distance in the mapped modality space, etc. It is evident that the larger the distance, the higher the degree of mismatch between the sample edit image and the target text, and the larger the value of the first loss function.

[0061] It should be understood that the distance in formula (1) can also be replaced with the form "1-matching degree". The matching degree can be the similarity between the mapped sample edited image and the target text, such as cosine similarity.

[0062] Continue to refer to Figure 1 In step S150, the parameters of the edited model are updated based on the first loss function value.

[0063] The first loss function value reflects the degree of mismatch between the sample edited image and the target text. The gradient for updating the editing model's parameters can be calculated based on the first loss function value, and the parameters are updated using gradient descent. By updating the editing model's parameters, the first loss function value is continuously reduced until the training completion condition is met, such as the first loss function value converging or falling below a preset loss threshold, thus completing the training of the editing model.

[0064] In one implementation, if the encoder and decoder have been sufficiently trained, the parameters of the edit model can be updated only based on the first loss function value, while keeping the parameters of the encoder and decoder unchanged.

[0065] In one implementation, if at least one of the encoder and decoder has not been sufficiently trained, the parameters of the editing model can be updated based on the first loss function value, while simultaneously updating the parameters of at least one of the encoder and decoder. For example, the encoder, editing model, and decoder can be considered as a global image editing model, and the parameters of this global image editing model can be updated based on the first loss function value, thereby enabling the training of all three models.

[0066] In one implementation, the model training method may further include the following steps:

[0067] The value of the second loss function is determined based on the amount of change between the edited features and the original features of the sample.

[0068] The value of the second loss function can be positively correlated with the amount of change. That is, the greater the change between the edited features and the original features of the sample, the larger the value of the second loss function. In other words, the second loss function value reflects the degree of editing of the sample's features by the editing model. To prevent over-editing, the degree of editing can be kept as low as possible. For example, the second loss function can be as follows:

[0069] (2)

[0070] Where Loss2 represents the second loss function, w represents the feature increment of the edited feature compared to the original feature, i.e., the amount of change mentioned above. Loss2 uses the L2 loss function, but L1 or other types of loss functions can also be used. The larger the change in the feature, the higher the degree of editing performed by the editing model, and the larger the value of the second loss function.

[0071] The above-mentioned updating and editing of the model parameters based on the first loss function value may include the following steps:

[0072] Update the parameters of the edited model based on the first and second loss function values.

[0073] In other words, besides achieving accurate editing, the training objective of the editing model can also include minimizing the degree of editing. The parameters of the editing model are updated based on the values ​​of the first and second loss functions, thus training the model towards both accurate editing and minimizing the degree of editing.

[0074] In one implementation, the first loss function value and the second loss function value can be combined, such as by addition or weighting, to obtain a comprehensive loss function value. The parameter update gradient of the editing model can then be calculated based on the comprehensive loss function value, thereby updating the parameters.

[0075] In one implementation, both the first sample image and the edited sample image can be face images. Generally, if the first sample image is a face image, then the edited sample image is also a face image. The model training method may further include the following steps:

[0076] The facial features of the first sample image and the facial features of the edited sample image are extracted. The value of the third loss function is determined by matching the facial features of the first sample image and the facial features of the edited sample image.

[0077] Facial features can be used to identify the identity of a person, including the positional relationship and distribution characteristics of facial landmarks. The facial features of the first sample image are matched with the facial features of the edited sample image. The higher the matching degree of the facial features between the two images, the closer the faces in the two images are to the same person. Editing a facial image can preserve facial features, ensuring that the person in the images before and after editing is the same person. Therefore, a third loss function can be constructed and its value calculated based on the degree of mismatch between the facial features of the first sample image and the facial features of the edited sample image.

[0078] For example, the third loss function can be as follows:

[0079] (3)

[0080] In this diagram, Loss3 represents the third loss function, S1 represents the first sample image, and R represents the extracted facial features, which can be a pre-trained facial feature model like ArcFace. Dis represents the distance calculated between the facial features of the first sample image and the facial features of the edited sample image. This distance can be the cosine distance or Euclidean distance in the facial feature space. It is evident that the larger the distance, the greater the mismatch between the facial features of the first sample image and the facial features of the edited sample image; that is, the less the faces in the two images resemble the same person, and the larger the value of the third loss function.

[0081] It should be understood that the distance in formula (3) can also be replaced with the form of "1-matching degree". The matching degree can be the similarity between the facial features of the first sample image and the facial features of the sample edit image, such as cosine similarity.

[0082] In one implementation, the model training method may further include the following steps:

[0083] The original features of the sample are processed by the decoder to obtain the first sample composite image corresponding to the first sample image. The facial features of the first sample composite image and the facial features of the sample edit image are extracted. The value of the third loss function is determined by matching the facial features of the first sample composite image and the facial features of the sample edit image.

[0084] The first sample synthesized image is generated by the decoder decoding the original features of the sample without editing by the editing model. Because it is not a real image, it is called the first sample synthesized image. The sample edited image, on the other hand, is generated by the decoder decoding the edited features of the sample after editing by the editing model. Since the editing process of the editing model is expected not to change the facial features, the faces in the first sample synthesized image and the sample edited image should be the same person. Therefore, a third loss function can be constructed and its value calculated based on the degree of mismatch between the facial features of the first sample synthesized image and the facial features of the sample edited image.

[0085] For example, the third loss function can be as follows:

[0086] (4)

[0087] Where w represents the original features of the sample, decode(w) represents the first sample synthetic image generated by the decoder after processing the original features of the sample. Formula (4) uses the distance between the face features of the first sample synthetic image and the face features of the sample edit image as the third loss function value, or the distance can be replaced with the form of "1-matching degree".

[0088] In one implementation, updating the parameters of the edited model based on the first loss function value may include the following steps:

[0089] Update the model parameters based on the first and third loss function values.

[0090] In other words, besides achieving accurate editing, the training objective of the editing model can also include: keeping the face of the person before and after editing the same person. The parameters of the editing model are updated based on the values ​​of the first and third loss functions, allowing the editing model to be trained towards the two objectives of accurate editing and maintaining the face unchanged.

[0091] In one implementation, the first loss function value and the third loss function value can be combined, such as by addition or weighting, to obtain a comprehensive loss function value. The parameter update gradient of the editing model can then be calculated based on this comprehensive loss function value, thereby updating the parameters.

[0092] In one implementation, the parameters of the edited model can be updated based on the first loss function value, the second loss function value, and the third loss function value.

[0093] For example, a comprehensive loss function can be constructed by combining the first loss function, the second loss function, and the third loss function, as shown below:

[0094] (5)

[0095] Here, LossG represents the comprehensive loss function, and λ1 and λ2 are the weights of the second and third loss functions, respectively. These weights can be determined empirically or based on specific circumstances, and are used to balance the proportions of the first, second, and third loss functions. The gradient for updating the parameters of the editing model is calculated based on the value of the comprehensive loss function, and the parameters are then updated accordingly. This achieves the following three training objectives: enabling the editing model to accurately edit the original features of the sample (indirectly the original image of the sample); minimizing the degree of editing; and ensuring that the face before and after editing remains that of the same person.

[0096] In one implementation, a comprehensive loss function value can be calculated, which may include a first loss function value and at least one of a second loss function value and a third loss function value. The parameters of the edit model are updated based on the comprehensive loss function value, and the parameters of at least one of the encoder and decoder are also updated.

[0097] In one implementation, reference Figure 2 As shown, before processing the first sample image using the encoder to obtain the original features of the sample corresponding to the first sample image, the model training method may further include the following steps S210 to S240:

[0098] Step S210: Obtain the second sample image, which is a face image.

[0099] Step S220: Input the second sample image into the generation model composed of the encoder and decoder to output the second sample synthesized image corresponding to the second sample image;

[0100] Step S230: Extract the facial features of the second sample image and the facial features of the synthesized second sample image. Determine the value of the fourth loss function by matching the facial features of the second sample image and the facial features of the synthesized second sample image.

[0101] Step S240: Update the parameters of the generated model based on the value of the fourth loss function.

[0102] The second sample image is used to train the generative model. In one implementation, the second sample image and the second sample image can come from the same dataset, so that the dataset can be reused when training the generative model and the editing model.

[0103] The generative model consists of an encoder and a decoder, excluding the editing model. Inputting a second sample image into the generative model, after encoding and decoding, generates a new image, called the synthesized second sample image. The synthesized second sample image has not undergone text editing and should be quite similar to the second sample image. Since the second sample image is a face image, and the second sample face image is also a face image, and the faces in both images should belong to the same person, facial features can be extracted from both the second sample image and the synthesized second sample image. Based on the degree of mismatch between the two facial features, a fourth loss function is constructed and its value is calculated for training the generative model.

[0104] For example, the fourth loss function can be as follows:

[0105] (6)

[0106] Where Loss4 represents the fourth loss function, S2 represents the second sample image, encode represents encoding, and decode(encode(S2)) represents the second sample composite image generated by first encoding and then decoding the second sample image. Dis represents the distance calculated between the facial features of the second sample image and the facial features of the second sample composite image. This distance can be the cosine distance or Euclidean distance in the facial feature space. It can be seen that the larger the distance, the higher the mismatch between the facial features of the second sample image and the facial features of the second sample composite image, that is, the less the faces in the two images resemble the same person, and the larger the value of the fourth loss function. Alternatively, the distance in formula (6) can be replaced with the form "1-matching degree". The matching degree can be the similarity between the facial features of the second sample image and the facial features of the second sample composite image, such as cosine similarity.

[0107] The gradient for updating the parameters of the generative model is calculated based on the value of the fourth loss function, and then the parameters are updated to train the generative model so that it can accurately encode and decode images.

[0108] Figure 2 The steps shown can be performed at any time. Figure 1 This process is performed before the editing model; that is, the generative model can be trained first to obtain a trained encoder and decoder. Therefore, in... Figure 1 In this step, the editing model can be trained only with fixed encoder and decoder parameters, which helps to accelerate the training process of the editing model.

[0109] In one implementation, when the second sample image is input into the generative model consisting of an encoder and a decoder, the decoder is a trained decoder. For example, the decoder can be pre-trained using a real face dataset, or a pre-trained StyleGAN model can be used as the decoder. Accordingly, updating the parameters of the generative model based on the fourth loss function value may include the following steps:

[0110] The decoder parameters are fixed, and the encoder parameters are updated based on the fourth loss function value.

[0111] In other words, in step S240, only the encoder parameters can be updated while keeping the decoder parameters unchanged. This is equivalent to training the decoder first, then the encoder, and finally the editing model, which helps to further reduce data acquisition costs. As the different training processes described above can reuse the same dataset, and compared to training the encoder, decoder, and editing model simultaneously, training each part separately allows each part to converge faster, improving training efficiency.

[0112] In one implementation, after training the decoder, encoder, and editing model separately, the three parts can be combined into a global image editing model. Further training can then be used to fine-tune the three parts, for example, by using any of the aforementioned comprehensive loss functions, thereby further improving the model quality.

[0113] Figure 3 The diagram illustrates the model training method. The first sample image is input into the encoder, which outputs the original sample features w. The original sample features w are then input into the editing model corresponding to the target text "a girl with a smiling face" (which can also be replaced with other target text), and the model outputs feature increments. w; Add the original features of the sample to the feature increment to obtain the sample edit feature w+ w; Edit sample features w+ The input is the decoder, and the output is the edited sample image. The first loss function value, Loss1, is calculated by matching the target text with the edited sample image; based on the feature increment... The value of w determines the second loss function value Loss2; the third loss function value Loss3 is determined based on the difference in facial identity between the first sample image and the edited sample image; the parameters of the edit model are updated based on Loss1, Loss2, and Loss3 to achieve model training.

[0114] Exemplary embodiments of this disclosure also provide an image editing method. Figure 4 The flowchart of the image editing method is shown, which may include the following steps S410 and S420:

[0115] Step S410: Obtain the image to be edited;

[0116] Step S420: Use a pre-trained global image editing model to edit the image to be edited according to the target text to obtain the target image corresponding to the image to be edited.

[0117] The image to be edited can be of any type and obtained from any source.

[0118] The global image editing model includes an encoder, an editing model, and a decoder, such as the aforementioned global image editing model. The editing model is trained using the model training method described in this exemplary embodiment, for example, by executing... Figure 1 The editing model is trained by following the steps shown.

[0119] Furthermore, the encoder or decoder in the global image editing model can be trained using the model training method in this exemplary embodiment, for example, by executing... Figure 2 The encoder is trained by following the steps shown.

[0120] The image to be edited is input into a pre-trained global image editing model, which can output a corresponding target image, which is the image after editing the image to be edited according to the target text.

[0121] Figure 5 The diagram illustrates an example of image editing. Using a global image editing model, the image to be edited can be processed according to three different target texts: "a girl with a smiling face," "a female child," and "cartoon sketch." This results in three target images with different styles, corresponding to the three target texts.

[0122] In one implementation, the editing model in the global image editing model differs for different target texts. For example, after inputting the image to be edited, if the specified target text is "a girl with a smile face," then the global image editing model consisting of an encoder, the editing model corresponding to "a girl with a smile face," and a decoder is used to process the image and output the target image. If the specified target text is "a female child," then the global image editing model consisting of an encoder, the editing model corresponding to "a female child," and a decoder is used to process the image and output the target image. Because the editing models are different, the output target images are also different.

[0123] based on Figure 4 This method can accurately edit the image to be edited based on the target text, generating a high-quality target image that contains the information of the target text and has a realistic feel.

[0124] Figure 6 The system architecture of the operating environment of this exemplary embodiment is illustrated. This system architecture 600 may include a terminal 610 and a server 620. The terminal 610 may be an electronic device such as a mobile phone, tablet computer, personal computer, smart wearable device, or drone. The server 620 generally refers to the backend system providing image editing-related services in this exemplary embodiment, and may be a single server or a cluster of multiple servers. The terminal 610 and the server 620 can be connected via a wired or wireless communication link for data interaction.

[0125] In one embodiment, the model training method of this exemplary embodiment can be executed by the server 620, the trained model can be deployed on the terminal 610, and the image editing method of this exemplary embodiment can be executed by the terminal 610. For example, the terminal 610 can use images in the local album as images to be edited, output the target image and store it.

[0126] In one embodiment, the server 620 can execute the model training method of this exemplary embodiment to obtain the trained model, and execute the image editing method of this exemplary embodiment to obtain the target image corresponding to the image to be edited.

[0127] In one embodiment, the terminal 610 can execute the model training method of this exemplary embodiment to obtain the trained model, and execute the image editing method of this exemplary embodiment to obtain the target image corresponding to the image to be edited.

[0128] As can be seen from the above, the execution subject of the model training method in this exemplary embodiment can be the aforementioned terminal 610 or server 620, and the execution subject of the image editing method can also be the aforementioned terminal 610 or server 620.

[0129] Exemplary embodiments of this disclosure also provide a model training apparatus for image editing. (Reference) Figure 7 As shown, the model training device 700 may include:

[0130] The encoding processing module 710 is configured to process the first sample image using an encoder to obtain the original sample features corresponding to the first sample image;

[0131] The editing module 720 is configured to input the original features of the sample into the editing model to be trained, and to edit the original features of the sample according to the target text through the editing model to obtain the edited features of the sample.

[0132] The decoding processing module 730 is configured to process the sample editing features using the decoder to obtain the sample editing image corresponding to the sample editing features;

[0133] The loss function processing module 740 is configured to determine a first loss function value by matching the sample edited image with the target text;

[0134] The parameter update module 750 is configured to update the parameters of the edited model based on the first loss function value.

[0135] In one implementation, the above-described method of determining the first loss function value by matching the sample edited image with the sample text includes:

[0136] The sample edited image and the target text are mapped to information of the same modality. The value of the first loss function is determined by matching the mapped sample edited image and the target text.

[0137] In one implementation, the loss function processing module 740 is further configured to:

[0138] The value of the second loss function is determined based on the amount of change between the edited features and the original features of the sample.

[0139] The parameters of the edited model updated based on the first loss function value include:

[0140] Update the parameters of the edited model based on the first and second loss function values.

[0141] In one implementation, both the first sample image and the sample edit image are face images; the loss function processing module 740 is further configured to:

[0142] The facial features of the first sample image and the facial features of the edited sample image are extracted. The third loss function value is determined by matching the facial features of the first sample image and the facial features of the edited sample image. Alternatively, the original features of the sample are processed by a decoder to obtain the first sample composite image corresponding to the first sample image. The facial features of the first sample composite image and the facial features of the edited sample image are extracted. The third loss function value is determined by matching the facial features of the first sample composite image and the facial features of the edited sample image.

[0143] The parameters of the edited model updated based on the first loss function value include:

[0144] Update the model parameters based on the first and third loss function values.

[0145] In one implementation, the above-mentioned inputting the original features of the samples into the editing model to be trained includes:

[0146] Among multiple editing models that correspond one-to-one with multiple sample texts, obtain the editing model corresponding to the target text, and input the original features of the sample into the editing model corresponding to the target text.

[0147] In one embodiment, the model training apparatus 700 may further include a model training generation module, configured to perform the following steps before the encoding processing module 710 processes the first sample image using an encoder to obtain the original sample features corresponding to the first sample image:

[0148] Obtain the second sample image, which is a face image;

[0149] The second sample image is input into the generative model consisting of an encoder and a decoder to output the second sample synthesized image corresponding to the second sample image;

[0150] The facial features of the second sample image and the facial features of the synthesized second sample image are extracted. The value of the fourth loss function is determined by matching the facial features of the second sample image and the facial features of the synthesized second sample image.

[0151] The parameters of the generated model are updated based on the value of the fourth loss function.

[0152] In one implementation, when the generative model training module inputs the second sample image into the generative model consisting of an encoder and a decoder, the decoder is a trained decoder. The above-mentioned updating of the generative model parameters based on the fourth loss function value includes:

[0153] The decoder parameters are fixed, and the encoder parameters are updated based on the fourth loss function value.

[0154] Exemplary embodiments of this disclosure also provide an image editing apparatus. (See reference...) Figure 8 As shown, the image editing device 800 includes:

[0155] Image acquisition module 810 is configured to acquire an image to be edited;

[0156] The image editing module 820 is configured to use a pre-trained global image editing model to edit the image to be edited according to the target text, so as to obtain the target image corresponding to the image to be edited.

[0157] The global image editing model includes an encoder, an editing model, and a decoder. The editing model is trained by the model training method in this exemplary embodiment.

[0158] The specific details of each part of the above-mentioned device have been described in detail in the method section of the implementation plan. For any undisclosed details, please refer to the implementation plan of the method section, and therefore will not be repeated here.

[0159] Exemplary embodiments of this disclosure also provide a computer-readable storage medium that can be implemented as a program product including program code, which, when run on an electronic device, causes the electronic device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. In an alternative embodiment, the program product can be implemented as a portable compact disc read-only memory (CD-ROM) including program code and can run on an electronic device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0160] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0161] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0162] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0163] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0164] Exemplary embodiments of this disclosure also provide an electronic device, which may be, for example, the control node described above. The electronic device may include a processor and a memory. The memory stores executable instructions for the processor, such as program code. The processor executes these executable instructions to perform the model training method or image editing method of this exemplary embodiment.

[0165] The following is for reference. Figure 9 The electronic device is illustrated by way of a general-purpose computing device. It should be understood that... Figure 9 The electronic device 900 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0166] like Figure 9 As shown, the electronic device 900 may include: a processor 910, a memory 920, a bus 930, an I / O (input / output) interface 940, and a network adapter 950.

[0167] The memory 920 may include volatile memory, such as RAM 921 and cache unit 922, and may also include non-volatile memory, such as ROM 923. The memory 920 may also include one or more program modules 924, including but not limited to: operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0168] Bus 930 is used to connect different components of electronic device 900 and may include data bus, address bus and control bus.

[0169] Electronic device 900 can communicate with one or more external devices 2000 (such as keyboard, mouse, external controller, etc.) through I / O interface 940.

[0170] Electronic device 900 can communicate with one or more networks via network adapter 950. For example, network adapter 950 can provide mobile communication solutions such as 3G / 4G / 5G, or wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication. Network adapter 950 can communicate with other modules of electronic device 900 via bus 930.

[0171] although Figure 9 Other hardware and / or software modules, including but not limited to: displays, microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, may also be configured in electronic device 900.

[0172] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0173] Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be embodied in entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.” Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0174] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is defined only by the appended claims.

Claims

1. A model training method for image editing, characterized in that, include: The first sample image is processed using an encoder to obtain the original features of the sample corresponding to the first sample image; The original features of the sample are input into the editing model to be trained. The editing model then edits the original features of the sample according to the target text to obtain the sample edit features. The target text is used to indicate the direction of image editing. The sample editing features are processed using a decoder to obtain the sample editing image corresponding to the sample editing features; The value of the first loss function is determined by matching the sample edited image with the target text; The parameters of the edited model are updated based on the first loss function value.

2. The method according to claim 1, characterized in that, The step of determining the first loss function value by matching the sample edited image with the target text includes: The sample edited image and the target text are mapped to information of the same modality, and the first loss function value is determined by matching the mapped sample edited image and the target text.

3. The method according to claim 1, characterized in that, The method further includes: The value of the second loss function is determined based on the amount of change between the edited features of the sample and the original features of the sample; The step of updating the parameters of the edited model based on the first loss function value includes: The parameters of the edited model are updated based on the first loss function value and the second loss function value.

4. The method according to claim 1, characterized in that, Both the first sample image and the sample edit image are face images; the method further includes: Extract facial features from the first sample image and the edited sample image; determine the value of the third loss function by matching the facial features of the first sample image and the edited sample image; or The decoder is used to process the original features of the sample to obtain the first sample composite image corresponding to the first sample image. The facial features of the first sample composite image and the facial features of the sample edit image are extracted. The value of the third loss function is determined by matching the facial features of the first sample composite image and the facial features of the sample edit image. The step of updating the parameters of the edited model based on the first loss function value includes: The parameters of the edited model are updated based on the first loss function value and the third loss function value.

5. The method according to claim 1, characterized in that, The step of inputting the original features of the sample into the editing model to be trained includes: Among multiple editing models that correspond one-to-one with multiple sample texts, the editing model corresponding to the target text is obtained, and the original features of the sample are input into the editing model corresponding to the target text.

6. The method according to any one of claims 1 to 5, characterized in that, Before processing the first sample image using an encoder to obtain the original features of the sample corresponding to the first sample image, the method further includes: Obtain a second sample image, which is a face image; The second sample image is input into the generation model composed of the encoder and the decoder to output the second sample synthesized image corresponding to the second sample image; The facial features of the second sample image and the facial features of the synthesized image of the second sample are extracted. The value of the fourth loss function is determined by matching the facial features of the second sample image and the facial features of the synthesized image of the second sample. The parameters of the generative model are updated based on the value of the fourth loss function.

7. The method according to claim 6, characterized in that, When the second sample image is input into the generative model composed of the encoder and the decoder, the decoder is a trained decoder; updating the parameters of the generative model according to the fourth loss function value includes: The parameters of the decoder are fixed, and the parameters of the encoder are updated according to the value of the fourth loss function.

8. An image editing method, characterized in that, include: Get the image to be edited; The image to be edited is processed according to the target text using a pre-trained global image editing model to obtain the target image corresponding to the image to be edited; The global image editing model includes an encoder, an editing model, and a decoder, wherein the editing model is trained by the model training method described in any one of claims 1 to 7.

9. A model training device for image editing, characterized in that, include: The encoding processing module is configured to process the first sample image using an encoder to obtain the original sample features corresponding to the first sample image; The editing module is configured to input the original features of the sample into the editing model to be trained, and to edit the original features of the sample according to the target text through the editing model to obtain the sample edit features; the target text is used to indicate the direction of image editing. The decoding processing module is configured to process the sample editing features using a decoder to obtain the sample editing image corresponding to the sample editing features; The loss function processing module is configured to determine a first loss function value by matching the sample edited image with the target text; The parameter update module is configured to update the parameters of the edited model based on the first loss function value.

10. An image editing device, characterized in that, include: The image acquisition module is configured to acquire the image to be edited; The image editing module is configured to use a pre-trained global image editing model to edit the image to be edited according to the target text, so as to obtain the target image corresponding to the image to be edited; The global image editing model includes an encoder, an editing model, and a decoder, wherein the editing model is trained by the model training method described in any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 8.

12. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 8 by executing the executable instructions.

Images