Method, device and equipment for editing image based on text and readable storage medium
By acquiring the different features of the description text of the image and performing feature mapping processing, the problem of poor generalization ability of image editing in the prior art is solved, and a more efficient image editing effect is achieved.
Patent Information
- Application Number
- CN202411181762.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-13
AI Technical Summary
When existing GAN, VAE and other models are used for image editing, the generalization ability is poor, and parameters need to be retrained and adjusted for specific topics, making it difficult to adapt to image editing of different topics.
By obtaining the original image of the image to be edited and the description text of the target image, extracting the different features of the target text, using the feature mapping module to perform feature mapping processing, generating the target mapping features, and generating the edited image through the image generation module to realize image editing.
It improves the generalization ability of image editing, reduces the need for parameter adjustment during image editing of different topics, and improves the fit and editing accuracy of the edited image with the target image.
Smart Images
Figure CN119131197A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method, apparatus, computer device and computer-readable storage medium for editing images based on text. Background Art
[0002] With the continuous development of science and technology, the requirements for image editing capabilities are also increasing. Different from the traditional image editing methods that modify image content such as color, contrast, and adding filters, deep learning technology is often combined with image editing to meet users' diverse and intelligent image editing needs. For example, when making promotional posters in financial scenarios, you can use the model learned based on artificial intelligence to modify some details of the existing posters.
[0003] In the related art, models such as GAN and VAE are used for image editing. However, the inventors of the present application found in the actual research and development process that when using models such as GAN and VAE for image editing, the models are usually trained for specific subject images. When it is necessary to edit images of different subjects, the editing quality often does not meet the requirements, and parameters need to be retrained and adjusted for the specific subject, and the image editing generalization ability is poor. Summary of the invention
[0004] The present application provides a method, apparatus, computer device and computer-readable storage medium for text-based image editing, which belongs to the field of image processing technology and can capture the difference information between an original image and a target image to edit the image to be edited, thereby improving the generalization capability of image editing.
[0005] In a first aspect, the present application provides a method for editing an image based on text, the method comprising:
[0006] Obtain a first original image description text and a first target image description text of the image to be edited;
[0007] Acquire target text difference features between the first original image description text and the first target image description text;
[0008] By using the trained feature mapping module, feature mapping processing is performed according to the target text difference feature, the encoding feature of the first target image description text and the encoding feature of the image to be edited, so as to obtain the target mapping feature of the image to be edited;
[0009] The edited image of the image to be edited is generated by a trained image generation module according to the target mapping features and the encoding features of the image to be edited.
[0010] In a second aspect, the present application provides a device for editing an image based on text, the device for editing an image based on text comprising:
[0011] A first acquisition unit, used to acquire a first original image description text and a first target image description text of the image to be edited;
[0012] A second acquisition unit, configured to acquire target text difference features between the first original image description text and the first target image description text;
[0013] A mapping unit, configured to perform feature mapping processing according to the target text difference feature, the encoding feature of the first target image description text and the encoding feature of the image to be edited through a trained feature mapping module, so as to obtain a target mapping feature of the image to be edited;
[0014] A generating unit is used to generate an edited image of the image to be edited according to the target mapping features and the encoding features of the image to be edited by using a trained image generating module.
[0015] In a third aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the text-based image editing method when executing the computer program.
[0016] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is loaded by a processor to execute the text-based image editing method.
[0017] In the present application, on the first hand, by using the first original image description text and the first target image description text of the image to be edited to extract the target text difference features, the feature mapping processing is performed according to the target text difference features, the encoding features of the first target image description text and the encoding features of the image to be edited to obtain the target mapping features; through the trained image generation module, the target mapping features and the encoding features of the image to be edited are used to generate the edited image of the image to be edited; thus, the image is edited using the text information, and since the text information of different themes can fully capture the image content to be edited, there is no need to repeatedly adjust the parameters for different themes to generate images, thereby improving the generalization ability of image editing. On the second hand, since the target mapping features extracted by the trained feature mapping module can be integrated with the target image description text information, the target image information can be fully captured to modify the image to be edited, so that the edited image of the image to be edited is more closely aligned with the actual required image. On the third aspect, since the target mapping features extracted by the trained feature mapping module can fuse the difference features between the first original image description text and the first target image description text of the image to be edited, the difference information between the original image and the target image can be fully captured to edit the image to be edited, reducing the probability of changes in the content in the image that is irrelevant to the edit description text, and improving the editing accuracy of the text-based image editing. Therefore, the present application can capture the difference information between the original image and the target image to edit the image to be edited, and improve the generalization ability of image editing. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0019] Figure 1 It is a flowchart of a method for editing an image based on text provided in an embodiment of the present application;
[0020] Figure 2 It is a schematic diagram of a framework for feature mapping module training in an embodiment of the present application;
[0021] Figure 3 This is a schematic diagram of a framework for training an image generation module in an embodiment of the present application;
[0022] Figure 4 This is a flow chart of an embodiment of feature mapping module training in an embodiment of the present application;
[0023] Figure 5 This is a schematic diagram of an embodiment of the flow chart of the image generation module training in the embodiment of the present application;
[0024] Figure 6 is a schematic diagram of the structure of an embodiment of a device for editing images based on text provided in an embodiment of the present application;
[0025] Figure 7 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0028] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.
[0029] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art can recognize that the present application can also be implemented without using these specific details. In other examples, the known process will not be elaborated in detail to avoid unnecessary details that make the description of the present application embodiment obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in accordance with the embodiments of the present application.
[0030] Embodiments of the present application provide a method, apparatus, computer device, and computer-readable storage medium for editing an image based on text.
[0031] The executor of the method based on text editing images in the embodiment of the present application may be the device based on text editing images provided in the embodiment of the present application, or the computer device provided in the embodiment of the present application, wherein the device based on text editing images may be implemented in hardware or software.
[0032] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0033] See also Figure 1 , Figure 1 1 is a flowchart of a method for editing an image based on text provided in an embodiment of the present application. The method for editing an image based on text includes steps 101 to 104, wherein:
[0034] 101. Obtain a first original image description text and a first target image description text of an image to be edited.
[0035] Among them, the image to be edited is an image that needs to be edited and modified. The method of editing images based on text in this embodiment can be applied to a variety of different business scenarios. Images in different business scenarios can be used as images to be edited. For example, when making a poster, some details of the existing poster need to be modified, and the existing poster is the image to be edited.
[0036] The first original image description text is the description text of the image to be edited.
[0037] The first target image description text is the description text of the target image of the image to be edited.
[0038] There are many ways to implement step 101, which include, for example:
[0039] (1) The user inputs two description texts respectively, one is a text describing the image to be edited, and the other is a text describing the image obtained after the image to be edited is edited. At this time, in step 101, the text describing the image to be edited input by the user can be obtained as the first original image description text of the image to be edited; the text describing the image obtained after the image to be edited input by the user is obtained as the first target image description text of the image to be edited.
[0040] (2) The user inputs a segment of editing description text, and in step 101, the editing description text input by the user is identified and decomposed to obtain a first original image description text and a first target image description text of the image to be edited. At this time, step 101 may specifically include the following steps 1011 to 1012:
[0041] 1011. Obtain the editing description text of the image to be edited.
[0042] The editing description text is a text used to describe the modified content of the image to be edited (such as the content before modification, the content after modification, the modification position, etc.).
[0043] Exemplarily, the user can input a paragraph of description text about the modified content of the image to be edited (such as the content before modification, the content after modification, the modification position, etc.) in the form of voice or text input. In step 1011, the description text about the modified content of the image to be edited input by the user is received as the editing description text of the image to be edited. For example, taking the poster production scene and the image to be edited being an existing poster as an example, if some details of the existing poster need to be modified, the user can input a paragraph of words by voice or text to describe the places in the existing poster that need to be modified, the content before modification, the content after modification, etc. of the existing poster, and the editing description text of the existing poster can be obtained by converting the paragraph of words input by the user by voice or text into text in a corresponding processing format.
[0044] 1012. Perform recognition processing on the editing description text to obtain a first original image description text and a first target image description text of the image to be edited.
[0045] Exemplarily, step 1012 may include the following steps A1 to A5:
[0046] A1. Segment processing is performed on the edit description text to obtain each description sentence of the edit description text.
[0047] In some embodiments, Python may be used to segment the edit description text into sentences to obtain the description sentences of the edit description text.
[0048] A2. Classify the description sentences to obtain the categories to which the description sentences belong, wherein the categories include original image categories, target image categories, and difference categories.
[0049] Exemplarily, first, each description sentence is embedded and represented using a text embedding model (such as a BERT model) to obtain the embedding representation features of each description sentence; then, the embedding representation features of each description sentence are used for classification to obtain the category to which each description sentence belongs. Further, in order to improve the classification accuracy of each description sentence, the embedding representation features of each description sentence and the embedding representation features of the preceding and following sentences of each description sentence can be used for classification prediction to obtain the category to which each description sentence belongs; thus, on the one hand, since the text embedding model can fully capture the semantic information of each description sentence, and the embedding representation features of the preceding and following sentences of each description sentence are integrated for classification prediction, the category to which each description sentence belongs can be fully captured by the semantic information of the preceding and following contexts, thereby improving the accuracy of dividing the first original image description text and the first target image description text, and then the ability to capture the target text difference features between the original image and the target image, and then the difference content modification can be performed on the image to be edited, and the editing accuracy of the text-based editing image can be improved to a certain extent.
[0050] For example, for the edit description text "There is an image A, which contains icon a1, font a2, and font a3 of size 2. Please change icon a1 from red to yellow, and font a2 from blue to bold red, and then output the modified image B. The size of font a3 in image B remains unchanged", first, sentence processing is performed through step A1 to obtain 6 sentences of the edit description text:
[0051] Clause 1: Now there is an image A
[0052] Sentence 2: Image A contains icon a1, font a2, and font a3 of size 2.
[0053] Sentence 3: Please change icon a1 from red to yellow
[0054] Clause 4: Change the font size of A2 from blue to bold red
[0055] Clause 5: Then output the modified image B
[0056] Clause 6: The size of font a3 in image B remains unchanged
[0057] Through classification in step A2, it can be determined that the categories of sentence 1 and sentence 2 are both original image categories; the categories of sentence 3 and sentence 4 are both difference categories; and the categories of sentence 5 and sentence 6 are both target image categories.
[0058] A3. Extract difference information for the target description sentence belonging to the difference class to obtain original image information and target image information of the target description sentence.
[0059] Specifically, the description sentence belonging to the difference category is taken as the target description sentence; then,
[0060] For ease of understanding, the example of step A2 above is continued. Since clauses 3 and 4 are difference categories, clauses 3 and 4 can be used as target description clauses respectively. At this time, difference information extraction can be performed on clause 3 to obtain the original image information of clause 3 "icon a1 is red" and the target image information of clause 3 "icon a1 is yellow". Similarly, difference information extraction is performed on clause 4 to obtain the original image information of clause 4 "font a2 is blue" and the target image information of clause 4 "font a2 is red and bold".
[0061] A4. Based on the description sentences belonging to the original image category and the original image information of the target description sentence, a first original image description text of the image to be edited is formed.
[0062] For ease of understanding, the example of step A3 above is continued. Since clauses 1 and 2 belong to the original image category, clauses 3 and 4 belong to the difference category, the original image information of clause 3 is "icon a1 is red", and the original image information of clause 4 is "font a2 is blue". Therefore, the first original image description text of the image to be edited can be "Now there is an image A, and in image A there are icon a1, font a2, and font a3 of size 2. Icon a1 is red, and font a2 is blue".
[0063] A5. Based on the description sentences belonging to the target image category and the target image information of the target description sentences, a first target image description text of the image to be edited is formed.
[0064] For ease of understanding, the example of step A3 above is continued. Since clauses 5 and 6 belong to the target image category, clauses 3 and 4 belong to the difference category, the target image information of clause 3 is "icon a1 is yellow", and the target image information of clause 4 is "font a2 is red and bold". Therefore, the first target image description text of the image to be edited can be "then output the modified image B, the size of font a3 in image B remains unchanged, icon a1 is yellow, and font a2 is red and bold".
[0065] 102. Obtain target text difference features between the first original image description text and the first target image description text.
[0066] The target text difference feature is used to indicate the difference between the encoding feature of the first original image description text and the encoding feature of the first target image description text.
[0067] For example, Figure 2As shown, the first original image description text and the first target image description text can be feature encoded by a text encoder to obtain the encoding features of the first original image description text and the encoding features of the first target image description text. Exemplarily, the text encoder can be a CLIP (Contrastive Language-Image Pre-training) text encoder, which converts the input natural language text (such as the first original image description text, or the first target image description text, etc.) into a representation vector. The vector obtained by the conversion can capture the semantics and content features of the text. Then, the difference features between the encoding features of the first original image description text and the encoding features of the first target image description text are obtained as the target text difference features.
[0068] 103. Perform feature mapping processing according to the target text difference features, the encoding features of the first target image description text and the encoding features of the image to be edited through the trained feature mapping module to obtain the target mapping features of the image to be edited.
[0069] The target mapping feature is a mapping feature obtained by performing feature mapping based on the target text difference feature, the encoding feature of the first target image description text and the encoding feature of the image to be edited through a trained feature mapping module.
[0070] Exemplarily, first, the target text difference features, the encoding features of the first target image description text, and the encoding features of the image to be edited can be input into a trained feature mapping module; then, in the trained feature mapping module, the specific feature mapping process can be as follows: first, the target text difference features, the encoding features of the first target image description text, and the encoding features of the image to be edited can be spliced together in a preset splicing order (for example, the target text difference features, the encoding features of the first target image description text, and the encoding features of the image to be edited are spliced in sequence) to obtain the target spliced features; then, mapping is performed according to the target spliced features (for example, the mapping process may include sampling, convolution and other operations) to obtain the target mapping features.
[0071] In order to better understand the embodiments of the present application, the training process of the feature mapping module is first introduced below, so as to more clearly understand how the feature mapping module in the embodiments of the present application establishes a mapping relationship between two feature spaces, so as to successfully capture the feature space mapping relationship between the image to be edited and the edited image, thereby improving the editing accuracy of the text-based image editing and reducing the probability of changes in the content in the image that is not related to the editing description text. For example, please refer to Figure 2 and Figure 4 , the training process of the feature mapping module can be as follows: steps 401 to 405:
[0072] 401. Obtain a first sample pair and a second sample pair.
[0073] Specifically, in this embodiment, the feature mapping module is trained using a training data set, wherein each sample data in the training data set includes a first sample pair and a second sample pair. For each sample data in the training data set, refer to steps 402 to 405 to obtain the constraint loss between the image difference feature and the first sample mapping feature, and train the feature mapping module.
[0074] The first sample pair includes a first image and a first text, and the second sample pair includes a second image and a second text.
[0075] The first text is a description text of the first image, and the second text is a description text of the second image.
[0076] The first image is an image that needs to be edited.
[0077] The second image is an edited image corresponding to the first image.
[0078] like Figure 2 As shown in FIG. 1 , “text 1-image 1” represents a text-image pair, and “text 2-image 2” also represents a text-image pair. In the embodiment of the present application, in order to realize the use of text to edit images, two text-image pairs are used during training: the first text-the first image (reference Figure 2 Chinese text-image pair "text 1-image 1"), second text-second image (reference Figure 2 The text-image pair "text2-image2" constructs the constraint loss of the mapping feature; the construction process of the constraint loss of the mapping feature is as follows:
[0079] ① Two texts (a first text and a second text) are respectively input into the CLIP text encoder for encoding to obtain respective encoding information, and two images (a first image and a second image) are respectively input into the CLIP image encoder for encoding to obtain respective encoding information;
[0080] For example, Figure 2 As shown, text 1 is encoded by the CLIP text encoder to obtain text encoding 1, and text 2 is encoded by the CLIP text encoder to obtain text encoding 2.
[0081] For example, Figure 2 As shown, image 1 is encoded by the CLIP image encoder to obtain image encoding 1, and image 2 is encoded by the CLIP image encoder to obtain image encoding 2.
[0082] ② The difference features of the two texts, the encoding features of the first image, and the encoding features of the first text are input into the feature mapping module, and the feature mapping module performs feature mapping to obtain mapping features;
[0083] ③ Taking the difference features of the two images as constraints, learning is performed based on the constraint loss between the difference features of the two images and the mapping features obtained in ②. As a result, the feature mapping module learns the feature space mapping relationship between the image before editing and the image after editing; through training, the feature mapping module can learn how to predict the mapping relationship between the input image and the target image, so as to generate a new image using the mapping features of the feature mapping module, and realize editing the image to be edited (such as image 1) through text (such as text 1 and text 2) to obtain the edited image (such as image 2).
[0084] For example, Figure 2 As shown, text code 1, image code 1, and the difference features between text code 1 and text code 2 are input into the feature mapping module to obtain mapping features; then, the difference features between image code 1 and image code 2 are used as constraints.
[0085] 402. Obtain a first text difference feature between the first text and the second text.
[0086] The first text difference feature is used to indicate the difference between the encoding feature of the first text and the encoding feature of the second text.
[0087] Exemplarily, the first text and the second text are respectively encoded by a text encoder to obtain a first text encoding feature of the first text and a second text encoding feature of the second text; and a difference feature between the first text encoding feature and the second text encoding feature is obtained as the first text difference feature.
[0088] For example, Figure 2 As shown, taking the case where the first text is text 1 and the second text is text 2 as an example, first, text 1 and text 2 are respectively input into the CLIP text encoder, and text 1 is encoded by the CLIP text encoder to obtain text encoding 1, and text 2 is encoded by the CLIP text encoder to obtain text encoding 2. Then, the difference feature between text encoding 1 and text encoding 2 is calculated as the first text difference feature.
[0089] 403. Perform feature mapping according to the first text difference feature, the encoding feature of the first text, and the encoding feature of the first image through a preset feature mapping module to obtain a first sample mapping feature.
[0090] The first sample mapping feature is a mapping feature obtained by performing feature mapping based on the first text difference feature, the encoding feature of the first text and the encoding feature of the first image through a preset feature mapping module.
[0091] Exemplarily, in the feature mapping module, the specific feature mapping process can be as follows: first, the first text difference feature, the encoding feature of the first text, and the encoding feature of the first image can be spliced together in a preset splicing order (for example, the first text difference feature, the encoding feature of the first text, and the encoding feature of the first image are spliced in sequence) to obtain a first spliced feature; then, mapping is performed according to the first spliced feature to obtain a first sample mapping feature.
[0092] For ease of understanding, the example of step 402 is continued. For example, Figure 2 As shown, the preset feature mapping module can first concatenate the difference features between text code 1 and text code 2, text code 1, and image code 1 in sequence to obtain a first concatenated feature; then, perform mapping based on the first concatenated feature to obtain a first sample mapping feature.
[0093] 404. Obtain image difference features between the first image and the second image.
[0094] Exemplarily, the first image and the second image are respectively encoded by an image encoder to obtain a second image coding feature of the first image and a second image coding feature of the second image; and a difference feature between the first image coding feature and the second image coding feature is obtained as the image difference feature.
[0095] The image difference feature is used to indicate the difference between the encoding feature of the first image and the encoding feature of the second image.
[0096] For example, Figure 2 As shown, taking the first image as image 1 and the second image as image 2 as an example, first, image 1 and image 2 are respectively input into the CLIP image encoder, image 1 is encoded by the CLIP image encoder to obtain image code 1, and image 2 is encoded by the CLIP image encoder to obtain image code 2. Then, the difference feature between image code 1 and image code 2 is calculated as the image difference feature.
[0097] 405 . Based on the constraint loss between the image difference feature and the first sample mapping feature, train the preset feature mapping module to obtain a trained feature mapping module.
[0098] For example, Figure 2 As shown, the image difference feature and the first sample mapping feature (reference Figure 2 The constraint loss between the "mapping features" in Figure 2 The parameters of the preset feature mapping module are adjusted until the training stop condition is met to obtain a trained feature mapping module.
[0099] 104. Generate an edited image of the image to be edited according to the target mapping features and the encoding features of the image to be edited by using a trained image generation module.
[0100] There are many ways to implement step 104, illustratively including:
[0101] (1) Generate an edited image of the image to be edited by using the target mapping features and the encoding features of the image to be edited.
[0102] Specifically, the target mapping features and the coding features of the image to be edited can be input into a preset image generation module; then, through the preset image generation module, Gaussian noise is first added to the coding features of the image to be edited to obtain the noise coding map of the image to be edited; then, according to the target mapping features and the random time step, the noise coding map of the image to be edited is denoised to obtain the denoised coding map of the coding map image to be edited as the edited coding map of the image to be edited; finally, the edited coding map of the image to be edited is decoded by a decoder to obtain the edited image of the image to be edited. In each time step t, Gaussian noise is first added to the coding features of the image to be edited to obtain the noise coding map of the image to be edited at time step t; then, the noise z of the image to be edited at time step t is predicted by using the target mapping features, the noise coding map of the image to be edited at time step t and the coding features of time step t. t , and then remove the noise z at time step t from the noise coding map of the image to be edited at time step t t , the denoised coded image of the image to be edited at the time step t is obtained; and so on, the denoised coded image of the image to be edited at the last time step t is finally obtained, which is used as the edited coded image of the image to be edited, and the edited coded image of the image to be edited is decoded by the decoder to obtain the edited image of the image to be edited.
[0103] It can be seen that, through the trained image generation module, the target mapping features and the encoding features of the image to be edited can be used to generate the edited image of the image to be edited; therefore, on the first hand, since the target mapping features extracted by the trained feature mapping module can fuse the difference features between the first original image description text and the first target image description text of the image to be edited, the difference information between the original image and the target image can be fully captured to modify the details of the image to be edited, reduce the probability of changes in the content irrelevant to the editing description text in the image, and improve the editing accuracy of the text-based editing image. On the second hand, since the target mapping features extracted by the trained feature mapping module can fuse the target image description text information, the target image information can be fully captured to modify the image to be edited, so that the edited image of the image to be edited is more closely aligned with the actual required image.
[0104] (2) Generate an edited image of the image to be edited using the target mapping features, the coding features of the image to be edited, and the coding features of the coding description text. In this case, step 104 may specifically include: obtaining the coding features of the editing description text; and generating an edited image of the image to be edited according to the target mapping features, the coding features of the image to be edited, and the coding features of the editing description text through a trained image generation module.
[0105] Specifically, the target mapping features, the coding features of the image to be edited, and the coding features of the editing description text can be input into a preset image generation module; then, through the preset image generation module, Gaussian noise is first added to the coding features of the image to be edited to obtain the noise coding map of the image to be edited; then, according to the coding features of the editing description text, the target mapping features and the random time step, the noise coding map of the image to be edited is denoised to obtain the denoised coding map of the coding map image to be edited as the edited coding map of the image to be edited; finally, the edited coding map of the image to be edited is decoded by a decoder to obtain the edited image of the image to be edited. In each time step t, Gaussian noise is first added to the coding features of the image to be edited to obtain the noise coding map of the image to be edited at time step t; then, the noise z of the image to be edited at time step t is predicted by using the coding features of the editing description text, the target mapping features, the noise coding map of the image to be edited at time step t and the coding features of time step t. t , and then remove the noise z at time step t from the noise coding map of the image to be edited at time step t t, and obtain the denoised coded image of the image to be edited at the time step t; and so on, finally obtain the denoised coded image of the image to be edited at the last time step t as the edited coded image of the image to be edited, and decode the edited coded image of the image to be edited by the decoder to obtain the edited image of the image to be edited. Thus, the edited image of the image to be edited can be generated in combination with the coding features of the edit description text. Since the edit description text can more comprehensively describe the difference information between the original image and the target image, the difference modification can be more targeted on the image to be edited, reducing the probability of changes in the content in the image that is not related to the edit description text, and improving the editing accuracy of the text-based image editing.
[0106] In order to better understand the embodiments of the present application, the training process of the image generation module is first introduced below. For example, please refer to Figure 3 and Figure 5 , the training process of the image generation module can be as follows: steps 501 to 505:
[0107] 501. Obtain a second original image description text and a second target image description text of a sample image.
[0108] The second original image description text is the description text of the image to be edited.
[0109] The second target image description text is the description text of the target image of the image to be edited.
[0110] The method of obtaining the second original image description text and the second target image description text of the sample image in step 501 is similar to the method of obtaining the first original image description text and the first target image description text of the image to be edited in step 101. For details, please refer to the relevant instructions of step 101, which will not be repeated here.
[0111] 502. Obtain a second text difference feature between the second original image description text and the second target image description text.
[0112] The second text difference feature is used to indicate the difference between the encoding feature of the second original image description text and the encoding feature of the second target image description text.
[0113] For example, Figure 3 As shown, first, the second original image can be described by the text encoder (refer to Figure 3 Text in 1), second target graph description text (reference Figure 3 2) to perform feature encoding to obtain the encoding features of the second original image description text (reference Figure 3 The second target graph describes the encoding features of the text (reference Figure 3Then, the difference feature between the encoding feature of the second original image description text and the encoding feature of the second target image description text is obtained as the second text difference feature.
[0114] 503. Perform feature mapping according to the second text difference feature, the encoding feature of the second target image description text, and the encoding feature of the sample image through the trained feature mapping module to obtain a second sample mapping feature.
[0115] The second sample mapping feature is a mapping feature obtained by performing feature mapping based on the second text difference feature, the encoding feature of the second target image description text and the encoding feature of the sample image through a preset feature mapping module.
[0116] For example, Figure 3 As shown, the second text difference feature and the second target graph describe the encoding features of the text (reference Figure 3 The text encoding in 1) and the encoding features of the sample image (reference Figure 3 The “image encoding” in the image description text) is input into the trained feature mapping module; then, through the trained feature mapping module, the second text difference feature, the encoding feature of the second target image description text and the encoding feature of the sample image are first spliced together in a preset splicing order (for example, the second text difference feature, the encoding feature of the second target image description text and the encoding feature of the sample image are spliced in sequence) to obtain a second spliced feature; then, mapping is performed according to the second spliced feature to obtain a second sample mapping feature.
[0117] In this embodiment, model training is divided into two stages. In the first stage, the text and image encoding respectively use the pre-trained CLIP text encoder and the pre-trained CLIP image encoder. When training is performed in steps 401 to 405, the parameters of the pre-trained CLIP text encoder and the pre-trained CLIP image encoder are frozen, and the feature mapping module is trained in a targeted manner to improve the model training speed. In the second stage, the feature mapping module trained in steps 401 to 405 is used to input the predicted mapping features and the input image together into the image generation module to generate the edited image, and the generated edited image is constrained by the target image as the true value.
[0118] 504. Prediction is performed according to the second sample mapping feature and the coding feature of the sample image through a preset image generation module to obtain a predicted edited coding image of the sample image.
[0119] In some embodiments, a diffusion model may be used as an image generation module. For example, a stable diffusion model may be used as an image generation module.
[0120] In some embodiments, in order to improve the accuracy of text-to-image editing and reduce the training time, a Lora layer is added to the pre-trained diffusion model as an image generation module. During the training of the image generation module, all parameters other than the Lora layer of the image generation module are frozen, and the Lora layer is trained. Therefore, since the low-rank adaptation (Low-Rank Adaptation, Lora) layer can add an address matrix to the weight matrix of the pre-trained model, it can be updated during the fine-tuning process to learn the text editing description information. During the fine-tuning process, the low-rank adaptation layer is updated to minimize the loss of the text-based editing image. The Lora layer achieves more effective fine-tuning by focusing the adaptation process on the low-rank matrix. Compared with the comprehensive fine-tuning of the entire model, it can reduce the computational and memory overhead, thereby improving the training efficiency of the image generation model and reducing resource loss.
[0121] Specifically, Figure 3 As shown, the second sample mapping feature (reference Figure 3 The "mapping features" in ) and the encoded features of the sample image (reference Figure 3 The sample image is firstly encoded with Gaussian noise, and then, according to the second sample mapping feature and the random time step, the noise encoding image of the sample image is denoised to obtain a denoised encoding image, which is used as the predicted edited encoding image of the sample image (reference Figure 3 In each time step t, Gaussian noise is first added to the coding features of the sample image to obtain the noise coding image c of the sample image at time step t. θ (y); Then, using the second sample mapping feature, the noise coding map c of the sample image at time step t θ (y) and the encoding features of time step t, predict the noise z of the sample image at time step t t , and then the noise coding map c of the sample image at time step t θ (y) Remove the noise z at time step t t , get the denoised encoding image ε of the sample image at time step t θ .
[0122] 505. Based on the loss between the predicted edited coded image of the sample image and the actual edited coded image of the sample image, train the preset image generation module to obtain a trained image generation module.
[0123] For example, Figure 3 As shown, first, the prediction loss function of the reverse diffusion process of the image generation module is constructed (refer to Figure 3 The “LOSS” in the image generation module, for example, the prediction loss function of the reverse diffusion process of the image generation module can be shown in the following formula 1; then, according to the prediction loss function, the predicted edited coding image of the sample image is calculated (reference Figure 3 ) and the actual edited coded image of the sample image (ref. Figure 3 The loss between the predicted edited coded image of the sample image and the actual edited coded image of the sample image is used to adjust the Lora layer parameters in the preset image generation module to obtain a trained image generation module.
[0124] For example, the prediction loss function can be shown as the following formula 1:
[0125]
[0126] In formula 1, t is the time step, z t is the noise at time t, ε is the scale-free noise sample, and ε θ is a denoising network, used to remove the noise z at time t t At inference time, the random noise tensor is sampled and the noise is removed to produce a new latent image, which is then converted to an image for output via a decoder.
[0127] From the above content, it can be seen that, on the one hand, by using the first original image description text and the first target image description text of the image to be edited to extract the target text difference features, through the trained image generation module, using the target mapping features and the encoding features of the image to be edited, the edited image of the image to be edited is generated; thus, the image is edited using text information, because the text information of different themes can fully capture the image content to be edited, there is no need to repeatedly adjust the parameters for different themes to generate images, thereby improving the generalization ability of image editing. On the other hand, since the target mapping features extracted by the trained feature mapping module can be integrated with the target image description text information, the target image information can be fully captured to modify the image to be edited, so that the edited image of the image to be edited is more in line with the actual required image. On the third hand, since the target mapping features extracted by the trained feature mapping module can be integrated with the difference features between the first original image description text and the first target image description text of the image to be edited, the difference information between the original image and the target image can be fully captured to edit the image to be edited, reducing the probability of changes in the content irrelevant to the edit description text in the image, and improving the editing accuracy of the text-based image editing. Therefore, the present application can capture the difference information between the original image and the target image to edit the image, thereby improving the generalization ability of image editing.
[0128] In addition, in order to better implement the method based on text editing image in the embodiment of the present application, on the basis of the method based on text editing image, the embodiment of the present application also provides a device based on text editing image, such as Figure 6 FIG. 1 is a schematic diagram of a structure of an embodiment of a device for editing text images provided in an embodiment of the present application. The device 600 for editing text images includes:
[0129] A first acquisition unit 601 is used to acquire a first original image description text and a first target image description text of the image to be edited;
[0130] The second acquisition unit 602 is used to acquire target text difference features between the first original image description text and the first target image description text;
[0131] A mapping unit 603 is used to perform feature mapping processing according to the target text difference feature, the encoding feature of the first target image description text and the encoding feature of the image to be edited through a trained feature mapping module to obtain a target mapping feature of the image to be edited;
[0132] The generating unit 604 is configured to generate an edited image of the image to be edited according to the target mapping features and the encoding features of the image to be edited by using a trained image generating module.
[0133] In some embodiments, the apparatus for editing images based on text further includes a training unit (not shown in the figure), wherein the training unit is specifically used to:
[0134] Acquire a first sample pair and a second sample pair, wherein the first sample pair includes a first image and a first text, and the second sample pair includes a second image and a second text;
[0135] Acquire a first text difference feature between the first text and the second text;
[0136] By using a preset feature mapping module, feature mapping is performed according to the first text difference feature, the encoding feature of the first text, and the encoding feature of the first image to obtain a first sample mapping feature;
[0137] Acquire image difference features between the first image and the second image;
[0138] Based on the constraint loss between the image difference feature and the first sample mapping feature, the preset feature mapping module is trained to obtain a trained feature mapping module.
[0139] In some embodiments, the training unit is specifically used for:
[0140] Encode the first text and the second text respectively by a text encoder to obtain a first text encoding feature of the first text and a second text encoding feature of the second text;
[0141] A difference feature between the first text encoding feature and the second text encoding feature is obtained as the first text difference feature.
[0142] In some embodiments, the training unit is specifically used for:
[0143] Encode the first image and the second image respectively by using an image encoder to obtain a second image coding feature of the first image and a second image coding feature of the second image;
[0144] A difference feature between the first image coding feature and the second image coding feature is obtained as the image difference feature.
[0145] In some embodiments, the training unit is specifically used for:
[0146] Obtain a second original image description text and a second target image description text of the sample image;
[0147] Acquire a second text difference feature between the second original image description text and the second target image description text;
[0148] By using the trained feature mapping module, feature mapping is performed according to the second text difference feature, the encoding feature of the second target image description text, and the encoding feature of the sample image to obtain a second sample mapping feature;
[0149] By using a preset image generation module, prediction is performed according to the second sample mapping feature and the coding feature of the sample image to obtain a predicted edited coding image of the sample image;
[0150] Based on the loss between the predicted edited coded image of the sample image and the actual edited coded image of the sample image, the preset image generation module is trained to obtain a trained image generation module.
[0151] In some embodiments, the first acquiring unit 601 is specifically used to:
[0152] Get the editing description text of the image to be edited;
[0153] Performing recognition processing on the editing description text to obtain a first original image description text and a first target image description text of the image to be edited;
[0154] The step of generating an edited image of the image to be edited by the trained image generation module according to the target mapping feature and the encoding feature of the image to be edited comprises:
[0155] Obtaining the encoding features of the edit description text;
[0156] The edited image of the image to be edited is generated by a trained image generation module according to the target mapping features, the encoding features of the image to be edited, and the encoding features of the editing description text.
[0157] In some embodiments, the first acquiring unit 601 is specifically used to:
[0158] Performing sentence processing on the edit description text to obtain each description sentence of the edit description text;
[0159] Classifying the description sentences to obtain categories to which the description sentences belong, wherein the categories include original image categories, target image categories, and difference categories;
[0160] Extracting difference information for the target description sentence belonging to the difference class to obtain original image information and target image information of the target description sentence;
[0161] Based on the description sentences belonging to the original image category and the original image information of the target description sentence, forming a first original image description text of the image to be edited;
[0162] Based on the description sentences belonging to the target image category and the target image information of the target description sentences, a first target image description text of the image to be edited is formed.
[0163] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous text-based image editing method embodiment, which will not be repeated here.
[0164] See also Figure 7 , Figure 7 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a server.
[0165] like Figure 7 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.
[0166] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any method for editing an image based on text.
[0167] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0168] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any method for editing an image based on text.
[0169] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0170] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0171] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:
[0172] Obtain a first original image description text and a first target image description text of an image to be edited; obtain target text difference features between the first original image description text and the first target image description text; perform feature mapping processing according to the target text difference features, the encoding features of the first target image description text and the encoding features of the image to be edited through a trained feature mapping module to obtain target mapping features of the image to be edited; generate an edited image of the image to be edited according to the target mapping features and the encoding features of the image to be edited through a trained image generation module.
[0173] A person of ordinary skill in the art will appreciate that all or part of the steps in the above-mentioned method for text-based image editing may be accomplished through instructions, or through instruction-controlled related hardware, and the instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0174] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored. The computer programs can be loaded by a processor to execute any one of the text-based image editing methods provided in the embodiments of the present application.
[0175] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0176] In the above-mentioned text editing image-based device and computer-readable storage medium embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-described text editing image-based device, computer-readable storage medium and corresponding units can refer to the description of the text editing image-based method in the above embodiment, and will not be repeated here.
[0177] The above is a detailed introduction to a method, device, computer equipment and computer-readable storage medium based on text editing images provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for editing an image based on text, characterized in that: The method comprises: Obtain a first original image description text and a first target image description text of the image to be edited; Acquire target text difference features between the first original image description text and the first target image description text; By using the trained feature mapping module, feature mapping processing is performed according to the target text difference feature, the encoding feature of the first target image description text and the encoding feature of the image to be edited, so as to obtain the target mapping feature of the image to be edited; The edited image of the image to be edited is generated by a trained image generation module according to the target mapping features and the encoding features of the image to be edited.
2. The method for editing an image based on text according to claim 1, characterized in that: The training process of the trained feature mapping module is as follows: Acquire a first sample pair and a second sample pair, wherein the first sample pair includes a first image and a first text, and the second sample pair includes a second image and a second text; Acquire a first text difference feature between the first text and the second text; By using a preset feature mapping module, feature mapping is performed according to the first text difference feature, the encoding feature of the first text, and the encoding feature of the first image to obtain a first sample mapping feature; Acquire image difference features between the first image and the second image; Based on the constraint loss between the image difference feature and the first sample mapping feature, the preset feature mapping module is trained to obtain a trained feature mapping module.
3. The method for editing an image based on text according to claim 2, characterized in that: The obtaining of a first text difference feature between the first text and the second text includes: Encode the first text and the second text respectively by a text encoder to obtain a first text encoding feature of the first text and a second text encoding feature of the second text; A difference feature between the first text encoding feature and the second text encoding feature is obtained as the first text difference feature.
4. The method for editing an image based on text according to claim 2, characterized in that: The acquiring the image difference feature between the first image and the second image includes: Encode the first image and the second image respectively by using an image encoder to obtain a second image coding feature of the first image and a second image coding feature of the second image; A difference feature between the first image coding feature and the second image coding feature is obtained as the image difference feature.
5. The method for editing an image based on text according to claim 1, characterized in that: The training process of the trained image generation module is as follows: Obtain a second original image description text and a second target image description text of the sample image; Acquire a second text difference feature between the second original image description text and the second target image description text; By using the trained feature mapping module, feature mapping is performed according to the second text difference feature, the encoding feature of the second target image description text, and the encoding feature of the sample image to obtain a second sample mapping feature; By using a preset image generation module, prediction is performed according to the second sample mapping feature and the coding feature of the sample image to obtain a predicted edited coding image of the sample image; Based on the loss between the predicted edited coded image of the sample image and the actual edited coded image of the sample image, the preset image generation module is trained to obtain a trained image generation module.
6. The method for editing an image based on text according to claim 1, characterized in that: The step of obtaining a first original image description text and a first target image description text of the image to be edited includes: Get the editing description text of the image to be edited; Performing recognition processing on the editing description text to obtain a first original image description text and a first target image description text of the image to be edited; The step of generating an edited image of the image to be edited by the trained image generation module according to the target mapping feature and the encoding feature of the image to be edited comprises: Obtaining the encoding features of the edit description text; The edited image of the image to be edited is generated by a trained image generation module according to the target mapping features, the encoding features of the image to be edited, and the encoding features of the editing description text.
7. The method for editing an image based on text according to claim 6, characterized in that: The step of performing recognition processing on the edit description text to obtain a first original image description text and a first target image description text of the image to be edited includes: Performing sentence processing on the edit description text to obtain each description sentence of the edit description text; Classifying the description sentences to obtain categories to which the description sentences belong, wherein the categories include original image categories, target image categories, and difference categories; Extracting difference information for the target description sentence belonging to the difference class to obtain original image information and target image information of the target description sentence; Based on the description sentences belonging to the original image category and the original image information of the target description sentence, forming a first original image description text of the image to be edited; Based on the description sentences belonging to the target image category and the target image information of the target description sentences, a first target image description text of the image to be edited is formed.
8. A device for editing images based on text, characterized in that: The device for editing an image based on text comprises: A first acquisition unit, used to acquire a first original image description text and a first target image description text of the image to be edited; A second acquisition unit, configured to acquire target text difference features between the first original image description text and the first target image description text; A mapping unit, configured to perform feature mapping processing according to the target text difference feature, the encoding feature of the first target image description text and the encoding feature of the image to be edited through a trained feature mapping module, so as to obtain a target mapping feature of the image to be edited; A generating unit is used to generate an edited image of the image to be edited according to the target mapping features and the encoding features of the image to be edited by using a trained image generating module.
9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the method for editing an image based on text according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the method for editing an image based on text as claimed in any one of claims 1 to 7.