Image editing method, device, equipment, storage medium and program product
By introducing pixel-level text rendering technology and a diffusion model, the problem of poor image quality in text editing scenarios in existing technologies has been solved, and high-quality image editing has been achieved in complex text editing scenarios such as Chinese text.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA MOBILE JIUTIAN ARTIFICIAL INTELLIGENCE TECHNOLOGY (BEIJING) CO LTD
- Filing Date
- 2026-02-03
- Publication Date
- 2026-06-05
AI Technical Summary
Existing image editing methods perform poorly in text editing scenarios, making it difficult to guarantee the quality of image editing, especially in complex text editing scenarios such as Chinese. The poor image editing effect of the model can easily lead to errors in the strokes or structure of the generated text.
Pixel-level text rendering technology is introduced. A large language model is used to perform semantic understanding and operation intent classification of text prompts, generating a text rendering image of the text to be edited. The text rendering image, the image to be edited, and the text prompts are then input into a pre-trained image editing model for editing. A diffusion model is used for pixel-level image editing.
This improved the model's performance in text editing scenarios, generated accurate images, and ensured the quality of image editing.
Smart Images
Figure CN122156362A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to an image editing method, apparatus, device, storage medium, and program product. Background Technology
[0002] Instructional image editing refers to the process where users directly describe their editing intentions using natural language, and the model then edits the image based on the user's description.
[0003] In some related technologies, instruction-based image editing methods mainly include two categories: image editing methods based on text-to-image technology and image editing methods based on image-to-image technology. Image editing methods based on text-to-image technology mainly utilize a Large Language Model (LLM) to optimize the user's text prompts, and then use the text-to-image model to generate and edit the image based on the optimized text prompts. Image editing methods based on image-to-image technology require the image to be edited and the text prompts as input to the model, and the model performs image editing based on the image to be edited and the text prompts to generate the edited image.
[0004] In the context of image item editing, the above solutions can all achieve good image editing results. This is because when generating and editing specific items in an image, the model can learn from massive amounts of data to understand the visual texture, shape, and color features of specific items. When the user asks the model to generate or edit a specific item, the model will create an image region that conforms to the visual statistical rules of the specific item based on the text prompts. This editing process is continuous and abstract, and existing models are good at handling this kind of visual abstraction process.
[0005] However, in text editing scenarios within images, such as when a user's text prompts them to add, delete, or modify a piece of text in the image to be edited, the above-mentioned solutions are ineffective. Each character in the text is a discrete symbol, and each character has a strict and unique shape definition. The strokes and structure of the characters cannot deviate in any way. This requires the model to have pixel-level precision editing capabilities, but existing models usually do not have pixel-level editing capabilities and cannot directly generate accurate text symbols. They can only generate a rough character shape based on the visual statistical rules of the characters themselves, which easily leads to errors in the strokes or structure of the generated characters (such as missing strokes), resulting in errors in the edited image. Especially in complex text editing scenarios such as Chinese, the image editing quality of the model is usually difficult to guarantee.
[0006] In summary, existing image editing methods perform poorly in text editing scenarios and cannot guarantee the quality of image editing. Summary of the Invention
[0007] This application provides an image editing method, apparatus, device, storage medium, and program product to solve the technical problem that existing image editing methods perform poorly in text editing scenarios and are difficult to guarantee the quality of image editing.
[0008] In a first aspect, embodiments of this application provide an image editing method, comprising: acquiring text prompts; the text prompts being prompts describing a user's editing instructions; based on a large language model, performing semantic understanding and operation intent classification on the text prompts to determine the editing operation type of the editing instruction and the text to be edited; if the editing operation type is text editing and the text to be edited is not empty text, then performing text rendering processing on the text to be edited to generate a text rendering image of the text to be edited; inputting the text rendering image, the image to be edited, and the text prompts into a pre-trained image editing model to obtain a first target image output by the image editing model; wherein, the first target image is obtained by the image editing model editing the image to be edited based on the text rendering image and the text prompts.
[0009] In one embodiment, the image editing model includes an image encoder, an adapter, a variational autoencoder, a text encoder, and a diffusion model; the image encoder is used to encode the text-rendered image to generate a first image feature; the variational autoencoder is used to extract features from the image to be edited to generate a second image feature; the text encoder is used to extract features from text prompts to generate text features; the adapter is used to project the first image features onto the semantic space of the text features to generate a target image feature; and the diffusion model is used to edit the image to be edited based on the second image feature, the text feature, and the target image feature to generate a first target image.
[0010] In one embodiment, the diffusion model includes a first cross-attention layer, a second cross-attention layer, and a self-attention layer; the first cross-attention layer is used to perform feature weighted fusion based on target image features and second image features to generate a first fused feature; the second cross-attention layer is used to perform feature weighted fusion based on text features and second image features to generate a second fused feature; and the self-attention layer is used to edit the image to be edited based on the first fused feature and the second fused feature to generate a first target image.
[0011] In one embodiment, after semantic understanding and operation intent classification of text prompts based on a large language model, and determining the editing operation type of the editing instruction and the text to be edited, the method further includes: if the text to be edited is empty text, then inputting the image to be edited and the text prompts into the image editing model to obtain a second target image output by the image editing model; wherein, the second target image is obtained by the image editing model after editing the image to be edited according to the text prompts.
[0012] In one embodiment, the image editing model includes a variational autoencoder, a text encoder, and a diffusion model; the variational autoencoder is used to extract features from the image to be edited to generate second image features; the text encoder is used to extract features from text prompts to generate text features; and the diffusion model is used to edit the image to be edited based on the second image features and the text features to generate a second target image.
[0013] In one embodiment, the diffusion model includes a second cross-attention layer and a self-attention layer; the second cross-attention layer is used to perform feature weighted fusion based on text features and second image features to generate a second fused feature; the self-attention layer is used to edit the image to be edited based on the second fused feature to generate a second target image.
[0014] Secondly, embodiments of this application provide an image editing device, comprising: an acquisition module for acquiring text prompts; the text prompts being prompts describing a user's editing instructions; a determination module for performing semantic understanding and operation intent classification on the text prompts based on a large language model, and determining the editing operation type of the editing instruction and the text to be edited; a rendering module for performing text rendering processing on the text to be edited if the editing operation type is text editing and the text to be edited is not empty text, thereby generating a text rendering image of the text to be edited; and an editing module for inputting the text rendering image, the image to be edited, and the text prompts into a pre-trained image editing model to obtain a first target image output by the image editing model; wherein the first target image is obtained by the image editing model editing the image to be edited based on the text rendering image and the text prompts.
[0015] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the image editing methods described above.
[0016] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the image editing methods described above.
[0017] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements any of the image editing methods described above.
[0018] The image editing method, apparatus, device, storage medium, and program product provided in this application introduce pixel-level text rendering technology. After obtaining text prompts describing the user's editing instructions, a large language model is first used to perform semantic understanding and operation intent classification on the text prompts to determine the editing operation type and the text to be edited. If the editing operation type is text editing and the text to be edited is not empty, then pixel-level text rendering processing is performed on the text to be edited to generate a text rendering image corresponding to the text to be edited. Then, the text rendering image, the image to be edited, and the text prompts are input into a pre-trained image editing model. The image editing model can then use the text rendering image and the text prompts to perform text rendering processing. The image editing model edits the image to be edited, generating the first target image after editing. Since the image editing model does not need to directly edit the text to be edited on the image to be edited based on the text prompts, but instead converts the text to be edited into a corresponding text rendering image, the model then edits the image to be edited based on the text rendering image and the text prompts. This transforms the model's direct text generation process into a visual abstraction process of the text rendering image and the image to be edited. Because the model is better at handling visual abstraction processes, it can fully utilize its editing capabilities to generate an accurate first target image, which is beneficial to improving the model's performance in text editing scenarios and thus ensuring the quality of image editing. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is one of the flowcharts illustrating the image editing method provided in the embodiments of this application.
[0021] Figure 2 This is the second flowchart illustrating the image editing method provided in the embodiments of this application.
[0022] Figure 3 This is a schematic diagram of the processing flow of the image editing model provided in the embodiments of this application.
[0023] Figure 4 This is a schematic diagram of the structure of the image editing device provided in the embodiments of this application.
[0024] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0026] Please see Figures 1 to 3 , Figure 1 This is one of the flowcharts illustrating the image editing method provided in the embodiments of this application. Figure 2 This is the second schematic flowchart of the image editing method provided in the embodiments of this application. Figure 3 This is a schematic diagram of the processing flow of the image editing model provided in the embodiments of this application.
[0027] In this embodiment, the image editing method is applied to an instruction-based image editing system based on intent classification. The instruction-based image editing system includes a semantic parsing module, a text rendering module, and a text-adaptive image editing module.
[0028] like Figure 1 As shown in the embodiments of this application, the image editing method includes steps S110 to S140, and the specific steps are as follows: S110: Get the text prompt word.
[0029] Text prompts are prompts that describe the user's editing instructions.
[0030] S120: Based on a large language model, semantic understanding and operation intent classification of text prompt words are performed to determine the editing operation type of the editing instruction and the text to be edited.
[0031] Specifically, the semantic parsing module is equipped with a large language model. After obtaining the text prompts input by the user, the semantic parsing module can input the text prompts into the large language model.
[0032] The text prompts are prompts that describe the user's editing instructions.
[0033] Furthermore, the large language model can perform semantic understanding and operation intent classification on text prompts to determine the editing operation type of the editing instruction and the text to be edited.
[0034] The editing operation types include text editing types and operation types that do not involve text editing.
[0035] Optionally, if the editing operation type of the editing instruction is an operation type that does not involve text editing, it can be assumed that the text prompt entered by the user does not indicate that the image to be edited is to be edited. In this case, the text to be edited is empty text (i.e. the text content of the text to be edited is empty).
[0036] Optionally, if the editing operation type of the editing instruction is text editing, it can be assumed that the text prompt entered by the user indicates that text editing is to be performed on the image to be edited. In this case, the text to be edited may be empty text or it may not be empty text.
[0037] Optionally, text editing types include text addition, text deletion, and text modification.
[0038] Optionally, if the text editing type is text addition, the text to be edited is not empty text, and the text content to be edited is the text that the user needs to add in the text prompt.
[0039] Optionally, if the text editing type is text deletion, the text to be edited is empty text, because the instruction-based image editing system only needs to delete the existing text in the image to be edited according to the text prompts, and does not need to edit the text additionally. Therefore, the text content of the text to be edited is empty.
[0040] Optionally, if the text editing type is text modification, the text to be edited is not empty text, and the text content of the text to be edited is the text that the user needs to modify in the text prompt.
[0041] S130: If the editing operation type is text editing type and the text to be edited is not empty text, then the text to be edited is rendered to generate a text rendering image of the text to be edited.
[0042] Specifically, if the editing operation type is text editing and the text to be edited is not empty text (i.e., the text editing type is text addition or text modification), it means that the user needs to edit new text in the image to be edited. At this time, the text rendering module can perform pixel-level text rendering processing on the text to be edited and generate a standardized text rendering image corresponding to the text to be edited.
[0043] by Figure 2For example, suppose the user inputs the text prompt: "Add the text 'Artificial Intelligence' to the image." After the semantic parsing module inputs this text prompt into the large language model, the large language model can perform semantic understanding and operation intent classification on the text prompt, determining that the editing operation type of the editing instruction is text addition, and the text to be edited is "Artificial Intelligence." At this time, since the editing operation type of the editing instruction is text editing, and the text to be edited is not empty text, the text rendering module can perform pixel-level text rendering processing on the text to be edited according to the style of black background and white text, and the size of 512×512 pixels, generating a standardized text rendering image corresponding to the text to be edited, and ensuring that the text "Artificial Intelligence" in the text rendering image is presented in a single-line centered form, with the font size dynamically changing according to the number of characters in the text to be edited. The text rendering image can use a single-channel format.
[0044] S140: Input the text rendering image, the image to be edited, and the text prompt into the pre-trained image editing model to obtain the first target image output by the image editing model.
[0045] The first target image is obtained by the image editing model after editing the image to be edited based on the text rendering image and text prompts.
[0046] Specifically, the text-adaptive image editing module is equipped with a pre-trained image editing model. After obtaining the standardized text rendering image corresponding to the text to be edited, the image editing module can input the text rendering image, the image to be edited, and the text prompts into the image editing model. The image editing model can perform precise pixel-level image editing on the image to be edited based on the text rendering image and the text prompts, and generate and output the edited first target image.
[0047] The image editing method provided in this application introduces pixel-level text rendering technology. After obtaining the text prompts describing the user's editing instructions, a large language model is first used to perform semantic understanding and operation intent classification on the text prompts to determine the editing operation type and the text to be edited. If the editing operation type is text editing and the text to be edited is not empty, then pixel-level text rendering processing is performed on the text to be edited to generate a text rendering image corresponding to the text to be edited. Then, the text rendering image, the image to be edited, and the text prompts are input into a pre-trained image editing model. The image editing model can then process the text to be edited based on the text rendering image and the text prompts. The image editing process generates the first target image after editing. Since the image editing model does not need to directly edit the text to be edited on the image to be edited based on the text prompts, but instead converts the text to be edited into a corresponding text rendering image, the model then edits the image to be edited based on the text rendering image and the text prompts. This transforms the model's direct text generation process into a visual abstraction process of the text rendering image and the image to be edited. Because the model is better at handling visual abstraction processes, it can fully utilize its editing capabilities to generate an accurate first target image, which is beneficial to improving the model's performance in text editing scenarios and thus ensuring the quality of image editing.
[0048] In some embodiments, the image editing model includes an image encoder, an adapter, a variational autoencoder, a text encoder, and a diffusion model; the image encoder is used to encode the text-rendered image to generate a first image feature; the variational autoencoder is used to extract features from the image to be edited to generate a second image feature; the text encoder is used to extract features from text prompts to generate text features; the adapter is used to project the first image features onto the semantic space of the text features to generate a target image feature; and the diffusion model is used to edit the image to be edited based on the second image feature, the text feature, and the target image feature to generate a first target image.
[0049] like Figure 3 As shown, the image editing model includes an image encoder, an adapter, a variational autoencoder (VAE), a text encoder, and a diffusion model.
[0050] Specifically, if the editing operation type is text editing and the text to be edited is not empty text (i.e., the text editing type is text addition or text modification), it means that the user needs to edit new text in the image to be edited. At this time, the text rendering module can perform pixel-level text rendering processing on the text to be edited and generate a standardized text rendering image corresponding to the text to be edited.
[0051] Furthermore, the image editing module is equipped with a pre-trained image editing model. After obtaining the standardized text rendering image corresponding to the text to be edited, the image editing module can input the text rendering image, the image to be edited, and the text prompts into the image editing model. The image editing model has three network branches, which are used to process the text rendering image, the image to be edited, and the text prompts, respectively.
[0052] Specifically, such as Figure 3 As shown, the image encoder is a sigclip image encoder, which can be used to encode text-rendered images, convert the text-rendered images into a series of visual features, and expand these visual features into visual tokens as the first image features.
[0053] A variational autoencoder is a feature extractor that can be used to extract features from an image to be edited, generating a second image feature.
[0054] Meanwhile, the text encoder is used to extract features from text prompts and generate text features.
[0055] Furthermore, the image editing model can input the first image features encoded by the image encoder into a lightweight adapter, which can project the first image features into the semantic space of the text features to generate the target image features.
[0056] It should be noted here that the process of the image encoder encoding and generating the first image features can be regarded as the process of converting the text-rendered image into a "visual language" that the model can understand. The process of projecting the first image features into the semantic space of the text features can be regarded as the process of translating the "visual language" into a "textual language" that the model can understand, so that the subsequent diffusion model can process the target image features in the same way as it processes text features.
[0057] Furthermore, the second image features, text features, and target image features are input into the diffusion model, which performs the denoising process. During the denoising process, the diffusion model can edit the image to be edited based on the second image features, text features, and target image features to generate the first target image.
[0058] In some embodiments, the diffusion model includes a first cross-attention layer, a second cross-attention layer, and a self-attention layer; the first cross-attention layer is used to perform feature weighted fusion based on target image features and second image features to generate a first fused feature; the second cross-attention layer is used to perform feature weighted fusion based on text features and second image features to generate a second fused feature; and the self-attention layer is used to edit the image to be edited based on the first fused feature and the second fused feature to generate a first target image.
[0059] In this embodiment, the diffusion model uses a Diffusion Transformer network (DiT network), which includes a first cross-attention layer, a second cross-attention layer, and a self-attention layer. The two cross-attention layers are used to independently process target image features and text features, respectively.
[0060] Specifically, after inputting the second image features, text features, and target image features into the diffusion model, the first cross-attention layer can extract cross-attention features from the target image features and the second image features respectively, and then perform a weighted sum of the two features after cross-attention feature extraction to generate the first fused feature.
[0061] Similarly, after inputting the second image features, text features, and target image features into the diffusion model, the second cross-attention layer can extract cross-attention features from the text features and the second image features respectively, and then perform a weighted sum of the two features after cross-attention feature extraction to generate the second fused feature.
[0062] Furthermore, the first fusion feature and the second fusion feature are input into the self-attention layer, which can edit the image to be edited based on the first fusion feature and the second fusion feature to generate the first target image.
[0063] In some embodiments, after semantic understanding and operation intent classification of text prompts based on a large language model, and determining the editing operation type of the editing instruction and the text to be edited, the method further includes: if the text to be edited is empty text, then inputting the image to be edited and the text prompts into the image editing model to obtain a second target image output by the image editing model; wherein, the second target image is obtained by the image editing model after editing the image to be edited according to the text prompts.
[0064] Specifically, the semantic parsing module is equipped with a large language model. After obtaining the text prompts input by the user, the semantic parsing module can input the text prompts into the large language model.
[0065] The text prompts are prompts that describe the user's editing instructions.
[0066] Furthermore, the large language model can perform semantic understanding and operation intent classification on text prompts to determine the editing operation type of the editing instruction and the text to be edited.
[0067] The editing operation types include text editing types and operation types that do not involve text editing.
[0068] If the editing operation type of the editing command is an operation type that does not involve text editing, it can be assumed that the text prompt entered by the user does not indicate that the image to be edited is to be edited. In this case, the text to be edited is empty text (that is, the text content of the text to be edited is empty).
[0069] If the editing operation type of the editing command is text editing, it can be assumed that the text prompt entered by the user indicates that text editing is to be performed on the image to be edited. In this case, the text to be edited may be empty text or it may not be empty text.
[0070] The text editing types include adding text, deleting text, and modifying text.
[0071] If the text editing type is "text addition", then the text to be edited is not empty text; the text content to be edited is the text that the user needs to add in the text prompt.
[0072] If the text editing type is text deletion, the text to be edited is empty text. This is because the instruction-based image editing system only needs to delete the existing text in the image to be edited based on the text prompts, and does not need to edit the text additionally. Therefore, the text content to be edited is empty.
[0073] If the text editing type is text modification, then the text to be edited is not empty text; the text content to be edited is the text that the user needs to modify as indicated in the text prompt.
[0074] Furthermore, if the text to be edited is empty text, that is, the editing operation type of the editing instruction is an operation type that does not involve text editing or the text editing type is text deletion, it means that the user does not need to edit new text in the image to be edited. In this case, the image to be edited and the text prompt can be directly input into the image editing model. The image editing model can directly perform precise pixel-level image editing on the image to be edited based on the text prompt, and generate and output the edited second target image.
[0075] Please continue reading. Figure 2 ,from Figure 2 As can be seen from the processing flow, in the text editing scenario of images, the core role of the semantic parsing module is to use a large language model to determine whether the instruction-based image editing system should execute based on the specific content of the text prompts. Figure 2 The process is shown in the dotted line section, which precisely categorizes the user's intent into one of four editing operations (i.e., operations that do not involve text editing, text addition, text deletion, and text modification), so that subsequent image editing can be performed according to the type of editing operation.
[0076] In some embodiments, the image editing model includes a variational autoencoder, a text encoder, and a diffusion model; the variational autoencoder is used to extract features from the image to be edited to generate second image features; the text encoder is used to extract features from text prompts to generate text features; and the diffusion model is used to edit the image to be edited based on the second image features and the text features to generate a second target image.
[0077] like Figure 3 As shown, the image editing model includes an image encoder, an adapter, a variational autoencoder (VAE), a text encoder, and a diffusion model.
[0078] If the text to be edited is empty text, that is, the editing operation type of the editing instruction is an operation type that does not involve text editing or the text editing type is text deletion, it means that the user does not need to edit new text in the image to be edited. In this case, the image to be edited and the text prompt can be directly input into the image editing model.
[0079] The image editing model itself has three network branches, which are used to process the text-rendered image, the image to be edited, and the text prompt, respectively. However, if the text to be edited is empty, the network branch processing the text-rendered image will not perform any operation. Figure 3 The process shown in the dashed line will no longer be executed.
[0080] A variational autoencoder is a feature extractor that can directly extract features from an image to be edited, generating a second image feature, when the text to be edited is empty.
[0081] At the same time, the text encoder can extract features from text prompts and generate text features.
[0082] Furthermore, the second image features and text features are input into the diffusion model, which performs the denoising process. During the denoising process, the diffusion model can directly edit the image to be edited based on the second image features and text features to generate the second target image.
[0083] In some embodiments, the diffusion model includes a second cross-attention layer and a self-attention layer; the second cross-attention layer is used to perform feature weighted fusion based on text features and second image features to generate a second fused feature; the self-attention layer is used to edit the image to be edited based on the second fused feature to generate a second target image.
[0084] In this embodiment, the diffusion model uses a Diffusion Transformer network (DiT network), which includes a first cross-attention layer, a second cross-attention layer, and a self-attention layer. The two cross-attention layers are used to independently process target image features and text features, respectively.
[0085] However, when the text to be edited is empty, there is no need to generate a text rendering image corresponding to the text to be edited. The diffusion model also does not need to process the target image features corresponding to the text rendering image. At this time, the first cross-attention layer, which was originally used to process the target image features, will not perform any operation.
[0086] When the text to be edited is empty, the diffusion model can directly input the text features and target image features into the second cross-attention layer. The second cross-attention layer can extract cross-attention features from the text features and the second image features respectively, and then perform a weighted sum of the two features after cross-attention feature extraction to generate the second fused feature.
[0087] Furthermore, the second fusion feature is input into the self-attention layer, which can edit the image to be edited based on the second fusion feature to generate the second target image.
[0088] The image editing method provided in this application embodiment sets up a diffusion model in the image editing model. The diffusion model uses two independent cross-attention layers to perform cross-attention feature extraction and feature weighted fusion on the target image features and text features respectively. This makes the cross-attention network compatible with both images with and without text rendering, thus expanding the applicability of the solution.
[0089] Understandably, a command-based image editing system needs to be trained before it can be used for image editing.
[0090] In this embodiment, the training process of the instruction-based image editing system can be divided into three stages: the pre-training stage, the text rendering fine-tuning stage, and the full fine-tuning stage.
[0091] Specifically, in the pre-training phase, a large number of first sample text prompts are first obtained. The text to be edited described in each first sample text prompt is empty text, and the editing operation type of the editing instruction is either an operation type that does not involve text editing or a text editing type that is text deletion. Then, the first sample image to be edited corresponding to each first sample text prompt and the sample target image corresponding to each first sample image to be edited are obtained. Then, the instruction-based image editing system is pre-trained based on all first sample text prompts, all first sample images to be edited corresponding to all first sample text prompts, and all sample target images corresponding to all first sample images to be edited.
[0092] by Figure 2 For example, the pre-training phase actually uses a large amount of training data to train the instruction-based image editing system to learn and execute commands. Figure 2 The processing flow for the non-dashed lines in the text.
[0093] Furthermore, in the text rendering fine-tuning stage, a large number of second sample text prompts are first acquired. The text to be edited described in each second sample text prompt is not empty text, and the editing operation type of the editing instruction is text editing, which is either text addition or text modification. For each second sample text prompt, text rendering processing is performed on the text to be edited corresponding to that second sample text prompt to generate a sample text rendering image of that text to be edited. The second sample image to be edited corresponding to each second sample text prompt and the sample target image corresponding to each second sample image to be edited are acquired. Then, all weight parameters obtained by the instruction-based image editing system during the pre-training stage are frozen. Based on all second sample text prompts, the sample text rendering images corresponding to the text to be edited described in all second sample text prompts, the second sample images to be edited corresponding to all second sample text prompts, and the sample target images corresponding to all second sample images to be edited, the instruction-based image editing system is trained for text rendering fine-tuning.
[0094] by Figure 2 For example, the text rendering fine-tuning stage actually uses a large amount of additional training data on top of pre-training to train the instruction-based image editing system to learn and execute... Figure 2 After the text rendering fine-tuning stage, the processing flow of the dotted line part in the image is such that the instruction-based image editing system has the ability to adapt to text rendering scenarios.
[0095] Furthermore, in the full fine-tuning stage, it is necessary to acquire all the training data from the pre-training stage and the text rendering fine-tuning stage. Then, randomly select some data corresponding to the first sample text prompt words and the second sample text prompt words from these training data as the training data for this stage. Then, using the training data for this stage, the gradient of all weight parameters obtained by the instruction-based image editing system in the pre-training stage and the text rendering fine-tuning stage is updated to obtain the final instruction-based image editing system.
[0096] The image editing method provided in this application includes a text-adapted image editing module and a text-adapted instruction-based image editing system. This optimizes the text editing effect in image-text editing scenarios and can simultaneously support text editing and other types of editing, which helps ensure the applicability of the solution.
[0097] This application also provides an image editing device. Please refer to... Figure 4 , Figure 4 This is a schematic diagram of the structure of an image editing device provided in an embodiment of this application. In this embodiment, the image editing device includes an acquisition module 410, a determination module 420, a rendering module 430, and an editing module 440.
[0098] The acquisition module 410 is used to acquire text prompt words.
[0099] Text prompts are prompts that describe the user's editing instructions.
[0100] The determination module 420 is used to perform semantic understanding and operation intent classification on text prompt words based on a large language model, and to determine the editing operation type of the editing instruction and the text to be edited.
[0101] The rendering module 430 is used to perform text rendering processing on the text to be edited if the editing operation type is text editing type and the text to be edited is not empty text, and to generate a text rendering image of the text to be edited.
[0102] The editing module 440 is used to input the text rendering image, the image to be edited, and the text prompt into the pre-trained image editing model to obtain the first target image output by the image editing model.
[0103] The first target image is obtained by the image editing model after editing the image to be edited based on the text rendering image and text prompts.
[0104] In some embodiments, the image editing model includes an image encoder, an adapter, a variational autoencoder, a text encoder, and a diffusion model; the image encoder is used to encode the text-rendered image to generate a first image feature; the variational autoencoder is used to extract features from the image to be edited to generate a second image feature; the text encoder is used to extract features from text prompts to generate text features; the adapter is used to project the first image features onto the semantic space of the text features to generate a target image feature; and the diffusion model is used to edit the image to be edited based on the second image feature, the text feature, and the target image feature to generate a first target image.
[0105] In some embodiments, the diffusion model includes a first cross-attention layer, a second cross-attention layer, and a self-attention layer; the first cross-attention layer is used to perform feature weighted fusion based on target image features and second image features to generate a first fused feature; the second cross-attention layer is used to perform feature weighted fusion based on text features and second image features to generate a second fused feature; and the self-attention layer is used to edit the image to be edited based on the first fused feature and the second fused feature to generate a first target image.
[0106] In some embodiments, after semantic understanding and operation intent classification of text prompts based on a large language model, and determining the editing operation type of the editing instruction and the text to be edited, the method further includes: if the text to be edited is empty text, then inputting the image to be edited and the text prompts into the image editing model to obtain a second target image output by the image editing model; wherein, the second target image is obtained by the image editing model after editing the image to be edited according to the text prompts.
[0107] In some embodiments, the image editing model includes a variational autoencoder, a text encoder, and a diffusion model; the variational autoencoder is used to extract features from the image to be edited to generate second image features; the text encoder is used to extract features from text prompts to generate text features; and the diffusion model is used to edit the image to be edited based on the second image features and the text features to generate a second target image.
[0108] In some embodiments, the diffusion model includes a second cross-attention layer and a self-attention layer; the second cross-attention layer is used to perform feature weighted fusion based on text features and second image features to generate a second fused feature; the self-attention layer is used to edit the image to be edited based on the second fused feature to generate a second target image.
[0109] This application also provides an electronic device. Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute image editing methods.
[0110] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0111] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image editing methods provided by the above methods.
[0112] This application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the image editing methods provided by the above methods.
[0113] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. An image editing method, characterized in that, include: Get the text prompt words; The text prompts are prompts that describe the user's editing instructions; Based on a large language model, the text prompts are semantically understood and classified according to their operational intent to determine the editing operation type and the text to be edited. If the editing operation type is text editing type and the text to be edited is not empty text, then the text to be edited is subjected to text rendering processing to generate a text rendering image of the text to be edited; The text-rendered image, the image to be edited, and the text prompt are input into a pre-trained image editing model to obtain the first target image output by the image editing model. The first target image is obtained by the image editing model editing the image to be edited based on the text-rendered image and the text prompt.
2. The image editing method according to claim 1, characterized in that, The image editing model includes an image encoder, an adapter, a variational autoencoder, a text encoder, and a diffusion model; The image encoder is used to encode the text-rendered image to generate a first image feature; The variational autoencoder is used to extract features from the image to be edited and generate second image features; The text encoder is used to extract features from the text prompt words and generate text features; The adapter is used to project the first image features onto the semantic space of the text features to generate target image features; The diffusion model is used to edit the image to be edited based on the second image features, the text features, and the target image features to generate the first target image.
3. The image editing method according to claim 2, characterized in that, The diffusion model includes a first cross-attention layer, a second cross-attention layer, and a self-attention layer; The first cross-attention layer is used to perform feature weighted fusion based on the target image features and the second image features to generate a first fused feature; The second cross-attention layer is used to perform feature weighted fusion based on the text features and the second image features to generate a second fused feature; The self-attention layer is used to edit the image to be edited based on the first fusion feature and the second fusion feature to generate the first target image.
4. The image editing method according to claim 1, characterized in that, After determining the editing operation type and the text to be edited based on the large language model by performing semantic understanding and operation intent classification on the text prompt words, the method further includes: If the text to be edited is empty, then the image to be edited and the text prompt are input into the image editing model to obtain the second target image output by the image editing model; The second target image is obtained by the image editing model editing the image to be edited according to the text prompts.
5. The image editing method according to claim 4, characterized in that, The image editing model includes a variational autoencoder, a text encoder, and a diffusion model; The variational autoencoder is used to extract features from the image to be edited and generate second image features; The text encoder is used to extract features from the text prompt words and generate text features; The diffusion model is used to edit the image to be edited based on the second image features and the text features to generate the second target image.
6. The image editing method according to claim 5, characterized in that, The diffusion model includes a second cross-attention layer and a self-attention layer; The second cross-attention layer is used to perform feature weighted fusion based on the text features and the second image features to generate a second fused feature; The self-attention layer is used to edit the image to be edited based on the second fusion feature to generate the second target image.
7. An image editing device, characterized in that, include: The acquisition module is used to acquire text prompt words; The text prompts are prompts that describe the user's editing instructions; The determination module is used to perform semantic understanding and operation intent classification on the text prompt words based on a large language model, and to determine the editing operation type and the text to be edited by the editing instruction; The rendering module is used to perform text rendering processing on the text to be edited and generate a text rendering image of the text to be edited if the editing operation type is text editing type and the text to be edited is not empty text. The editing module is used to input the text rendering image, the image to be edited, and the text prompt into a pre-trained image editing model to obtain the first target image output by the image editing model; The first target image is obtained by the image editing model editing the image to be edited based on the text-rendered image and the text prompt.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the image editing method as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image editing method as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image editing method as described in any one of claims 1 to 6.