A semantic image editing method and device
By decoupling structural and detail latent features from the original image, predicting the editing region and generating residuals respectively, the problem of inaccurate editing in existing technologies is solved, and high-precision image editing is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-02-06
- Publication Date
- 2026-06-09
AI Technical Summary
Existing image editing methods lack explicit distinction between different semantic levels, which can easily lead to cascading effects during fine-grained editing operations, making it difficult to limit the scope of modifications and affecting usability in practical creative design, industrial drawing, game art, and advertising generation scenarios.
By extracting structural and detail latent features from the original image, decoupling structural and detail-level features, predicting structural and detail editing regions respectively, and using a residual generative network to generate edited features, accurate editing is achieved.
It enables precise editing within the corresponding editing area, reduces the risk of accidental operation in non-editing areas, and improves editing accuracy and stability.
Smart Images
Figure CN122176094A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image generation technology, and in particular to a semantic image editing method and apparatus. Background Technology
[0002] With the development of artificial intelligence technology, image editing technology based on natural language instructions has received widespread attention in the field of computer vision. This image editing method maps the input image to a latent representation, combines the instruction text to modulate or update the latent features, and generates an image editing result that conforms to the semantic description.
[0003] Existing image editing methods typically encode images using a unified set of continuous latent features. These latent features simultaneously represent multiple levels of information, including the image's geometric structure, spatial layout, local texture, and stylistic attributes. Due to the lack of explicit distinction between different semantic levels, when users only wish to perform fine-grained editing operations, the model often struggles to limit the scope of modification, easily leading to cascading effects. For example, in scenarios where only the material or texture of an object is modified, the model may simultaneously change the object's shape or the overall composition; replacing local objects often results in the destruction of the background structure or unnecessary changes to the global style. This feature hybridity problem limits the model's ability to express "editable granularity," affecting its usability in practical creative design, industrial drafting, game art, and advertising generation scenarios. Summary of the Invention
[0004] In view of this, the purpose of this application is to provide a semantic image editing method and apparatus.
[0005] To achieve the above objectives, embodiments of this application provide a semantic image editing method, including:
[0006] Extract the corresponding original latent features and target latent features from the original image and the target image respectively; wherein, the target image is obtained by editing the original image according to the editing instruction text; The edit command text is encoded to obtain a command vector representation; Extract the original structural latent features and original detail latent features, as well as the target structural latent features and target detail latent features, from the original latent features and the target latent features, respectively. Based on the cross-modal model, structural localization features aligned with the original structural latent features and detail localization features aligned with the original detail latent features are generated respectively. Perturbations are added to the original structural latent features and the original detail latent features respectively. The changes in structural consistency scores between the original structural latent features and the instruction vector representation before and after the perturbations are calculated, as well as the changes in detail consistency scores between the original detail latent features and the instruction vector representation before and after the perturbations. Generate the structural response features of the original structural latent features after graph structure propagation, and the detailed response features of the original detailed latent features after graph structure propagation; Based on the structural positioning features, the structural differences between the original structural latent features and the target structural latent features, the structural consistency score change, and the structural response features, a structural editing region is generated. Based on the detailed localization features, the detailed differences between the original detailed latent features and the target detailed latent features, the change in detailed consistency score, and the detailed response features, a detailed editing region is generated. Based on the original structural latent features, original detail latent features, instruction vector representation, structural editing region, and detail editing region, a residual generation network is used to predict structural residuals and detail residuals. Based on the original structural latent features, structural residuals, and structural editing regions, edited structural latent features are generated; based on the original detail latent features, detail residuals, and detail editing regions, edited detail latent features are generated. The edited image is reconstructed based on the edited structural latent features and detail latent features.
[0007] Optionally, the extraction of original structural latent features and original detail latent features, target structural latent features and target detail latent features from the original latent features and target latent features respectively includes: The original latent features are subjected to discrete wavelet transform processing to obtain the low-frequency structure and high-frequency details corresponding to the original image. The low-frequency structure is used as the original structural latent feature, and the high-frequency detail is used as the original detail latent feature. The target latent features are subjected to discrete wavelet transform processing to obtain the low-frequency structure and high-frequency details corresponding to the target image. The low-frequency structure is used as the target structural latent feature, and the high-frequency details are used as the target detail latent feature.
[0008] Optionally, the generation of the structural response features of the original structural latent features after graph structure propagation, and the detailed response features of the original detailed latent features after graph structure propagation, includes: The feature map composed of the original structural latent features is divided into multiple structural image blocks. The structural similarity between the structural image blocks is calculated. The structural similarity is used to form a structural adjacency matrix. Based on the structural adjacency matrix, the structural response features after graph structure propagation are generated. The feature map composed of the original latent detail features is divided into multiple detail image blocks. The detail similarity between detail image blocks is calculated, and the detail similarity is used to form a detail adjacency matrix. The detail response features after graph structure propagation are generated based on the detail adjacency matrix.
[0009] Optionally, the graph structure propagation iteration process is represented as: (1) in, H (t) For the first t The feature response state at the next iteration; W This refers to the structural adjacency matrix or the detail adjacency matrix; H (0) The initial feature response state is the change in structural consistency score for the original structural latent feature and the change in detail consistency score for the original detail latent feature. As a propagation factor; H (t+1) The updated feature response state is obtained after propagation through the graph structure. After iterative stabilization, the result of the original structural latent feature is the structural response feature, and the result of the original detail latent feature is the detail response feature.
[0010] Optionally, the change in structural consistency score between the original structural latent features and the instruction vector representation before and after the perturbation includes: Calculate the first CLIP score between the original structural latent features before perturbation and the instruction vector representation, and the second CLIP score between the original structural latent features and the instruction vector representation after perturbation; Calculate the difference between the first CLIP score and the second CLIP score; Calculate the change in detail consistency score between the original latent detail features and the instruction vector representation before and after the perturbation, including: Calculate the third CLIP score between the original latent detail features and the instruction vector representation before perturbation, and the fourth CLIP score between the original latent detail features and the instruction vector representation after perturbation; Calculate the difference between the third CLIP score and the fourth CLIP score.
[0011] Optionally, based on the structural location features, the structural differences between the original structural latent features and the target structural latent features, the change in the structural consistency score, and the structural response features, a structural editing region is generated, using the following method: (2) in, A attnFor structural positioning features, Due to structural differences, S C This represents the change in structural consistency score. G C For structural response characteristics, For normalization operations, , , , These are the weights of the corresponding items.
[0012] Optionally, based on the detailed localization features, the detailed differences between the original detailed latent features and the target detailed latent features, the change in detailed consistency score, and the detailed response features, a detailed editing region is generated, using the following method: (3) in, A attn To locate features for details, For minor differences, S F The change in the score for consistency in details. G F For detailed response features, , , , These are the weights of the corresponding items.
[0013] Optionally, based on the original structural latent features, structural residuals, and structural editing regions, edited structural latent features are generated, using the following method: (4) in, C 0 represents the original structural latent feature. H C This is the structure editing area. For structural residuals, C edit This refers to the edited structural latent features.
[0014] Optionally, based on the original latent detail features, detail residuals, and detail editing regions, edited latent detail features are generated, using the following method: (5) in, F 0 represents the original detail latent feature. H F For detailed editing area, For minor imperfections, F edit For the edited details and latent features.
[0015] This application also provides a semantic image editing device, including: The first extraction module is used to extract corresponding original latent features and target latent features from the original image and the target image, respectively; wherein, the target image is obtained by editing the original image according to the editing instruction text; The encoding module is used to encode the editing instruction text to obtain an instruction vector representation; The second extraction module is used to extract the original structural latent features and original detail latent features, the target structural latent features and the target detail latent features from the original latent features and the target latent features, respectively. The alignment module is used to generate, based on the cross-modal model, structural localization features aligned with the original structural latent features and detail localization features aligned with the original detail latent features. The sensitivity scoring module is used to add perturbations to the original structural latent features and the original detail latent features respectively, and calculate the change in structural consistency score between the original structural latent features and the instruction vector representation before and after the perturbation, as well as the change in detail consistency score between the original detail latent features and the instruction vector representation before and after the perturbation. The graph structure propagation module is used to generate the structural response features of the original structural latent features after graph structure propagation, and the detailed response features of the original detailed latent features after graph structure propagation. The structural region generation module is used to generate a structural editing region based on the structural positioning features, the structural differences between the original structural latent features and the target structural latent features, the structural consistency score change, and the structural response features. The detail region generation module is used to generate a detail editing region based on the detail localization features, the detail differences between the original detail latent features and the target detail latent features, the change in detail consistency score, and the detail response features. The residual prediction module is used to predict structural residuals and detail residuals using a residual generation network based on the original structural latent features, original detail latent features, instruction vector representation, structural editing region, and detail editing region. The feature editing module is used to generate edited structural latent features based on the original structural latent features, structural residuals, and structural editing regions, and to generate edited detail latent features based on the original detail latent features, detail residuals, and detail editing regions. An image generation module is used to reconstruct the edited image based on the edited structural latent features and detail latent features.
[0016] As can be seen from the above description, the semantic image editing method and apparatus provided in this application extracts original latent features from the original image, decomposes original structural latent features and original detail latent features from the original latent features, decouples structural features and detail features, predicts corresponding structural editing regions and detail editing regions based on the decoupled original structural latent features and original detail latent features, predicts structural residuals within the structural editing region and detail residuals within the detail editing region, generates edited structural latent features based on the original structural latent features, structural residuals, and structural editing regions, generates edited detail latent features based on the original detail latent features, detail residuals, and detail editing regions, and reconstructs the edited image based on the edited structural latent features and detail latent features. This application can accurately perform editing operations in the corresponding editing regions, reduce the risk of accidental editing of non-editing regions, and effectively improve editing accuracy, stability, and controllability. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a schematic diagram of the method flow of an embodiment of this application; Figure 2 This is a schematic diagram of the training process in an embodiment of this application; Figure 3 This is a schematic diagram of the editing process in an embodiment of this application; Figure 4 This is a block diagram of the device structure according to an embodiment of this application; Figure 5 This is a block diagram of the electronic device structure according to an embodiment of this application. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.
[0020] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0021] In related technologies, natural language instruction-driven image editing is generally based on a unified latent feature editing paradigm, such as achieving editing control by inverting latent variables or constructing instruction image pairs. While this offers a certain degree of editing flexibility, the high coupling between structure and detail in the latent features makes it difficult to guarantee editing stability. To address this issue, some studies introduce additional structural conditions or auxiliary control signals, such as constraining the image generation process through edge maps, depth maps, or pose information to improve structural consistency; other methods attempt to improve editing accuracy through object-level or geometric-level interactive operations. However, these methods typically rely on manually provided masks or explicit control inputs, making it difficult to achieve fully automated editing region localization. Some studies focus on editing region prediction and local control, introducing region-aware mechanisms into the model to enable instruction text to act on specific spatial locations. However, these methods often emphasize coarse-grained region localization, making it difficult to distinguish between structural and detail changes at the spatial and semantic levels. Moreover, most of these methods still apply updates within the unified latent features, failing to achieve decoupled editing control of structure and detail at the residual level. When faced with complex natural scenes or multi-attribute editing instructions, cross-semantic level interference still occurs.
[0022] In view of this, embodiments of this application provide a semantic image editing method that decouples structural-level features and detail-level features. Based on the structural-level features and detail-level features, it predicts the corresponding structural editing regions and their structural residuals, as well as the detail editing regions and their detail residuals. Based on the structural-level features, the structural residuals are injected into the structural editing regions to obtain edited latent structural features. Based on the detail-level features, the detail residuals are injected into the detail editing regions to obtain edited latent detail features. Based on the edited latent structural features and detail latent features, the edited image is reconstructed. This method requires no manual assistance, automatically predicts the editing regions, and accurately performs editing operations within the corresponding regions, reducing the risk of accidental editing of non-editing regions and effectively improving editing accuracy.
[0023] The technical solution of this application will be further described in detail below through specific embodiments.
[0024] like Figure 1 , 2 As shown, this application provides a semantic image editing method, including: S101: Extract the corresponding original latent features and target latent features from the original image and the target image respectively; wherein, the target image is obtained by editing the original image according to the editing instructions text; In this embodiment, training samples are constructed for training the semantic image editing model. The training samples include several triplets consisting of an original image, editing instruction text, and a target image. In each triplet, the original image is edited according to the editing instruction text to obtain the corresponding target image. The target image serves as the ground truth label and is pre-set during the training phase, providing the model with the "standard answer." The semantic image editing model includes an image encoder, a text encoder, a cross-modal model, a frequency domain decomposition module, an image decoder (or image decoding network), a CLIP multimodal model, a graph structure module, an edit region prediction module, a residual generation network, and an editing module.
[0025] During training, the original image is input into the image encoder, which then extracts data from the original image. I Extracting semantically expressive raw latent features from 0, denoted as: Z 0= E ( I 0), where, This represents the raw latent features extracted by the image encoder. Z 0 is a hybrid latent feature tensor containing structural and detail information of the image. C represents the number of channels. Similarly, the target image is input into the image encoder, and the image encoder extracts data from the target image... I Extracting latent features of the target containing structural and detailed information. Z1, represented as Z 1= E ( I 1). Optionally, the image encoder is implemented based on the Vision Transformer (ViT) architecture.
[0026] S102: Encode the editing instruction text to obtain an instruction vector representation; In this embodiment, the editing instruction text will be... T In the input text encoder, the text encoder generates instruction vector representations corresponding to the editing instruction text. V T Optionally, the text encoder can be implemented based on the BERT (Bidirectional Encoder Representations from Transformers) model.
[0027] S103: Extract the original structural latent features and original detail latent features, as well as the target structural latent features and target detail latent features from the original latent features and the target latent features, respectively; In this embodiment, frequency domain decomposition is performed on the original latent features extracted from the original image to extract the original structural latent features and original detail latent features. Similarly, frequency domain decomposition is performed on the target latent features extracted from the target image to extract the target structural latent features and target detail latent features. Specifically, this includes: Discrete wavelet transform is performed on the original latent features to obtain the low-frequency structure and high-frequency details corresponding to the original image. The low-frequency structure is used as the original structural latent feature, and the high-frequency detail is used as the original detail latent feature. Discrete wavelet transform is performed on the latent features of the target to obtain the low-frequency structure and high-frequency details corresponding to the target image. The low-frequency structure is taken as the latent feature of the target structure, and the high-frequency details are taken as the latent feature of the target details.
[0028] In this embodiment, the frequency domain decomposition module is used to analyze the original latent features. Z A two-dimensional discrete wavelet transform is performed using orthogonal Daubechies wavelets. Low-pass and high-pass convolution operations are performed in the horizontal and vertical directions, respectively, followed by a 2x downsampling. Daubechies wavelet transform is then applied to each channel of the original latent features, resulting in four sub-bands. LL 0, LH 0, HL 0, HH 0= DWT ( Z 0), where, LL The 0-frequency subband corresponds to the low-frequency structural components of the image. LH 0, HL 0, HH The three high-frequency subbands correspond to the high-frequency detail components of the image, representing details such as texture, material, and lighting in different directions.
[0029] Through frequency domain decomposition, the original latent features can be decomposed into original structural latent features representing the low-frequency structure. C 0= LL 0, and the original detail latent features representing high-frequency details. F 0 = Concat( LH 0, HL 0, HH 0), where Concat() represents the feature concatenation operation. Following the same frequency domain decomposition method, the target latent features are... Z 1. Decompose into target structural latent features representing low-frequency structures C 1, and latent features representing high-frequency details of the target. F 1.
[0030] Specifically, the original or target structural latent features can be used to describe the geometric shape, spatial layout, and global relationships of objects in the original or target image. Meanwhile, the original or target detail latent features, composed of three high-frequency sub-bands, can be used to describe details such as texture directionality, material repetition patterns, and local illumination variations in the original or target image. This constructs a dual-scale editable feature representation at both the structural and detail levels, achieving explicit decoupling of image structural and detail information. Subsequent processing, such as editing region prediction, can be performed based on the decoupled structural and detail latent features.
[0031] S104: Based on the cross-modal model, generate structural localization features aligned with the original structural latent features by instruction vector representation, and detail localization features aligned with the original detail latent features by instruction vector representation; In this embodiment, based on a preset cross-modal model, the correspondence between the instruction vector representation and the original structural latent features, and between the instruction vector representation and the original detail latent features are established respectively, and structural localization features aligned with the instruction vector representation and the original structural latent features are generated, as well as detail localization features aligned with the instruction vector representation and the original detail latent features.
[0032] In some approaches, cross-modal models are implemented based on the cross-attention module of the Transformer architecture. Through cross-modal attention mechanisms, textual semantics are aligned with dual-scale latent features to generate structural localization features. A C and detailed positioning features A F , represented as .
[0033] S105: Add perturbations to the original structural latent features and the original detail latent features respectively, and calculate the change in structural consistency score between the original structural latent features and the instruction vector representation before and after the perturbation, as well as the change in detail consistency score between the original detail latent features and the instruction vector representation before and after the perturbation. In this embodiment, frequency-limited small perturbations are added to the original structural latent features and the original detail latent features, respectively. The CLIP multimodal model is used to calculate the change in consistency score between the features before and after the perturbation and the instruction vector representation. The change in consistency score is used to evaluate the sensitivity of structural and detail-level editing. This sensitivity measures the importance of the corresponding spatial location to the instruction vector representation, that is, the degree to which editing this location contributes to completing the editing instruction. The CLIP multimodal model is used to embed the image and instruction text into the same vector space and evaluate the similarity between them. The specific principles and structure of this model are not described in detail.
[0034] In some implementations, respectively in the latent features of the original structure C 0 and original details latent features F Inject controlled micro-perturbations at each spatial location of 0 In the original structural latent features C 0 and original details latent features F 0. Apply frequency-limited wavelet domain perturbation, then perform inverse wavelet transform on the original structural latent features and original detail latent features with added perturbation, and then use an image decoder to decode them to obtain the reconstructed images corresponding to the original structural latent features with added perturbation and the reconstructed images of the original detail latent features with added perturbation, respectively.
[0035] In some embodiments, the change in structural consistency score between the original structural latent features and the instruction vector representation before and after the perturbation is calculated, including: Calculate the first CLIP score between the original structural latent features before perturbation and the instruction vector representation, and the second CLIP score between the original structural latent features and the instruction vector representation after perturbation; Calculate the difference between the first CLIP score and the second CLIP score; Calculate the change in detail consistency score between the original latent detail features and the instruction vector representation before and after perturbation, including: Calculate the third CLIP score between the original latent details feature and the instruction vector representation before perturbation, and the fourth CLIP score between the original latent details feature and the instruction vector representation after perturbation; Calculate the difference between the third CLIP score and the fourth CLIP score.
[0036] In this embodiment, for the original structural latent features before perturbation, a first CLIP score is calculated between the original structural latent features and the instruction vector representation. For the original structural latent features after perturbation, a second CLIP score is calculated between the original structural latent features and the instruction vector representation. The difference between the first CLIP score and the second CLIP score is calculated as the change in the structural consistency score between the original structural latent features and the instruction vector representation before and after perturbation, which can be expressed as follows: ,in, C 0 represents the original structural latent features before the perturbation was added. I ( C 0) is the reconstructed image after restoring the original structural latent features before perturbation back to the image space. To add the perturbation to the original structural latent features. To restore the original structural latent features, after perturbation, to the reconstructed image in image space, T This is the instruction text.
[0037] For the original latent detail features before perturbation, calculate the third CLIP score between the original latent detail features and the instruction vector representation. For the original latent detail features after perturbation, calculate the fourth CLIP score between the original latent detail features and the instruction vector representation. Calculate the difference between the third and fourth CLIP scores as the change in detail consistency score between the original latent detail features and the instruction vector representation before and after perturbation, which can be expressed as: ,in, F 0 represents the original detailed latent features before adding perturbation. I ( F 0) is the reconstructed image after restoring the original latent details before perturbation back to the image space. To add the original detailed latent features after perturbation. To restore the original detailed latent features after perturbation back to the reconstructed image in image space, T This is the instruction text.
[0038] S106: Generate the structural response features of the original structural latent features after propagation in the graph structure, and the detailed response features of the original detailed latent features after propagation in the graph structure; In this embodiment, a graph structure module is used to construct graph structure relationships based on the feature similarity between each image patch of the original structural latent features and the original detail latent features. Semantic attention response, difference supervision response and sensitivity response are propagated and smoothed on the graph structure relationship to enhance the spatial consistency, coherence and stability of the editing region prediction results.
[0039] In some embodiments, the generation of structural response features of the original structural latent features after graph structure propagation, and the generation of detail response features of the original detail latent features after graph structure propagation, includes: The feature map composed of the original structural latent features is divided into multiple structural image blocks. The structural similarity between the structural image blocks is calculated. The structural similarity is used to form a structural adjacency matrix. Based on the structural adjacency matrix, the structural response features after the graph structure is propagated are generated. The feature map, composed of the original latent detail features, is divided into multiple detail image blocks. The detail similarity between detail image blocks is calculated, and a detail adjacency matrix is constructed from the detail similarity. The detail response features after graph structure propagation are generated based on the detail adjacency matrix.
[0040] In this embodiment, for the original structural latent features, the feature map composed of the original structural latent features is divided into multiple structural image patches, the structural similarity between the structural image patches is calculated, and a structural adjacency matrix is constructed from the structural similarity. W This structural adjacency matrix reflects the structural correlation strength between different regions in space. Similarly, for the original latent detail features, the feature map composed of the original latent detail features is divided into multiple detail image patches, and the detail similarity between the detail image patches is calculated. The detail similarity is then used to construct the detail adjacency matrix. W This detailed adjacency matrix reflects the semantic association strength between different regions in the space.
[0041] Matrix multiplication is used to pass the response value of each structural or detail image block to other related image blocks based on similarity. By iteratively updating features, the updated features absorb the propagation information from neighboring nodes (as in the first term of Equation (1)) and retain the original input features through reference terms (as in the second term of Equation (1), preventing information from being overly smoothed or lost during propagation. Propagation through graph structures can enhance spatial consistency. Moreover, by breaking through the limitations of local receptive fields, the editing region can be corrected globally based on semantic similarity, thereby making the boundaries of the heatmap smoother, avoiding isolated noise regions, and improving the coherence of the editing region.
[0042] The graph structure propagation iteration process is represented as: (1) in, H (t) For the first t The feature response state at the next iteration. At the initial time, H (0) The initial feature response state represents the change in structural consistency score for the original structural latent features. S C For the original detailed latent features, the change in detailed consistency score is...S F . The propagation factor is a constant between 0 and 1, used to balance the weight between using neighborhood information for propagation and maintaining the initial feature response state. H (t+1) The updated feature response state is obtained after propagation through the graph structure. After iterative stabilization, the result for the original structural latent features is the structural response feature, and the result for the original detail latent features is the detail response feature.
[0043] S107: Generate a structural editing region based on structural location features, structural differences between the original structural latent features and the target structural latent features, structural consistency score changes, and structural response features; In this embodiment, the latent features of the original structure are calculated. C 0 and target structural latent features C Structural differences between 1 = C 1- C 0. Based on the edit region prediction module, the structural editing region at the structural level is determined according to the established structural location features, structural differences, changes in structural consistency scores, and structural response features. Specifically, a learnable weighted fusion is performed based on the structural location features and their weights, structural differences and their weights, changes in structural consistency scores and their weights, and structural response features and their weights to generate the structural editing region. The method is as follows: (2) in, A attn For structural location features (i.e. A C ), Due to structural differences, S C This represents the change in structural consistency score. G C For structural response characteristics, For normalization operations (e.g., using the Sigmoid mapping function). , , , These are the weights of the corresponding items.
[0044] S108: Generate a detail editing region based on detail localization features, detail differences between original and target detail latent features, detail consistency score changes, and detail response features; Calculate the original detailed latent features F 0 and target details and latent features F Structural differences between 1 =F 1- F 0. Based on the edit region prediction module, the detailed editing region is determined according to the established detailed location features, detailed differences, changes in detailed consistency scores, and detailed response features. Specifically, a learnable weighted fusion is performed based on the detailed location features and their weights, the detailed differences and their weights, the changes in detailed consistency scores and their weights, and the detailed response features and their weights to generate the detailed editing region. The method is as follows: (3) in, A attn For detailed feature localization (i.e.) A F ), For minor differences, S F The change in the score for consistency in details. G F For detailed response features, , , , These are the weights of the corresponding terms. Therefore, the structural editing region and detail editing region generated according to formulas (2) and (3) are in the form of heatmaps (corresponding to...). Figure 2 The heatmaps (including structural and detail levels) represent the editing intensity at corresponding spatial locations, allowing for precise positioning of the area in the original image where editing commands are applied.
[0045] In some embodiments, structural differences are considered. and differences in details As a supervisory signal during the training process of the semantic image editing model, structural and detail differences are normalized to construct a true editing response distribution, which is used to supervise the prediction results of structural and detail editing regions. By using L2 loss and KL distribution matching constraints, the predicted structural and detail editing regions are made to statistically approximate the true editing difference distribution. This supervision compensates for the deficiency of attention in accurately corresponding to the "actually changed areas," thereby improving the accuracy of the predicted editing regions.
[0046] In some approaches, a learnable weighted fusion model can be used to fully integrate multi-source evidence, including structural location features, structural differences, changes in structural consistency scores, and structural response features. f fuse Encoding with editing instructions V T The weights of each corresponding item are dynamically generated for each condition, enabling the model to autonomously choose to rely more on structural evidence, detailed evidence, or semantic attention evidence based on the type of editing instruction.
[0047] For example, when the editing instruction is "replace the cat with a tiger," which is biased towards structural evidence, the model will increase the structural response features. G C weight This ensures that changes to object boundaries are accurately captured. When the editing command is "change the clothing to silk material," which leans towards detail-level evidence, the model will dynamically increase the detail consistency score. S F weight This is because such edits involve minimal structural changes, primarily focusing on detail channels. When the edit instruction is "place a ball on the grass," which leans towards semantic attention evidence, the model will increase the cross-modal attention weights. , Through structural positioning features A C Detailed positioning features A F This helps to determine spatial locations in the original image that are not significantly different but semantically related.
[0048] S109: Based on the original structural latent features, original detail latent features, instruction vector representation, structural editing region, and detail editing region, the residual generation network is used to predict structural residuals and detail residuals; In this embodiment, a residual generation network is constructed based on a multilayer perceptron to generate the original structural latent features. C 0. Instruction Vector Representation V T Structure editing area H C Input the residual generation network, and output the predicted structural residuals from the residual generation network. The structural residual mainly describes the changes in object shape, scale, and spatial relationship between the original image and the target image.
[0049] Original details and latent features F 0. Instruction Vector Representation V T Detailed editing area H F Input the residual generation network, and output the predicted detailed residuals from the residual generation network. The detail residual mainly describes the amount of change in texture, material, and local visual attributes between the original image and the target image.
[0050] S110: Generate edited structural latent features based on the original structural latent features, structural residuals, and structural editing regions; generate edited detailed latent features based on the original detailed latent features, detailed residuals, and detailed editing regions. In this embodiment, after determining the structural residuals and the structural editing region, the editing module injects the structural residuals into the structural editing region using an element-wise weighted method based on the original structural latent features, resulting in the edited structural latent features, represented as follows: (4) in, C edit This refers to the edited structural latent features.
[0051] Similarly, using the editing module, based on the original latent detail features, the detail residuals are injected into the detail editing area through element-wise weighting to obtain the edited latent detail features, represented as: (5) in, F edit For the edited details and latent features.
[0052] According to formulas (4) and (5), the structural residual is applied to the structural editing area and the detail residual is applied to the detail editing area, respectively. This enables decoupled editing of structural and detail features, avoiding the impact on different levels of editing and non-editing areas, and achieving precise local editing effects.
[0053] S111: Reconstruct the edited image based on the edited structural latent features and detail latent features.
[0054] In this embodiment, after determining the edited structural latent features and detail latent features, an image decoder is used to perform structural and detail fusion reconstruction based on the edited structural latent features and detail latent features to generate an edited image. During the training phase, after the model obtains the edited image, it calculates the difference between the edited image and the target image in the triplet using a loss function, and optimizes the model parameters based on the difference.
[0055] like Figure 3 As shown, the model is trained according to the above steps to obtain the trained semantic image editing model. The trained semantic image editing model is then used to implement the function of editing images according to editing instructions. In the application of the semantic image editing model, the structural difference and detail difference in formulas (2) and (3) can be taken as 0.
[0056] In some approaches, the image decoder generates geometric and layout information of the image primarily based on structural latent features during the reconstruction process. At the same time, it supplements high-frequency texture information using detail latent features and strictly follows the invertibility constraint of wavelet transform. Under the premise of maintaining frequency domain consistency, the image decoder fuses and reconstructs the edited structural latent features and detail latent features to obtain an edited image that conforms to the semantics of the edited text.
[0057] This application provides a semantic image editing method. It extracts original latent features containing structural and detail information from the original image, decomposes these features into original structural latent features and original detail latent features, decoupling structural and detail-level features to avoid interference from mixed latent features and improve editing stability. Based on the decoupled original structural and detail latent features, it integrates cross-modal attention, difference supervision, frequency domain sensitivity analysis, and graph structure propagation information to predict the corresponding structural and detail editing regions. This enables automatic and accurate prediction of editing regions without manual intervention, improving editing accuracy and interpretability. Furthermore, it predicts structural residuals based on original structural latent features, structural editing regions, and editing instructions; and predicts detail residuals based on original detail latent features, detail editing regions, and editing instructions. Through decoupling control of structural and detail residuals, independent editing operations can be implemented at different semantic levels, significantly reducing the risk of erroneous editing of non-editable regions and effectively improving editing accuracy.
[0058] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.
[0059] It should be noted that the above description describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims may be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0060] like Figure 4 As shown, this application provides a semantic image editing apparatus, including: The first extraction module is used to extract corresponding original latent features and target latent features from the original image and the target image, respectively; wherein, the target image is obtained by editing the original image according to the editing instruction text; The encoding module is used to encode the editing command text to obtain the command vector representation; The second extraction module is used to extract the original structural latent features and original detail latent features, as well as the target structural latent features and target detail latent features from the original latent features and the target latent features, respectively. The alignment module is used to generate structural localization features aligned with the original structural latent features and detailed localization features aligned with the original detailed latent features, respectively, based on the cross-modal model. The sensitivity scoring module is used to add perturbations to the original structural latent features and the original detail latent features respectively, and to calculate the change in structural consistency score between the original structural latent features and the instruction vector representation before and after the perturbation, as well as the change in detail consistency score between the original detail latent features and the instruction vector representation before and after the perturbation. The graph structure propagation module is used to generate the structural response features of the original structural latent features after graph structure propagation, and the detailed response features of the original detailed latent features after graph structure propagation. The structural region generation module is used to generate structural editing regions based on structural location features, structural differences between the original structural latent features and the target structural latent features, structural consistency score changes, and structural response features. The detail region generation module is used to generate a detail editing region based on detail localization features, detail differences between the original detail latent features and the target detail latent features, detail consistency score changes, and detail response features. The residual prediction module is used to predict structural residuals and detail residuals using a residual generation network based on the original structural latent features, original detail latent features, instruction vector representation, structural editing region, and detail editing region. The feature editing module is used to generate edited structural latent features based on the original structural latent features, structural residuals, and structural editing regions, and to generate edited detailed latent features based on the original detailed latent features, detailed residuals, and detailed editing regions. The image generation module is used to reconstruct the edited image based on the edited structural latent features and detail latent features.
[0061] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.
[0062] The apparatus described above is used to implement the corresponding methods in the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0063] Figure 5 This embodiment illustrates a more specific hardware structure of an electronic device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.
[0064] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.
[0065] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.
[0066] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.
[0067] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0068] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.
[0069] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.
[0070] The electronic devices described above are used to implement the corresponding methods in the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0071] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.
[0072] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.
[0073] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.
[0074] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.
[0075] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this disclosure.
Claims
1. A semantic image editing method, characterized in that, include: Extract the corresponding original latent features and target latent features from the original image and the target image respectively; wherein, the target image is obtained by editing the original image according to the editing instruction text; The edit command text is encoded to obtain a command vector representation; Extract the original structural latent features and original detail latent features, as well as the target structural latent features and target detail latent features, from the original latent features and the target latent features, respectively. Based on the cross-modal model, structural localization features aligned with the original structural latent features and detail localization features aligned with the original detail latent features are generated respectively. Perturbations are added to the original structural latent features and the original detail latent features respectively. The changes in structural consistency scores between the original structural latent features and the instruction vector representation before and after the perturbations are calculated, as well as the changes in detail consistency scores between the original detail latent features and the instruction vector representation before and after the perturbations. Generate the structural response features of the original structural latent features after graph structure propagation, and the detailed response features of the original detailed latent features after graph structure propagation; Based on the structural positioning features, the structural differences between the original structural latent features and the target structural latent features, the structural consistency score change, and the structural response features, a structural editing region is generated. Based on the detailed localization features, the detailed differences between the original detailed latent features and the target detailed latent features, the change in detailed consistency score, and the detailed response features, a detailed editing region is generated. Based on the original structural latent features, original detail latent features, instruction vector representation, structural editing region, and detail editing region, a residual generation network is used to predict structural residuals and detail residuals. Based on the original structural latent features, structural residuals, and structural editing regions, edited structural latent features are generated; based on the original detail latent features, detail residuals, and detail editing regions, edited detail latent features are generated. The edited image is reconstructed based on the edited structural latent features and detail latent features.
2. The method according to claim 1, characterized in that, The extraction of original structural latent features and original detail latent features, target structural latent features and target detail latent features from the original latent features and target latent features respectively includes: The original latent features are subjected to discrete wavelet transform processing to obtain the low-frequency structure and high-frequency details corresponding to the original image. The low-frequency structure is used as the original structural latent feature, and the high-frequency detail is used as the original detail latent feature. The target latent features are subjected to discrete wavelet transform processing to obtain the low-frequency structure and high-frequency details corresponding to the target image. The low-frequency structure is used as the target structural latent feature, and the high-frequency details are used as the target detail latent feature.
3. The method according to claim 1, characterized in that, The generated structural latent features and their structural response features after graph structure propagation, and the generated detailed latent features and their detailed response features after graph structure propagation, include: The feature map composed of the original structural latent features is divided into multiple structural image blocks. The structural similarity between the structural image blocks is calculated. The structural similarity is used to form a structural adjacency matrix. Based on the structural adjacency matrix, the structural response features after graph structure propagation are generated. The feature map composed of the original latent detail features is divided into multiple detail image blocks. The detail similarity between detail image blocks is calculated, and the detail similarity is used to form a detail adjacency matrix. The detail response features after graph structure propagation are generated based on the detail adjacency matrix.
4. The method according to claim 3, characterized in that, The graph structure propagation iteration process is represented as follows: （1） in, H (t) For the first t The feature response state at the next iteration; W This refers to the structural adjacency matrix or the detail adjacency matrix; H (0) The initial feature response state is the change in structural consistency score for the original structural latent feature and the change in detail consistency score for the original detail latent feature. As a propagation factor; H (t+1) The updated feature response state is obtained after propagation through the graph structure. After iterative stabilization, the result of the original structural latent feature is the structural response feature, and the result of the original detail latent feature is the detail response feature.
5. The method according to claim 1 or 4, characterized in that, The change in structural consistency score between the original structural latent features and the instruction vector representation before and after the perturbation includes: Calculate the first CLIP score between the original structural latent features before perturbation and the instruction vector representation, and the second CLIP score between the original structural latent features and the instruction vector representation after perturbation; Calculate the difference between the first CLIP score and the second CLIP score; Calculate the change in detail consistency score between the original latent detail features and the instruction vector representation before and after the perturbation, including: Calculate the third CLIP score between the original latent detail features and the instruction vector representation before perturbation, and the fourth CLIP score between the original latent detail features and the instruction vector representation after perturbation; Calculate the difference between the third CLIP score and the fourth CLIP score.
6. The method according to claim 1, characterized in that, Based on the structural location features, the structural differences between the original and target structural latent features, the structural consistency score change, and the structural response features, a structural editing region is generated. The method is as follows: （2） in, A attn For structural positioning features, Due to structural differences, S C This represents the change in structural consistency score. G C For structural response characteristics, For normalization operations, , , , These are the weights of the corresponding items.
7. The method according to claim 1, characterized in that, Based on the detailed localization features, the detailed differences between the original and target detailed latent features, the change in detailed consistency score, and the detailed response features, a detailed editing region is generated. The method is as follows: （3） in, A attn To locate features for details, For minor differences, S F The change in the score for consistency in details. G F For detailed response features, , , , These are the weights of the corresponding items.
8. The method according to claim 1, characterized in that, Based on the original structural latent features, structural residuals, and structural editing regions, the edited structural latent features are generated using the following method: （4） in, C 0 represents the original structural latent feature. H C This is the structure editing area. For structural residuals, C edit This refers to the edited structural latent features.
9. The method according to claim 1, characterized in that, Based on the original latent detail features, detail residuals, and detail editing regions, the edited latent detail features are generated using the following method: （5） in, F 0 represents the original detail latent feature. H F For detailed editing area, For minor imperfections, F edit For the edited details and latent features.
10. A semantic image editing device, characterized in that, include: The first extraction module is used to extract corresponding original latent features and target latent features from the original image and the target image, respectively; wherein, the target image is obtained by editing the original image according to the editing instruction text; The encoding module is used to encode the editing instruction text to obtain an instruction vector representation; The second extraction module is used to extract the original structural latent features and original detail latent features, the target structural latent features and the target detail latent features from the original latent features and the target latent features, respectively. The alignment module is used to generate, based on the cross-modal model, structural localization features aligned with the original structural latent features and detail localization features aligned with the original detail latent features. The sensitivity scoring module is used to add perturbations to the original structural latent features and the original detail latent features respectively, and calculate the change in structural consistency score between the original structural latent features and the instruction vector representation before and after the perturbation, as well as the change in detail consistency score between the original detail latent features and the instruction vector representation before and after the perturbation. The graph structure propagation module is used to generate the structural response features of the original structural latent features after graph structure propagation, and the detailed response features of the original detailed latent features after graph structure propagation. The structural region generation module is used to generate a structural editing region based on the structural positioning features, the structural differences between the original structural latent features and the target structural latent features, the structural consistency score change, and the structural response features. The detail region generation module is used to generate a detail editing region based on the detail localization features, the detail differences between the original detail latent features and the target detail latent features, the change in detail consistency score, and the detail response features. The residual prediction module is used to predict structural residuals and detail residuals using a residual generation network based on the original structural latent features, original detail latent features, instruction vector representation, structural editing region, and detail editing region. The feature editing module is used to generate edited structural latent features based on the original structural latent features, structural residuals, and structural editing regions, and to generate edited detail latent features based on the original detail latent features, detail residuals, and detail editing regions. An image generation module is used to reconstruct the edited image based on the edited structural latent features and detail latent features.