Image editing method and device, equipment and storage medium

Abstract

The invention relates to the technical field of computer vision, and discloses an image editing method and device, equipment and a storage medium, and the method comprises the steps: obtaining an input image corresponding to a text prompt in response to an editing request of the text prompt; generating a semantic mask based on a text prompt and an input image, wherein the semantic mask is used for identifying a target editing area matched with the text prompt in the input image; respectively processing the text prompt and the input image to generate a text embedding and noise image containing semantic information; inputting the semantic mask, the text embedding and the noise image into a noise prediction network, and outputting a noise prediction result; and adjusting the noise image according to the noise prediction result, and generating an edited image matched with the text prompt semantics. The target editing area is accurately positioned through the semantic mask, and the semantic consistency control of the editing area is realized in combination with a noise prediction adjustment mechanism, so that a large amount of labeled data or model retraining is not needed, and the accuracy of image editing is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image editing method, apparatus, device and storage medium. Background Technology

[0002] In recent years, text-driven image editing technology has rapidly developed in the field of computer vision, relying on artificial intelligence and deep learning, and has become a highly promising research direction. Among them, the diffusion model has shown excellent performance in image generation and editing tasks. However, when using the diffusion model to edit images based on natural language instructions, accurately determining the target region and controlling the editing shape still pose significant challenges. Without auxiliary information, relying solely on text descriptions can easily lead to problems such as "color overflow" and "text misalignment," resulting in inconsistent colors in the edited area and a mismatch between the semantics and the target area. Summary of the Invention

[0003] The main objective of this application is to provide an image editing method, apparatus, device, and storage medium, which aims to solve the technical problem that existing diffusion models rely solely on text descriptions when editing images, resulting in a mismatch between semantics and the target region.

[0004] To achieve the above objectives, this application proposes an image editing method, which includes: In response to an editing request from a text prompt, an input image corresponding to the text prompt is obtained; A semantic mask is generated based on the text prompt and the input image, and the semantic mask is used to identify the target editing region in the input image that matches the text prompt; The text prompt and the input image are processed separately to generate a text embedding containing semantic information and a noisy image; The semantic mask, the text embedding, and the noise image are input into the noise prediction network, and the noise prediction result is output. The noise image is adjusted based on the noise prediction results to generate an edited image that semantically matches the text prompt.

[0005] Optionally, the step of generating a semantic mask based on the text prompt and the input image, wherein the semantic mask is used to identify a target editing region in the input image that matches the text prompt, includes: Based on the text prompts, semantic segmentation is performed on the input image to obtain multiple semantic regions and corresponding initial masks; The relevance score between each semantic region and the text prompt is calculated using the initial mask; Based on the relevance score, target semantic categories that match the semantics of the text prompt are selected; Aggregate all instances of the initial mask for the target semantic category and perform fusion optimization to generate an optimized semantic mask; The optimized semantic mask is post-processed to generate a semantic mask.

[0006] Optionally, the step of aggregating all instances of the initial mask for the target semantic category and performing fusion optimization processing to generate an optimized semantic mask includes: Obtain instances of the initial mask belonging to the target semantic category in the input image to obtain a mask set; The weights of each instance in the mask set are calculated based on weighted normalization. The weights are weighted and aggregated with instances of the initial masks in the mask set to obtain the optimized semantic mask.

[0007] Optionally, the step of post-processing the optimized semantic mask to generate a semantic mask includes: Based on a preset threshold, the optimized semantic mask is converted into an initial segmentation mask through binarization. Based on morphological operations, the small noise regions of the initial segmentation mask are removed by opening operations, and the mask edges of the initial segmentation mask are smoothed by closing operations to obtain the target segmentation mask. The largest connected region in the target segmentation mask is determined, and the target segmentation mask is optimized based on the connected region to obtain a semantic mask.

[0008] Optionally, the step of inputting the semantic mask, the text embedding, and the noisy image into a noise prediction network and outputting a noise prediction result includes: Determine the time step corresponding to the noisy image; The semantic mask, the text embedding, the time step, and the noise image are input into the hierarchical feature extraction module of the noise prediction network to perform hierarchical feature extraction and reconstruction processing on the noise image, and output the corrected noise prediction result.

[0009] Optionally, the step of inputting the semantic mask, the text embedding, the time step, and the noisy image into the hierarchical feature extraction module of the noise prediction network, performing hierarchical feature extraction and reconstruction processing on the noisy image, and outputting the corrected noise prediction result includes: The semantic mask is mapped to a mask guidance tensor by dimensional expansion; The noise prediction network is used to predict the conditional noise and unconditional noise of the noisy image; Based on the mask guidance tensor and the preset image guidance coefficients, the difference between the conditional noise and the unconditional noise is weighted and adjusted, and the corrected noise prediction is calculated based on the adjusted noise difference to obtain the noise prediction result.

[0010] Optionally, the step of processing the text prompt and the input image respectively to generate a text embedding containing semantic information and a noisy image includes: The input image is processed using an unconditional model to generate an image embedding; The text prompt, the image embedding, and the preset unconditional text input condition generation model are used to generate a text embedding containing semantic information. Based on the semantic mask, noise is gradually injected into the input image within the masked area according to a preset noise scheduling strategy to obtain a noisy image.

[0011] Furthermore, to achieve the above objectives, this application also proposes an image editing apparatus, the image editing apparatus comprising: An image acquisition module is used to acquire an input image corresponding to the text prompt in response to an editing request from the text prompt; A mask generation module is used to generate a semantic mask based on the text prompt and the input image, wherein the semantic mask is used to identify the target editing region in the input image that matches the text prompt; The noise addition module is used to process the text prompt and the input image respectively to generate a text embedding containing semantic information and a noise image; The noise prediction module is used to input the semantic mask, the text embedding, and the noise image into the noise prediction network and output the noise prediction result; The noise adjustment module is used to adjust the noise image based on the noise prediction result and generate an edited image that matches the semantics of the text prompt.

[0012] In addition, to achieve the above objectives, this application also proposes an image editing device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image editing method as described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the image editing method described above.

[0014] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the image editing method described above.

[0015] This application discloses a method for responding to an editing request based on a text prompt, acquiring an input image corresponding to the text prompt; generating a semantic mask based on the text prompt and the input image, wherein the semantic mask is used to identify the target editing region in the input image that matches the text prompt; processing the text prompt and the input image separately to generate a text embedding containing semantic information and a noise image; inputting the semantic mask, text embedding, and noise image into a noise prediction network to output a noise prediction result; and adjusting the noise image according to the noise prediction result to generate an edited image that semantically matches the text prompt. By accurately locating the target editing region through a semantic mask and combining it with the noise prediction adjustment mechanism of the prediction model, semantic consistency control of the editing region is achieved. This allows for dynamic adaptation to editing needs in different scenarios without requiring a large amount of labeled data or model retraining, significantly improving the accuracy and controllability of image editing. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the first embodiment of the image editing method of this application; Figure 2 This is a framework diagram for semantically aware multimodal collaborative editing in this application; Figure 3 This is a flowchart illustrating the second embodiment of the image editing method of this application; Figure 4 This is a flowchart of the adaptive weighted mask fusion strategy of this application; Figure 5 This is a flowchart illustrating the third embodiment of the image editing method of this application; Figure 6 This is a framework diagram of the dual-scale noise control in this application; Figure 7 This is a schematic diagram of the module structure of the image editing device according to an embodiment of this application; Figure 8This is a schematic diagram of the device structure of the hardware operating environment involved in the image editing method in the embodiments of this application.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] In recent years, text-driven image editing technology has rapidly developed in the field of computer vision, relying on artificial intelligence and deep learning, and has become a highly promising research direction. Among them, diffusion models have shown excellent performance in image generation and editing tasks. The Prompt-to-Prompt model achieves fine-grained control over the generation process of the diffusion model by locally modifying prompt words, allowing users to guide changes in specific regions of the image by adjusting text descriptions. The InstructPix2Pix method proposes an end-to-end instruction-driven image editing framework, which learns the mapping from the input image and text instructions to the target editing result through a conditional diffusion model. This method does not require predefined editing operations and can directly modify images based on natural language instructions. At the same time, another multi-task learning framework can automatically adjust the generation strategy according to different restoration scenarios, such as handling image completion under different textures and lighting conditions. PowerPaint has shown better detail preservation capabilities than traditional methods in complex scene restoration. Although the above methods have made significant progress in the field of text-based image editing, the following core challenges still exist: 1. Inaccurate target region localization: Existing methods have difficulty accurately defining the scope of the editing operation. For example, when the user instruction is "change the color of the car," the model may not be able to accurately distinguish the vehicle area from the background in the image, leading to color overflow or incorrect editing. Prompt-to-Prompt and InstructPix2Pix primarily rely on global alignment of text and image, lacking a fine-grained control mechanism for local regions. 2. Insufficient generalization ability in complex scenes: When processing images containing multiple semantic objects or complex backgrounds, the editing accuracy of existing models drops significantly. For example, while PowerPaint can generate visually coherent content in image restoration tasks, there is still room for improvement in restoring fine structures (such as text and texture details).

[0023] Therefore, this application provides a semantically aware multimodal collaborative editing technology that achieves accurate recognition of target regions in images without fine-tuning or training. By generating masks that highly match natural language instructions and adjusting noise prediction, the image generation quality is significantly improved, ensuring that the edited region is semantically consistent with the input text prompt.

[0024] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, feature extraction, and program execution functions, such as a computer, or an electronic device capable of performing the above functions. The following description uses an image processing device as an example to illustrate this embodiment and the subsequent embodiments.

[0025] Based on this, embodiments of this application provide an image editing method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the image editing method of this application.

[0026] In this embodiment, the image editing method includes: Step S10: In response to the text prompt's editing request, obtain the input image corresponding to the text prompt.

[0027] It should be noted that text prompts are image editing requests submitted by the user in natural language, received by the image editing device or system. The input image can be the image data to be edited, located and loaded according to the editing requirements; this image can be a local image specified by the user.

[0028] Step S20: Generate a semantic mask based on the text prompt and the input image. The semantic mask is used to identify the target editing region in the input image that matches the text prompt.

[0029] It should be noted that semantic masks use pixel-level annotations (e.g., "1" to represent the target area and "0" to represent the non-target area) to clearly identify the area where editing operations need to be performed. Unlike traditional manually annotated masks, semantic masks are generated by automatically associating text semantics with image content through algorithms, and have the ability to dynamically adapt to different editing needs.

[0030] Specifically, when generating a semantic mask, the input image is first divided into multiple semantic regions (such as "car", "sky", "road" etc.) using a semantic segmentation model, and an initial mask is generated for each region. Then, the semantic relevance score between each region and the text prompt is calculated using a cross-modal model to select the most matching target semantic category (such as "car" in the text prompt). Next, all mask instances under the same category are aggregated, and an optimized semantic mask is generated through weighted normalization and dynamic thresholding algorithms. Finally, morphological operations and connected component analysis are performed to remove noise and retain the largest connected component to obtain the final semantic mask.

[0031] It should be understood that the generated mask serves as the basis for subsequent noise adjustment of the diffusion model, ensuring that editing operations only affect the target area while maintaining the structural integrity of non-target areas, thus achieving local editing without affecting the global scope.

[0032] Step S30: Process the text prompt and the input image respectively to generate a text embedding containing semantic information and a noisy image.

[0033] It is understandable that text embedding is the process of converting text prompts in natural language form into semantic vectors that can be processed by computers through text encoding techniques, while noisy images are intermediate products that add different levels of noise to the input image.

[0034] It should be understood that the initial embedding of the input image is generated through an unconditional model, and then the text prompts, image embeddings and unconditional text input conditional generation model are used to encode the text semantics into high-dimensional vectors using cross-modal mapping (such as the CLIP model), so that the semantic information of the text prompts (such as object category, attributes, actions) is transformed into a numerical representation that the model can understand.

[0035] Understandably, based on the generated semantic mask, noise is injected into the input image within the target area identified by the mask according to a preset noise scheduling strategy (such as gradually increasing the noise intensity), generating a multi-stage noisy image, which provides noisy input for the reverse denoising process of the diffusion model.

[0036] Furthermore, in order to achieve cross-modal semantic association modeling between text prompts and image content, and to ensure that noise only affects the target editing area, thus avoiding unintended editing caused by global noise adjustment in traditional methods, step S30 may include: The input image is processed using an unconditional model to generate an image embedding; the text prompt, the image embedding, and the preset unconditional text input condition generation model are combined to generate a text embedding containing semantic information; based on the semantic mask, noise is gradually injected into the input image within the mask region according to a preset noise scheduling strategy to obtain a noisy image.

[0037] It should be noted that image embedding is a high-dimensional semantic vector generated after encoding the input image using an unconditional model. It contains basic visual features such as object category, structure, and texture, but is not associated with any specific editing intent. Unconditional text is a pre-defined general text input (such as an image) used to contrast with conditional text prompts, assisting the conditional generation model in learning the semantic differences between texts and improving the accuracy of text embedding. The noise scheduling strategy is a pre-defined noise injection rule that defines how noise intensity changes with the time step of the diffusion model (e.g., linear increase, cosine decay), used to control the generation process and editing precision of noisy images.

[0038] It should be understood that traditional methods directly use text encoders to generate text embeddings, lacking semantic alignment with image content, leading to text misalignment. For example, the text prompt "red car" might mistakenly edit other red objects in the image. By using an unconditional model to generate image embeddings and a conditional generation model to integrate text prompts and image embeddings, a cross-modal mapping between text semantics and the actual image content can be established. For example, the unconditional model extracts the visual features of "car" from the input image as an image embedding, and the conditional generation model associates the text prompt "red car" with this embedding, generating a text embedding specific to the car region in the image, avoiding semantic ambiguity.

[0039] In one example, based on the input image x and the text prompt p, a mask m that accurately covers the target editing region is generated through a semantic segmentation process. Specifically, the input image x and the text prompt p are first input into the encoder to generate image features and text embeddings. After joint semantic analysis, the editing region is determined and the mask m is output. Within the mask region, noise is gradually injected into the original image x according to a preset noise scheduling strategy to generate a multi-stage noisy image. .

[0040] Step S40: Input the semantic mask, the text embedding, and the noise image into the noise prediction network and output the noise prediction result.

[0041] It should be noted that the noise prediction network is a core component in the diffusion model (such as U-Net). It is used to predict the noise distribution at the current time step based on the input semantic mask, text embedding, and noisy image, providing an adjustment basis for the reverse denoising process. The noise prediction result is the noise adjustment parameters output by the noise prediction network, which includes predicted values ​​for the noise intensity and distribution in the target region, and is used to guide subsequent noise removal and image feature reconstruction.

[0042] Specifically, firstly, the semantic mask is mapped to a mask-guided tensor through dimensionality expansion. The text embedding maintains a high-dimensional semantic vector form, and the noisy image carries the noise distribution at the current time step. Next, the preprocessed input data is fed into a noise prediction network (such as U-Net). The encoder performs multi-level feature extraction on the noisy image, and combined with the region constraints of the semantic mask and the semantic guidance of the text embedding, the noise distribution features are reconstructed in the decoder stage. The network filters target region features based on the mask-guided tensor, uses the semantic information of the text embedding to correct noise prediction bias, and outputs a three-dimensional noise prediction result: the noise prediction corresponding to the conditional text. Noise prediction for unconditional images and unconditional baseline noise Finally, by balancing the weights of the three factors through a guiding coefficient, a noise prediction update value aligned with the semantics of the text prompt is generated.

[0043] Step S50: Adjust the noise image according to the noise prediction result to generate an edited image that semantically matches the text prompt.

[0044] It should be understood that the edited image is the final image generated by eliminating predicted noise in the noisy image through a reverse denoising process and reconstructing features based on the semantics of the text prompt. The edited image must satisfy the requirement that the editing effect of the target region is consistent with the semantics of the text prompt (such as modification of color, shape, and attributes) and that the structure of the non-target region is intact.

[0045] Understandably, noise is removed layer by layer from the noisy image based on the corrected noise prediction value n, and the pixel features (such as color and texture) of the target region are reconstructed using an image decoder. For example, at time step t, the noisy image is adjusted using n. The pixel values ​​are used to generate a denoised image. After multiple iterations (repeated N time steps), the final output is an edited image x' that matches the semantics of the text prompt, achieving the effect of precise semantic editing of the target region and coherent preservation of the global structure.

[0046] In one example, reference Figure 2 , Figure 2 This is a framework diagram for semantic-aware multimodal collaborative editing in this application. Based on the input image x and the text prompt (Text p), the text prompt is encoded into a high-dimensional text embedding by a text encoder. This is used to guide image generation. The text message is "Change the color of the bench to red".

[0047] The first step is mask generation. Based on the image encoder and text prompts, the input image is semantically segmented to identify different objects or regions, resulting in a segmented image. Then, ADMF (Adaptive Dynamic Mask Fusion) is used to dynamically generate a semantic mask m (Mask) from the segmentation results, which is used to accurately cover the target editing area.

[0048] Next, proceed to step 2: adding noise. Within the masked area, noise is gradually injected into the original image x through the image encoder according to a preset noise scheduling strategy, generating a multi-stage noisy image. .

[0049] Then, step 3 is performed: dual-scale noise adjustment. The semantic mask m and the noisy image are then adjusted. Text is embedded into the Unet network, and the Unet network is used to control noisy images through a region-aware dual-scale noise control framework. Feature extraction is performed, and the noise distribution *n* is predicted at both the local detail layer and the global semantic layer. After multiple iterations and repeating this process for N time steps, noise removal and feature reconstruction are achieved through an image decoder. The final output is an edited image *x'* that semantically matches the text prompt. The Unet network is a convolutional neural network structure used for image segmentation.

[0050] In this embodiment, a method is disclosed that, in response to an editing request from a text prompt, an input image corresponding to the text prompt is acquired; a semantic mask is generated based on the text prompt and the input image, the semantic mask being used to identify the target editing region in the input image that matches the text prompt; the text prompt and the input image are processed separately to generate a text embedding containing semantic information and a noise image; the semantic mask, text embedding, and noise image are input into a noise prediction network, and a noise prediction result is output; the noise image is adjusted according to the noise prediction result to generate an edited image that semantically matches the text prompt. By accurately locating the target editing region through a semantic mask and combining it with the noise prediction adjustment mechanism of the prediction model, semantic consistency control of the editing region is achieved. This allows for dynamic adaptation to editing needs in different scenarios without requiring a large amount of labeled data or model retraining, significantly improving the accuracy and controllability of image editing.

[0051] Reference Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the image editing method of this application. Based on the first embodiment described above, a second embodiment of the image editing method of this application is proposed. In the second embodiment, step S20 includes: Step S201: Perform semantic segmentation on the input image based on the text prompt to obtain multiple semantic regions and corresponding initial masks.

[0052] It should be noted that a semantic region is a set of pixels in an image that have the same semantic label. For example, pixels belonging to the car category constitute a semantic region.

[0053] Understandably, when performing semantic segmentation, the key semantic elements in the text prompt are first extracted. For example, in the text prompt "edit the color of the bench in the park," the target object is parsed as "bench," and the action is "color modification." A semantic segmentation model such as SSA is then called to standardize the input image. Next, the preprocessed input image is fed into the semantic segmentation model. The model extracts image features through a convolutional neural network, clusters pixels into multiple regions based on semantic similarity, assigns a corresponding semantic label to each region, and generates an initial binary mask corresponding to each region.

[0054] Step S202: Calculate the relevance score between each semantic region and the text prompt using the initial mask.

[0055] Understandably, an image is divided into n distinct semantic regions using an SSA-based semantic segmentation framework, with each region assigned a corresponding semantic label. These regions are then compared with a binary mask. Related, Let represent the set of pixels belonging to the i-th semantic category. To evaluate the relevance of each region to the text cue p, a similarity score can be used. Quantify the alignment between the region and the prompt.

[0056] Step S203: Based on the relevance score, select the target semantic category that matches the semantics of the text prompt.

[0057] In one example, the region with the highest Si value was identified as the region most relevant to the text prompt:

[0058]

[0059] in, This represents the set of pixels most relevant to the text prompt, i.e., the target editing area.

[0060] Step S204: Aggregate all instances of the initial mask for the target semantic category and perform fusion optimization processing to generate an optimized semantic mask.

[0061] It should be understood that, in order to improve the overall accuracy and robustness of the mask and to solve the problem of incomplete target area coverage caused by segmentation, the optimal mask is determined... After identifying the corresponding semantic category, it is necessary to collect all mask instances in the image that belong to that semantic category. It may not fully cover the complete semantic range described by the text prompt. Therefore, all elements in the image related to... Mask instances belonging to the same semantic category will be aggregated into a mask set.

[0062] Furthermore, to ensure that high-confidence semantic regions dominate the mask, effectively improving the mask's coverage of multiple instance targets in complex scenes; avoiding semantic ambiguity caused by simple average aggregation, and making the generated mask more accurately reflect the semantic intent of the text prompt, thus enhancing the robustness of mask generation. Step S204 may include: Obtain instances of the initial mask belonging to the target semantic category in the input image to obtain a mask set; calculate the weight of each instance in the mask set based on weighted normalization; and perform weighted aggregation with each instance of the initial mask in the mask set to obtain an optimized semantic mask.

[0063] In one example, all of the images are related to Mask instances belonging to the same semantic category will be aggregated into a mask set M = To ensure accurate integration of these instances, a weighted normalization process is applied, with weights... The calculation is as follows:

[0064] Optimizing the semantic mask (OSM) is achieved by aggregating high-confidence mask instances to form a continuous intensity map.

[0065] Step S205: Post-process the optimized semantic mask to generate a semantic mask.

[0066] Furthermore, to address issues such as irregular edge topology, excessive regional noise, or poor consistency of weighted fusion masks, step S205 may include: Based on a preset threshold, the optimized semantic mask is converted into an initial segmentation mask through binarization; small noise regions of the initial segmentation mask are removed by opening operations according to morphological operations, and the mask edges of the initial segmentation mask are smoothed by closing operations to obtain the target segmentation mask; the largest connected region in the target segmentation mask is determined, and the target segmentation mask is optimized according to the connected region to obtain the semantic mask.

[0067] Understandably, OSM is converted into a final segmentation mask (FSM) through a dynamic thresholding binarization process, which employs dynamic thresholding. and weight Standard deviation The area with a mask value of 1 corresponds to the area height aligned with the text prompt p. During the diffusion process, it actively participates in noise injection and content modification, thereby ensuring that the editing process focuses on the area consistent with the semantics of the prompt.

[0068] After generating the FSM, post-processing is required to address issues such as irregular edge topology, excessive regional noise, or poor coherence of the weighted fusion mask. This process includes morphological operations and connected component analysis. Morphological operations remove small noise regions through opening (fopen) and smooth mask edges through closing (fclose). Considering the scale differences of the target region under different image content, the structuring element of the morphological operation can be set to a 5×5 convolution kernel, which can better balance denoising and edge preservation in most natural images.

[0069] Despite morphological optimization, a masked FSM may still contain multiple discontinuous regions, especially when dealing with multi-instance semantic categories. Suppose several connected regions are identified in the masked FSM, denoted as... Then the largest connected region should be preserved. The final optimized FSM is denoted by m, as shown below:

[0070]

[0071] in, This represents the coordinates of a pixel. It provides a text description for an image that needs editing. Traditional image editing methods may not accurately preserve the details of the original image or may not match the text description. This method allows the text description to precisely match the editing area.

[0072] In one example, reference Figure 4 , Figure 4 This is a flowchart of the adaptive weighted mask fusion strategy of this application. The input image is a flower image, and the text prompt is "flower". After performing semantic segmentation on the input image, multiple categories are obtained, such as "the water", "a green leaf", "a plant", etc. Then, semantic mask matching is performed. Based on the text prompt, the similarity score between each category mask and the text prompt is calculated, and the most matching semantic mask is selected. Next, weights are generated for the selected mask using an adaptive weighted fusion strategy. To ensure the accurate integration of these instances, a weighted normalization process is applied to perform weighted fusion and generate an optimized semantic mask. Finally, a structuring element B (5×5 convolution) is introduced to refine the different masks, ultimately generating a final mask m that can accurately match the text description, achieving high-precision positioning and detail preservation of the image editing area.

[0073] In this embodiment, a semantic segmentation method is disclosed for the input image based on the text prompt, resulting in multiple semantic regions and corresponding initial masks. A relevance score is calculated between each semantic region and the text prompt using the initial masks. Based on the relevance score, target semantic categories that semantically match the text prompt are selected. Instances of all initial masks for the target semantic categories are aggregated and fused for optimization to generate an optimized semantic mask. The optimized semantic mask is then post-processed to generate a final semantic mask. By leveraging a semantic segmentation model and cross-modal relevance calculation, and aggregating mask instances of the same semantic category, semantic-level filtering of the target editing region is achieved. This effectively solves the problem of incomplete target coverage caused by segmentation fragmentation, significantly improving the semantic alignment between the generated semantic mask and the text prompt, and providing precise regional constraints for subsequent editing operations.

[0074] Reference Figure 5 , Figure 5 This is a flowchart illustrating the third embodiment of the image editing method of this application. Based on the second embodiment described above, a third embodiment of the image editing method of this application is proposed.

[0075] In the third embodiment, step S40 includes: Step S401: Determine the time step corresponding to the noisy image.

[0076] It should be noted that the time step is a discrete parameter used in the diffusion model to characterize the noise injection stage. Each time step corresponds to a specific noise intensity and denoising process, and is the core variable controlling the timing of the reverse denoising process.

[0077] Understandably, the reverse denoising process of the diffusion model needs to be processed in reverse order of time steps (e.g., from t=T to t=0). Therefore, once the time step t of the noisy image is determined, its processing stage in the reverse process can be clearly identified. For example, if the noisy image corresponds to t=100 (total time step T=200), it is in the middle stage of reverse denoising and the denoising steps from t=100 to t=99 need to be processed first.

[0078] Step S402: Input the semantic mask, the text embedding, the time step, and the noise image into the hierarchical feature extraction module of the noise prediction network, perform hierarchical feature extraction and reconstruction processing on the noise image, and output the corrected noise prediction result.

[0079] Understandably, conventional classifier-free guidance methods achieve control over the realism and diversity of generated images by balancing conditional and unconditional generation. However, in application scenarios requiring local modifications or semantic guidance, the original classifier-free global control mode reveals a deficiency in coarse control granularity. Introducing a dual-scale control framework with local feature enhancement and global structural constraints, and integrating a triple guidance noise prediction system combining text guidance, image conditions, and classifier guidance, can reduce the impact of editing operations on the original scene structure.

[0080] Furthermore, in order to ensure both semantic consistency between the editing area and the text prompt, and to maintain the structural integrity of non-target areas, an efficient solution is provided for fine-grained image editing (such as color modification and background preservation). Step S402 may include: The semantic mask is mapped to a mask guidance tensor by dimensional expansion; the conditional noise and unconditional noise of the noisy image are predicted using the noise prediction network; the difference between the conditional noise and the unconditional noise is weighted and adjusted based on the mask guidance tensor and preset image guidance coefficients, and the corrected noise prediction is calculated based on the adjusted noise difference to obtain the noise prediction result.

[0081] It should be noted that the mask guidance tensor is a high-dimensional tensor obtained by dimensional expansion of the semantic mask. Its numerical distribution preserves the spatial location information of the target region and is used to selectively adjust the noise distribution of the target region during noise prediction. The tensor value of non-target regions is 0 and does not participate in noise adjustment. Conditional noise is the noise distribution predicted by the noise prediction network under input text prompts, containing semantic guidance information from the text prompts, and is used to guide noise adjustment in the target region. Unconditional noise is the noise distribution predicted by the noise prediction network under unconditional text prompts, reflecting the basic noise characteristics of the image and serving as a benchmark for conditional noise. The image guidance coefficient is a preset hyperparameter used to balance the influence weights of conditional and unconditional noise, control the strength of text semantic guidance, and avoid image distortion caused by over-editing.

[0082] Understandably, based on a classifier-free guidance mechanism, it is first necessary to predict the image noise corresponding to the conditional text. The text prompt indicates the noise corresponding to p. and unconditional noise Subsequently, the pre-set guiding coefficients of the model were integrated. The local feature enhancement tensor obtained by expanding the weighted mask by m dimensions The three types of noise are weighted and fused to obtain the final mixed noise.

[0083] The noise prediction update process is as follows:

[0084] Where n represents the weighted noise prediction, which integrates the final noise information at the current time step t. At a local scale, and Optimize target region features to ensure the editing results are highly consistent with the text semantics. At a global scale, This dual-scale framework constrains the overall structure of the image while maintaining consistency in non-target regions. It employs detailed region control and global consistency constraints, enabling fine-grained adjustments to image content while preserving its overall integrity.

[0085] In one example, reference Figure 6 , Figure 6 This is a framework diagram of the dual-scale noise control proposed in this application. First, the image, conditional text (such as "change the apple into an orange"), and unconditional text are input into the encoding module to generate a text embedding containing semantic information. At each time step t of the diffusion model, this text embedding is compared with the noise image at the corresponding time step. The input is fed into the Unet network, and after hierarchical feature extraction and reconstruction and noise prediction updates, the output noise prediction results in three dimensions are obtained. , , By introducing an adjustable guiding coefficient to dynamically balance the influence weights of textual and image conditions on noise prediction, the model gradually corrects prediction bias under multimodal information constraints, ultimately generating prediction noise that is highly semantically aligned with the input text and accurate in detail. Finally, down-sampling and up-sampling are performed in the Unet network, and noise cancellation and feature reconstruction are achieved through an image decoder, outputting the next noise map that semantically matches the text prompt. In this embodiment, the joint encoding mechanism of conditional text and image, along with an adjustable noise prediction framework, overcomes the technical limitations of traditional diffusion models in controlling semantic details during text-guided image editing.

[0086] In this embodiment, the method for determining the time step corresponding to the noisy image is disclosed. The semantic mask, the text embedding, the time step, and the noisy image are input into the hierarchical feature extraction module of the noise prediction network to perform hierarchical feature extraction and reconstruction processing on the noisy image, and output the corrected noise prediction result. By introducing the time step parameter, the noise prediction can be dynamically adjusted in combination with the temporal characteristics of the diffusion model, realizing the selective control of noise prediction in specific regions. While maintaining the overall image structure, it accurately controls the noise distribution in the target region, solving the problem of insufficient accuracy of traditional diffusion models in local feature editing.

[0087] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the image editing method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0088] This application also provides an image editing device, please refer to... Figure 7 The image editing device includes: Image acquisition module 10 is used to acquire an input image corresponding to the text prompt in response to the editing request of the text prompt; Mask generation module 20 is used to generate a semantic mask based on the text prompt and the input image, wherein the semantic mask is used to identify the target editing region in the input image that matches the text prompt; The noise addition module 30 is used to process the text prompt and the input image respectively to generate a text embedding containing semantic information and a noise image; Noise prediction module 40 is used to input the semantic mask, the text embedding and the noise image into the noise prediction network and output the noise prediction result; The noise adjustment module 50 is used to adjust the noise image according to the noise prediction result and generate an edited image that matches the semantics of the text prompt.

[0089] The image editing apparatus provided in this application, employing the image editing method described in the above embodiments, can solve the technical problem that existing diffusion models rely solely on text descriptions for image editing, leading to a mismatch between semantics and the target region. Compared with the prior art, the beneficial effects of the image editing apparatus provided in this application are the same as those of the image editing method provided in the above embodiments, and other technical features in the image editing apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0090] This application provides an image editing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the image editing method in Embodiment 1 above.

[0091] The following is for reference. Figure 8 The diagram illustrates a structural schematic of an image editing device suitable for implementing embodiments of this application. The image editing device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The image editing device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0092] like Figure 8 As shown, the image editing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the image editing device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the image editing device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows image editing devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0093] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0094] The image editing device provided in this application, employing the image editing method described in the above embodiments, can solve the technical problem that existing diffusion models rely solely on text descriptions for image editing, leading to a mismatch between semantics and the target region. Compared with the prior art, the beneficial effects of the image editing device provided in this application are the same as those of the image editing method provided in the above embodiments, and other technical features of this image editing device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0095] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0096] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0097] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the image editing method described in the above embodiments.

[0098] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0099] The aforementioned computer-readable storage medium may be included in an image editing device; or it may exist independently and not be assembled into an image editing device.

[0100] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by an image editing device, cause the image editing device to perform the image editing method described above.

[0101] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0103] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0104] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described image editing method. This solves the technical problem that existing diffusion models rely solely on text descriptions for image editing, leading to a mismatch between semantics and the target region. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the image editing method provided in the above embodiments, and will not be elaborated upon here.

[0105] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An image editing method characterized by, The image editing method comprises: In response to an editing request of a text prompt, an input image corresponding to the text prompt is obtained; A semantic mask is generated based on the text prompt and the input image, the semantic mask being used to identify a target editing region in the input image that matches the text prompt; The text prompt and the input image are processed respectively to generate a text embedding containing semantic information and a noise image; The semantic mask, the text embedding and the noise image are input into a noise prediction network to output a noise prediction result; The noise image is adjusted according to the noise prediction result to generate an edited image that is semantically matched with the text prompt.

2. The image editing method of claim 1, wherein, The step of generating a semantic mask based on the text prompt and the input image, the semantic mask being used to identify a target editing region in the input image that matches the text prompt, comprises: Performing semantic segmentation on the input image based on the text prompt to obtain a plurality of semantic regions and corresponding initial masks; Calculating a correlation score between each of the semantic regions and the text prompt through the initial masks; Based on the correlation score, a target semantic category that is semantically matched with the text prompt is screened out; All instances of the initial masks of the target semantic category are aggregated and fusion optimization processing is performed to generate an optimized semantic mask; The optimized semantic mask is post-processed to generate a semantic mask.

3. The image editing method of claim 2, wherein, The step of aggregating all instances of the initial masks of the target semantic category and performing fusion optimization processing to generate an optimized semantic mask, comprises: Obtaining instances of the initial masks of the target semantic category in the input image to obtain a mask set; Based on weighted normalization, the weights of each instance in the mask set are calculated; The weights are weighted aggregated with the instances of the initial masks in the mask set to obtain an optimized semantic mask.

4. The image editing method of claim 2, wherein, The step of post-processing the optimized semantic mask to generate a semantic mask, comprises: Based on a preset threshold, the optimized semantic mask is converted into an initial segmentation mask through binarization; According to a morphological operation, small noise regions of the initial segmentation mask are removed through an opening operation, and the mask edges of the initial segmentation mask are smoothed through a closing operation to obtain a target segmentation mask; The largest connected region in the target segmentation mask is determined, and the target segmentation mask is optimized according to the connected region to obtain a semantic mask.

5. The image editing method of claim 1, wherein, The step of inputting the semantic mask, the text embedding and the noise image into a noise prediction network to output a noise prediction result, comprises: Determining the time step corresponding to the noise image; The semantic mask, the text embedding, the time step and the noise image are input into a hierarchical feature extraction module of the noise prediction network for hierarchical feature extraction and reconstruction processing of the noise image, and a corrected noise prediction result is output.

6. The image editing method of claim 1, wherein, The step of inputting the semantic mask, the text embedding, the time step and the noise image into a hierarchical feature extraction module of the noise prediction network for hierarchical feature extraction and reconstruction processing of the noise image, and outputting a corrected noise prediction result, comprises: mapping the semantic mask into a mask-guided tensor through dimension expansion; predicting conditioned noise and unconditioned noise of the noise image by using the noise prediction network; weighting and adjusting the difference between the conditioned noise and the unconditioned noise based on the mask-guided tensor and a preset image-guided coefficient, and calculating a revised noise prediction according to the adjusted noise difference to obtain a noise prediction result.

7. The image editing method according to any one of claims 1 to 6, wherein, The steps of respectively processing the text prompt and the input image to generate the text embedding containing semantic information and the noise image include: processing the input image by using an unconditioned model to generate an image embedding; inputting the text prompt, the image embedding and a preset unconditioned text input condition generation model to generate the text embedding containing semantic information; based on the semantic mask, gradually injecting noise into the input image according to a preset noise scheduling strategy in the mask region to obtain a noise image.

8. An image editing apparatus characterized by comprising: The device comprises: An image acquisition module configured to acquire an input image corresponding to a text prompt in response to an editing request of the text prompt; A mask generation module configured to generate a semantic mask based on the text prompt and the input image, the semantic mask being used to identify a target editing region in the input image that matches the text prompt; A noise adding module configured to respectively process the text prompt and the input image to generate a text embedding containing semantic information and a noise image; A noise prediction module configured to input the semantic mask, the text embedding and the noise image into a noise prediction network to output a noise prediction result; A noise adjustment module configured to adjust the noise image according to the noise prediction result to generate an edited image that matches the semantic information of the text prompt.

9. An image editing apparatus characterized by comprising: The device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the image editing method according to any one of claims 1 to 7.

10. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the image editing method according to any one of claims 1 to 7.

Images