Image editing method and device and electronic equipment

Abstract

Embodiments of the invention provide an image editing method and apparatus, and an electronic device. The method comprises the steps of receiving an input image and a text instruction input by a user; determining an editing operation, an editing object and an editing attribute corresponding to the text instruction through an instruction decomposition model; according to the editing object, determining a region image of a corresponding target region in the input image through a segmentation model, and generating a mask image of the target region; performing image restoration on the mask image to obtain a restored image; performing image editing on the regional image according to the editing operation and / or the editing attribute to obtain an edited image; and fusing the repaired image and the edited image to obtain a fused image, and performing image optimization on the fused image to obtain a target image. According to the image input by the user and the text instruction, the input image can be automatically edited according to the text instruction, and the target image meeting the user requirement is obtained.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and relates to image processing technology, and in particular to an image editing method and apparatus, and electronic equipment thereof. Background Technology

[0002] With breakthroughs in deep learning technology, image editing technology has evolved from simple creation and generation to intelligent editing. In e-commerce, product display images often need to be adjusted to adapt to different display scenarios. Traditional manual image retouching methods suffer from low efficiency, high cost, and limited modification capabilities. While related intelligent image editing methods can achieve basic image editing functions, they struggle to accurately identify user editing commands and achieve corresponding image editing effects based on those commands. Summary of the Invention

[0003] This application provides an image editing method, apparatus, and electronic device that can solve the existing problems of difficulty in accurately recognizing user editing instructions and achieving corresponding image editing effects based on user instructions.

[0004] A first aspect of this application provides an image editing method, comprising: receiving an input image and a text instruction from a user; determining the editing operation, editing object, and editing attributes corresponding to the text instruction through an instruction decomposition model; determining a region image of a target region in the input image based on the editing object through a segmentation model, and generating a mask image of the target region; performing image restoration on the mask image to obtain a restored image; performing image editing on the region image according to the editing operation and / or editing attributes to obtain an edited image; fusing the restored image and the edited image to obtain a fused image; and performing image optimization on the fused image to obtain a target image.

[0005] According to an embodiment of this application, the instruction decomposition model includes a multi-task learning architecture for simultaneously executing an operation classification task, an object recognition task, and an attribute detection task, wherein: the operation classification task obtains the editing operation by performing verb semantic parsing on the text instruction; the object recognition task obtains the editing object by performing object instance segmentation on the text instruction; and the attribute detection task obtains the editing attribute by performing attribute word semantic parsing on the text instruction.

[0006] According to embodiments of this application, the editing operation includes at least one of the following operations: add operation, delete operation, replace operation, zoom operation, and move operation; the editing object includes at least one of the following objects: a single target object and a global object; the editing attribute includes at least one of the following attributes: image color, image texture, and attributes of the editing object.

[0007] According to an embodiment of this application, the step of image inpainting the mask image to obtain a repaired image includes: performing multi-level downsampling on the mask region of the mask image to obtain multiple downsampled images of the mask region; performing content recognition, missing content filling, and style transfer on each of the multiple downsampled images in descending order of resolution to obtain an updated image corresponding to each downsampled image; fusing the multiple updated images corresponding to the multiple downsampled images, and upsampling the fused image to obtain the repaired image.

[0008] According to an embodiment of this application, the step of image editing the region image based on the editing operation and / or editing attributes to obtain an edited image includes: if the editing operation is a replacement operation, inputting the editing attributes corresponding to the region image and the editing object into an image-generated image diffusion model; generating a latent feature map corresponding to the region image through the image-generated image diffusion model based on the editing attributes; and performing diffusion denoising on the latent feature map through the image-generated image diffusion model to obtain an edited image that conforms to the perspective effect of the region image.

[0009] According to an embodiment of this application, fusing the repaired image and the edited image to obtain a fused image includes: performing an edge dilation operation on the edited image, performing an edge erosion operation on the edited image after the edge dilation operation to obtain an edge-modified image corresponding to the edited image; and fusing the edge-modified image and the repaired image to obtain the fused image.

[0010] According to an embodiment of this application, the image optimization of the fused image includes: performing edge repair on the fused image using an edge repair model; and performing global optimization on the edge-repaired fused image.

[0011] According to an embodiment of this application, the global optimization of the edge-repaired fused image includes: tone optimization and style transfer of the edge-repaired fused image.

[0012] A second aspect of this application provides an image editing apparatus, comprising: a data receiving module for receiving image and text commands input by a user; a command decomposition module for determining the editing operation, editing object, and editing attributes corresponding to the text command through a command decomposition model; a region segmentation module for determining a region image of a target region in the input image based on the editing object through a segmentation model, and generating a mask image of the target region; an image restoration module for performing image restoration on the mask image to obtain a restored image; an image editing module for performing image editing on the region image according to the editing operation and / or editing attributes to obtain an edited image; and a fusion optimization module for fusing the restored image and the edited image to obtain a fused image, and performing image optimization on the fused image to obtain a target image.

[0013] A third aspect of this application provides an electronic device, including: a memory and a processor, wherein the processor executes computer-readable instructions stored in the memory to implement the image editing method.

[0014] The image editing method provided in this application can accurately understand the text commands input by the user and quickly decompose the corresponding editing operation type, editing object, and specific editing attribute parameters. Based on a segmentation model, it accurately locates the target region corresponding to the editing object in the image and generates a clear mask image to ensure controllable editing range. Image inpainting technology is used to intelligently fill and smooth the mask image to obtain a repaired image. The target region is then specifically edited according to the editing operation and / or editing attributes, and the edited image is seamlessly merged with the repaired image. A global optimization algorithm is used to adjust the merged image, ensuring that the output target image maintains its original image quality while naturally presenting the user's commands, achieving a highly efficient "what you see is what you get" editing experience. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram illustrating the application environment of an image editing method provided in an embodiment of this application.

[0017] Figure 2 Flowchart of the image editing method provided in the embodiments of this application Figure 1 .

[0018] Figure 3Flowchart of the image editing method provided in the embodiments of this application Figure 2 .

[0019] Figure 4 This is a schematic block diagram of an image editing device provided in an embodiment of this application.

[0020] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0023] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. Unless otherwise specified, the following embodiments and features described herein can be combined with each other.

[0024] Please see Figure 1 This is a schematic diagram illustrating the application environment of an image editing method provided in an embodiment of this application. Figure 1 As shown, user terminal 10 and server 20 communicate via a network. The network can be a wired network or a wireless network. The wired network can be any of a local area network (LAN), a metropolitan area network (MAN), or a wide area network (WAN), and can be any of the following technologies: Wireless Fidelity (Wi-Fi), ZigBee Wireless Networks (ZigBee), Ultra Wideband (UWB), Universal Serial Bus (USB), etc.

[0025] User terminal 10 can be an electronic device such as a mobile phone, tablet computer, multimedia playback device, personal computer (PC), or wearable device. User terminal 10 can be a client with various applications installed, such as educational applications, information applications, information broadcasting applications, live streaming applications, e-commerce applications, etc.

[0026] Server 20 is used to provide background services for applications in user terminal 10. For example, server 20 can be the backend server of the aforementioned e-commerce application, or server 20 can be a computer device. In one embodiment of this application, server 20 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center.

[0027] For example, an e-commerce application is used as an example. A user executes an input image and text command in a client (e.g., user terminal 10) with the e-commerce application installed. User terminal 10 sends the image and text command to server 20. Server 20 determines the editing operation, editing object, and editing attributes corresponding to the text command using an instruction decomposition model. Based on the editing object, server 20 determines the region image of the target area in the input image using a segmentation model, and generates a mask image of the target area. Server 20 performs image restoration on the mask image to obtain a restored image. Server 20 performs image editing on the region image according to the editing operation and / or editing attributes to obtain an edited image. Server 20 merges the restored image and the edited image to obtain a merged image, and performs image optimization on the merged image to obtain the target image. Server 20 sends the target image to user terminal 10, and user terminal 10 presents the target image to the user using a graphical user interface.

[0028] In this way, server 20 can accurately understand the text commands input by the user, quickly decompose the corresponding editing operation type, editing object, and specific editing attribute parameters. Based on the segmentation model, it accurately locates the target region corresponding to the editing object in the image and generates a clear mask image to ensure that the editing range is controllable. Image inpainting technology is used to intelligently fill and smooth the mask image to obtain the repaired image. The target region is then specifically edited according to the editing operation and / or editing attributes, and the edited image is seamlessly merged with the repaired image. A global optimization algorithm is used to adjust the merged image, so that the output target image maintains the original image quality while achieving a natural presentation of the user's commands, achieving a highly efficient "what you see is what you get" editing experience.

[0029] The following will refer to computer program products running on servers (such as...). Figure 1The example shown is server 20. Please refer to [link / reference]. Figure 2 The diagram shown is a flowchart of the image editing method provided in an embodiment of this application. Figure 1 In one embodiment of this application, the following steps are included: Step S201: Receive the input image and text instructions from the user.

[0030] In some embodiments of this application, the server 20 may provide an image upload portal for the user through a graphical user interface. For example, this image upload portal supports uploading images in formats such as JPG and PNG, with a file size limit of ≤5MB. Users can upload images through this portal, and the server 20 receives the uploaded images as input images through this portal.

[0031] In one example, after receiving the input image from the user, the image can be preprocessed and security verified. For instance, preprocessing might include compressing the super-resolution image (such as a 4K image) to 1080p resolution, generating a thumbnail for the user to quickly preview, allowing them to confirm whether they have uploaded an image that needs editing. Security verification might include classifying the image using an NSFW (NotSafeForWork) model to filter images containing inappropriate content and prompting the user to re-upload a different image.

[0032] In one example, the input image can be any user-uploaded image that requires editing. For instance, the input image could be a model showcasing a dress with a background containing a brand watermark; or, for another example, an image of a blue backpack displayed in an outdoor setting. In another example, the user-inputted image could be, for example... Figure 3 The image shown.

[0033] In some embodiments of this application, the server 20 may provide a text input entry point for the user in a graphical user interface. For example, this text input entry point allows the user to input free text, with a character limit of 100 characters. The user can input text through this text input entry point, and the server 20 receives the user's input text as a text instruction through this text input entry point.

[0034] In one example, text instructions are used to direct the user's desired editing of the input image. For example, text instructions might be exemplified as "Remove the watermark in the bottom right corner, keep the clothing texture clear," or "Change the backpack color to coral red, adjust the shoulder strap length to 45cm," etc. In another example, the text instructions entered by the user could be, for example... Figure 3 The image shows "Change the woman's dancing pose".

[0035] Based on the above embodiments, a graphical interface can be used to receive images and text commands, ensuring that the received content is compliant and clear, thus providing a foundation for subsequent accurate image editing.

[0036] Step S202: Determine the editing operation, editing object, and editing attributes corresponding to the text instruction through the instruction decomposition model.

[0037] In some embodiments of this application, the instruction decomposition model includes a multi-task learning architecture for simultaneously performing operation classification, object recognition, and attribute detection tasks, wherein: the operation classification task obtains the editing operation (action) by performing verb semantic parsing on the text instruction; the object recognition task obtains the editing object (object) by performing object instance segmentation on the text instruction; and the attribute detection task obtains the editing attribute (attribute) by performing attribute word semantic parsing on the text instruction.

[0038] In some embodiments of this application, the editing operation includes at least one of the following operations: add operation, delete operation, replace operation, scale operation, and move operation; the editing object includes at least one of the following objects: a single target object and a global object; the editing attribute includes at least one of the following attributes: image color, image texture, and the attribute of the editing object.

[0039] In one example, the multi-task learning architecture adopts a shared-separated architecture. The bottom shared layer is used as a feature extraction layer, and the top layer includes three parallel and separate task branches. The shared layer can be based on a pre-trained BERT-Base model to extract semantic feature vectors of text instructions, for example, the semantic feature vector has a dimension of 768. The three task branches include: (1) Operation classification task branch: outputting the probability distribution of 5 types of editing operations (add, delete, replace, scale, move) through a fully connected layer + Softmax. (2) Object recognition task branch: extracting the editing object and its position information by combining Conditional Random Field (CRF) + Bidirectional Long Short-Term Memory (BiLSTM) network, supporting single object and global object. (3) Attribute detection task branch: outputting the parameter values ​​of attributes such as color, texture, and size through a multi-label classification network, for example, color code #FF6B6B, texture complexity level 1 to 5.

[0040] In one example, when constructing the dataset for training the instruction decomposition model, data annotation can follow conventions such as verb + complement structures (e.g., "replace-background"), entity + scope (e.g., "product-local"), and key-value pairs (e.g., color: #4A90E2). To enhance model robustness, data augmentation strategies can be employed, including synonym substitution (e.g., expanding "delete" to "remove / remove"), attribute value perturbation (e.g., random offsetting of color codes), and instruction recombination (e.g., combining multiple simple instructions to generate complex instructions). Through structured annotation and dynamic augmentation, the robustness of the model can be improved, ensuring that the model can handle diverse user instructions.

[0041] In one example, taking the text instruction "Remove the watermark in the lower right corner while keeping the clothing texture clear," the instruction decomposition model can parse it as follows: the operation classification branch outputs "delete," the object recognition branch locates "watermark (local object)," and the attribute detection branch extracts "texture complexity: 4." For example, if the original image's texture complexity is 3, the edited image's texture complexity remains at 4, thus ensuring texture clarity. In another example, "Change the backpack color to coral red and adjust the shoulder strap length to 45cm" will be decomposed by the instruction decomposition model into "replace + scale" operations, with objects covering "backpack (global object) + shoulder strap (local object)," and attributes including color code #FF7F50 and length 45cm. In yet another example, for the text instruction "Change the woman's dancing pose in the image," the instruction decomposition model can identify the "replace" operation, locate "woman (local object)," and recommend "action type: ballet (attribute)."

[0042] In some embodiments of this application, the instruction decomposition model can parse different types of instruction text in different ways. In one example, for multi-operation instruction text, such as "first enlarge the product by 20%, then replace the background with a spring scene," the instruction decomposition model can generate the operation chain corresponding to the editing operation [scaling (20%), replacing (background)] through the operation sequence prediction module. In another example, for ambiguous instruction text, such as "make the image more vivid," the instruction decomposition model can trigger the attribute recommendation engine to output adjustable parameters for the user to choose from, such as brightness +15%, saturation +20%, contrast +10%, etc. In yet another example, for long text instruction text, such as text containing composite requirements for product details, background cleanup, lighting adjustment, and size output, the instruction decomposition model can extract key entities through a BiLSTM architecture and dynamically allocate operation weights to optimize the processing order.

[0043] In other embodiments of this application, a dynamic weight allocation mechanism can be introduced into the instruction decomposition model during training. This mechanism automatically adjusts the loss function coefficients of the instruction decomposition model based on the complexity of the text instructions. For example, it increases the weight for operation classification for simple instructions and enhances the weight for attribute detection for complex instructions. Furthermore, cross-task feature fusion (e.g., using operation classification results as prior information for attribute detection) can improve the accuracy of the model's attribute parsing. Additionally, a dedicated lexicon can be pre-built for the application environment, such as a pre-built e-commerce-specific lexicon (e.g., an object name lexicon), to improve the model's accuracy in object recognition.

[0044] Based on the above embodiments, the instruction decomposition model adopts a multi-task architecture, integrating the BERT-Base shared layer with three branches: operation classification, object recognition, and attribute detection. It can also combine CRF+BiLSTM and dynamic weight allocation techniques, and improve robustness through structured annotation and enhancement strategies. In e-commerce scenarios, it can achieve accurate parsing of complex instructions, supporting operation chain generation, fuzzy instruction recommendation, and long text entity extraction, providing accurate instruction information for subsequent image editing.

[0045] Step S203: Based on the editing object, determine the region image of the corresponding target area in the input image through the segmentation model, and generate the mask image of the target area.

[0046] In some embodiments of this application, the object to be edited (such as "woman") and the input image can be input into a pre-trained segmentation model. Segmentation models include, but are not limited to, Mask R-CNN (Mask Region-based Convolutional Neural Network), U-Net (U-shaped Network), or CLIPSeg (Contrastive Language-Image Pretraining-based Segmentation Network). The segmentation model outputs pixel-level segmentation results of the target region, generating a mask image with the same resolution as the original image, for example... Figure 3 As shown, the target area is black (0), while other areas remain unchanged.

[0047] In one example, the object to be edited (such as "woman") can be used as a text instruction. The text instruction and the input image are then fed into a pre-trained segmentation model, such as segmentation models including but not limited to Mask R-CNN, U-Net, or CLIPSeg.

[0048] In one example, Mask R-CNN receives a 1024×1024 resolution image of a model wearing a dress, generates candidate bounding boxes using RPN, outputting 2000 candidate regions, and selecting the top 300 high-confidence boxes as candidates. After RoIAlign alignment, the candidate bounding box features are mapped to a 14×14 resolution. A 28×28 mask is generated through four convolutional layers and upsampled to the original image resolution. The output is the mask image.

[0049] In one example, U-Net receives a 512×512 resolution knapsack display image. The encoder downsamples layer by layer to extract multi-scale features; for example, through four layers of convolutional pooling in the encoder, feature maps with sizes of 256×256, 128×128, 64×64, and 32×32 are obtained sequentially. The decoder then upsamples, for example, layer by layer, to 512×512, and combines skip connections to recover details. Finally, a probability map is generated based on the upsampled feature maps using the sigmoid activation function, and a mask image is output.

[0050] In one example, the CLIPSeg model encodes the text instruction "woman" into a 512-dimensional text embedding vector, and the CLIP visual encoder extracts a 2048-dimensional feature map from the input image. The CLIP model then fuses the text embedding with the image features through a Transformer decoder to generate a segmentation prototype. The prototype network combines mask coefficients in a linear combination to generate a pixel-level mask, outputting a mask image.

[0051] In one example, after obtaining the mask image, the segmentation model can perform post-processing optimizations on the mask image. For example, noise can be eliminated and holes filled through morphological operations (such as erosion, dilation, and opening), and edge fit can be improved through CRF refinement. The segmentation threshold can also be dynamically adjusted according to the target object type; for example, a lower threshold is used for small targets (such as watermarks), and a higher threshold is used for large regions (such as backgrounds) to balance recall and precision.

[0052] Based on the above embodiments, the target region image corresponding to the edited object in the input image can be accurately determined by the segmentation model and a mask image can be generated. Furthermore, the mask image can be post-processed and optimized to eliminate noise, fill holes, refine edges, and dynamically adjust the threshold to balance recall and precision.

[0053] Step S204: Perform image restoration on the mask image to obtain the restored image.

[0054] In some embodiments of this application, image inpainting of a mask image to obtain a repaired image includes: performing multi-level downsampling on the mask region of the mask image to obtain multiple downsampled images of the mask region; performing content recognition, missing content filling, and style transfer on each of the multiple downsampled images in descending order of resolution to obtain an updated image corresponding to each downsampled image; fusing the multiple updated images corresponding to the multiple downsampled images, and upsampling the fused image to obtain the repaired image. For example... Figure 3 As shown, the target area in the repaired image is restored to an image with a style consistent with other background areas.

[0055] In one example, when performing multi-level downsampling on the mask region of a mask image, a Gaussian pyramid algorithm can be used. For instance, if the original image resolution is 1024×1024, after the first downsampling, the resolution drops to 512×512; after the second downsampling, it drops to 256×256; and after the third downsampling, it drops to 128×128. Bicubic interpolation is used to maintain the edge sharpness of the downsampled images, with downsampling scale coefficients of 2, 4, and 8 respectively. This results in multiple downsampled images containing different levels of detail; higher resolutions retain more global structure, while lower resolutions focus more on preserving local textures.

[0056] In one example, based on decreasing resolution, high-resolution downsampled images (e.g., 512×512) can be processed using the HRNet (High-Resolution Network) semantic segmentation network to detect object contours and locate the boundaries of missing regions. For medium-resolution downsampled images (e.g., 256×256), gated convolution combined with partial convolution is used to generate padding, utilizing surrounding effective pixels to predict missing regions. For medium-resolution downsampled images (e.g., 128×128), Adaptive Instance Normalization (AdaIN) is applied to transfer original image style features (e.g., extracted via VGG19 (Visual Geometry Group 19-layer Network)) to the padding regions, maintaining color and texture consistency. When processing downsampled images at each resolution, adversarial loss (e.g., using a PatchGAN (Patch-based Generative Adversarial Network) discriminator) can be introduced to improve the realism of the generated images.

[0057] In one example, when fusing multiple updated images corresponding to multiple downsampled images, a Laplacian pyramid fusion strategy can be used. For example, the restoration results at each scale are upsampled to a uniform resolution (e.g., 512×512). Multiple updated images are then fused using weighted fusion to balance structure and detail; for example, a weight of 0.5 is set for large resolution, 0.3 for medium resolution, and 0.2 for small resolution, resulting in a preliminary fused image. Sub-pixel convolution is then used to perform a lossless 2x upsampling of the preliminary fused image, combined with CRF (Conditional Random Field) optimization for edge alignment, resulting in a restored image, for example, with a resolution of 1024×1024.

[0058] Based on the above embodiments, images of different detail levels in the mask region are obtained through multi-scale downsampling. Content recognition, missing data filling, and style transfer are performed sequentially from largest to smallest resolution, combined with adversarial loss to improve realism. Structural details are then balanced using Laplacian fusion weighting, subpixel convolutional upsampling, and CRF optimization of edges. The resulting restored image shows the target area naturally blending with the background area, exhibiting consistent style and precise edges.

[0059] Step S205: Perform image editing on the region image according to the editing operation and / or editing attributes to obtain the edited image.

[0060] In some embodiments of this application, image editing of a region image is performed based on an editing operation and / or editing attributes to obtain an edited image, including: if the editing operation is a replacement operation, inputting the editing attributes corresponding to the region image and the edited object into a raw image diffusion model; generating a latent feature map corresponding to the region image based on the editing attributes through the raw image diffusion model; and performing diffusion denoising on the latent feature map through the raw image diffusion model to obtain an edited image that conforms to the perspective effect of the region image. For example... Figure 3 As shown, the edited image is an image of the woman's region where the action has been altered.

[0061] In one example, the segmented region image is encoded into the latent space using a VAE autoencoder of a graph-generated graph diffusion model to generate an initial latent vector. If the editing operation is a replacement operation, such as "replace the woman's dancing pose in the image," the CLIP text encoder is used to convert the editing attributes (text instructions) into text embedding vectors. The text embeddings are then fused with the initial latent vectors using a cross-attention mechanism to generate a conditionally guided latent feature map.

[0062] In one example, the image-to-image diffusion model performs 200 steps of Gaussian noise addition on the latent feature map in the latent space, with a cosine decay strategy for noise scheduling (initial noise intensity 0.02, final intensity 1.0). A diffusion model using the U-Net architecture (containing 12 residual blocks) progressively denoises the noisy latent feature map, combining textual conditions with edge features of the region image (extracted via ControlNet) at each step. A Spatial Transformation Layer (STN) is introduced during the diffusion process to dynamically adjust the perspective of the generated content to match the original image, with an error threshold of 2 pixels to maintain perspective consistency between the generated image and the region image.

[0063] In one example, the VAE decoder of the image-to-image diffusion model upsamples the denoised latent feature map to the same resolution as the region image, generating a preliminary edited image. A histogram matching algorithm is then applied to align the color distribution of the generated region with the non-masked region of the original image, keeping the color difference ΔE within 2 to ensure color style alignment between the edited image and the region image.

[0064] Based on the above embodiments, after encoding the region image into the latent space, conditional guided features are generated by combining the editing attributes of CLIP text encoding. After 200 steps of cosine noise scheduling for diffusion denoising, edge features are extracted using ControlNet and perspective is dynamically adjusted by STN layer (error ≤ 2 pixels). Finally, an edited image with consistent style is generated by VAE decoding and histogram matching (color difference ΔE < 2).

[0065] Step S206: Fuse the repaired image and the edited image to obtain a fused image. Optimize the fused image to obtain the target image.

[0066] In some embodiments of this application, fusing the repaired image and the edited image to obtain a fused image includes: performing an edge dilation operation on the edited image, performing an edge erosion operation on the edited image after the edge dilation operation to obtain an edge-modified image corresponding to the edited image; and fusing the edge-modified image and the repaired image to obtain a fused image.

[0067] In one example, a morphological dilation operation can be performed on the edited image using a 5×5 rhombus structuring element to eliminate edge holes, with the dilation iteration count set to 2. Then, an erosion operation is performed to restore the original edge structure of the edited image, with the structuring element size adjusted to 3×3 and the iteration count set to 1, generating an edge-modified image. For example... Figure 3 As shown, the edge-modified image is an image of a woman's region with black borders, containing the black border region.

[0068] In one example, the repaired image and the edge-modified image can be α-blended to obtain a fused image, for example, α=0.7. The fusion formula can be expressed as Fused = 0.7*Repair + 0.3*Modified, where Repair represents the repaired image, Modified represents the edge-modified image, and Fused represents the fused image. Figure 3 As shown. In this design, a distance transformation weighting can be introduced into the overlapping area, with the weight linearly decreasing to 0.2 within 5 pixels of the editing area boundary to ensure a smooth edge transition.

[0069] Based on the above embodiments, morphological dilation of the edited image can eliminate edge holes, followed by erosion to restore the original structure and generate an edge-modified image. A weighted fusion strategy enables a seamless transition between the repaired and edited areas, effectively eliminating seam artifacts, reducing black border areas, improving the edge fit of the fused image, and achieving a natural and coherent visual effect.

[0070] In some embodiments of this application, image optimization of the fused image includes: performing edge repair on the fused image using an edge repair model; and performing global optimization on the edge-repaired fused image.

[0071] In one example, when performing edge restoration on a fused image using an edge restoration model, the adaptive thresholding Canny algorithm can be used to detect edges in the fused image, with a threshold ratio of 2:1 (high threshold = low threshold × 2). The U-Net++ model is combined to segment discontinuous edge regions, with an input size of 256×256, and Dice loss is used to optimize the segmentation accuracy of the edge regions. A fully connected conditional random field is constructed, with the weight of the binary potential term set to 0.3 and a spatial kernel scale of 10 pixels. Five iterations of the mean field are performed to correct edge alignment and improve the edge fit index, thereby achieving edge restoration of the fused image.

[0072] In some embodiments of this application, global optimization is performed on the edge-repaired fused image, including: tone optimization and style transfer of the edge-repaired fused image.

[0073] In one example, when performing tone optimization on the edge-repaired fused image, the image can be converted to the Lab color space, and histogram matching can be performed on the a and b channels (refer to the non-mask area of ​​the original image). A dynamic range compression algorithm is used to enhance contrast, and a clipping limit of 2.0 is set to avoid color distortion, thereby achieving color space correction of the edge-repaired fused image.

[0074] In one example, when performing style transfer on the edge-repaired fused image, the conv4_3 layer feature maps (512-dimensional) of the input image and the fused image can be extracted using a pre-trained VGG19. The Gram matrix difference between the feature maps is calculated, and AdaIN is used to transfer the style mean or variance of the original image to the fused image with a transfer strength α=0.8. A Wasserstein GAN (WGAN-GP) discriminator is introduced to optimize the generation of realism; the discriminator network contains 5 convolutional layers (stride 2).

[0075] Based on the above embodiments, high-quality image fusion is achieved through edge restoration models and global optimization. Specifically, adaptive Canny detection and U-Net++ segmentation are used to accurately locate discontinuous edges, and CRF iterative correction is combined to improve edge fit. Tones are optimized through Lab space histogram matching and dynamic range compression algorithms. VGG19 feature extraction and style transfer are used in conjunction with the WGAN-GP discriminator to optimize realism, ultimately generating images with consistent image style, seamless content fusion, and natural visual continuity.

[0076] In other embodiments, if the editing operation is an add operation, delete operation, zoom operation, or move operation, the corresponding operation can be performed on the edited object based on the editing attributes, using a method similar to the replacement operation.

[0077] For example, if the text instruction is "Add XX watermark to the lower right corner", then the editing operation is an add operation, and the editing object is "lower right corner, XX watermark". After step S202, the target area can be determined as the "lower right corner" area, and then "XX watermark" can be added to the "lower right corner" area in the input image according to step S205.

[0078] For example, if the text instruction is "Delete the watermark in the lower right corner", then the editing operation is a deletion operation, and the editing object is "the watermark in the lower right corner". After step S202, the target area can be determined as the "lower right corner" area. Then, according to step S203, the mask image of the "watermark in the lower right corner" area of ​​the input image is determined. Then, according to step S204, the mask image is repaired. The repaired image can present the effect of removing the "watermark in the lower right corner" area of ​​the input image.

[0079] Based on the above embodiments, by parsing the operation type and editing object in the text instructions, the target area can be automatically located and corresponding refined processing can be performed: For the add operation, the location specified in the instruction can be accurately identified as the target area, and the target object to be added can be identified. The target object is then seamlessly embedded in the target area using image synthesis technology. For the delete operation, the target object to be removed is located using object detection technology. After generating the corresponding mask image, an intelligent repair algorithm is used to fill the background, ensuring that the image content is natural and coherent after removal. For scaling or moving operations, similar target area localization and object recognition technologies can be used, combined with spatial transformation algorithms, to achieve proportional scaling or translation transformation of elements. In this way, multiple editing types can be supported through a unified processing framework, ensuring operational accuracy while significantly improving image editing efficiency through automated processes. The final output result maintains a high degree of visual consistency with the original image.

[0080] The image editing method provided in this application can accurately understand the text commands input by the user and quickly decompose the corresponding editing operation type, editing object, and specific editing attribute parameters. Based on a segmentation model, it accurately locates the target region corresponding to the editing object in the image and generates a clear mask image to ensure controllable editing range. Image inpainting technology is used to intelligently fill and smooth the mask image to obtain a repaired image. The target region is then specifically edited according to the editing operation and / or editing attributes, and the edited image is seamlessly merged with the repaired image. A global optimization algorithm is used to adjust the merged image, ensuring that the output target image maintains its original image quality while naturally presenting the user's commands, achieving a highly efficient "what you see is what you get" editing experience.

[0081] Please see Figure 4 This is a schematic diagram of an image editing device provided in an embodiment of this application. It is provided to meet one of the purposes of this application and is a functional embodiment of the image editing method of this application. The image editing device includes: a data receiving module 41, used to receive image and text commands input by a user; a command decomposition module 42, used to determine the editing operation, editing object, and editing attributes corresponding to the text command through a command decomposition model; a region segmentation module 43, used to determine the region image of the corresponding target region in the input image according to the editing object through a segmentation model, and to generate a mask image of the target region; an image restoration module 44, used to perform image restoration on the mask image to obtain a restored image; an image editing module 45, used to perform image editing on the region image according to the editing operation and / or editing attributes to obtain an edited image; and a fusion optimization module 46, used to fuse the restored image and the edited image to obtain a fused image, and to optimize the fused image to obtain a target image.

[0082] Another embodiment of this application also provides an electronic device. Figure 1 The application environment described is merely one example. In other exemplary embodiments, the computer program product implementing the image editing method of this application can also run on any electronic device with sufficient computing power (such as...). Figure 5 In the electronic device shown, the various steps of the image editing method are executed to provide image editing functionality.

[0083] Please see Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, in one embodiment of this application, the electronic device 800 can be a mobile phone, tablet computer, smart wearable device, augmented reality (AR) / virtual reality (VR) device, laptop computer, netbook, etc. This application embodiment does not impose any restrictions on the specific type of electronic device 800.

[0084] like Figure 5 As shown, the electronic device 800 may include, but is not limited to, a communication module 81, a memory 82, a processor 83, an input / output (I / O) interface 84, and a bus 85. The processor 83 is coupled to the communication module 81, the memory 82, and the I / O interface 84 via the bus 85.

[0085] Those skilled in the art will understand that the schematic diagram is merely an example of electronic device 800 and does not constitute a limitation on electronic device 800. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 800 may also include network access devices, etc.

[0086] The communication module 81 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as Universal Serial Bus (USB) and Controller Area Network (CAN). The wireless communication module may provide one or more wireless communication solutions such as Wireless Fidelity (Wi-Fi), Bluetooth (BT), mobile communication networks, Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR) technologies.

[0087] The memory 82 can be used to store computer-readable instructions and / or modules. The processor 83 implements various functions of the electronic device 800 by running or executing the computer-readable instructions and / or modules stored in the memory 82 and by calling the data stored in the memory 82. The memory 82 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 800, etc. The memory 82 may include non-volatile and volatile memory, such as: hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other storage devices.

[0088] The memory 82 can be an external memory and / or an internal memory of the electronic device 800. Furthermore, the memory 82 can be a memory in physical form, such as a memory stick, a TF card (Trans-flash Card), etc.

[0089] Processor 83 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. Processor 83 is the computational core and control center of electronic device 800, connecting various parts of electronic device 800 through various interfaces and lines, and executing the operating system of electronic device 800 and various installed application programs and program code.

[0090] For example, computer-readable instructions can be divided into one or more modules / submodules / units, which are stored in memory 82 and executed by processor 83 to complete this application. One or more modules / submodules / units can be a series of computer-readable instruction segments capable of performing specific functions, describing the execution process of the computer-readable instructions in electronic device 800. For example, computer-readable instructions can be divided into a data receiving module 41, an instruction decomposition module 42, a region segmentation module 43, an image restoration module 44, an image editing module 45, and a fusion optimization module 46.

[0091] If the modules / units integrated in the electronic device 800 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium, and when executed by a processor, they can implement the steps of the various method embodiments described above.

[0092] Computer-readable instructions include computer-readable instruction code, which can be in the form of source code, object code, executable file, or some intermediate form. Computer-readable media can include: any entity or device capable of carrying computer-readable instruction code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), and random access memory (RAM).

[0093] Combination Figures 2 to 4 The memory 82 in the electronic device 800 stores computer-readable instructions, and the processor 83 can execute the computer-readable instructions stored in the memory 82 to implement the method provided in the above embodiments. Specifically, the specific implementation method of the processor 83 for the above-mentioned computer-readable instructions can be found in [reference]. Figures 2 to 4 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0094] I / O interface 84 is used to provide a channel for user input or output. For example, I / O interface 84 can be used to connect various input and output devices, such as mouse, keyboard, touch device, display screen, etc., so that users can enter information or visualize information.

[0095] Bus 85 is used at least to provide a channel for communication between communication modules 81, memory 82, processor 83, and I / O interface 84 in electronic device 800.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0097] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0098] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0099] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0100] Furthermore, it is clear that the word "including" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. An image editing method, characterized in that, The method includes: Receive user input images and text commands; The editing operation, editing object, and editing attributes corresponding to the text instruction are determined through the instruction decomposition model. Based on the editing object, the region image of the corresponding target region in the input image is determined by the segmentation model, and a mask image of the target region is generated; The mask image is inpainted to obtain the repaired image; The region image is edited according to the editing operation and / or editing attributes to obtain an edited image; The repaired image and the edited image are merged to obtain a fused image. The fused image is then optimized to obtain the target image.

2. The image editing method according to claim 1, characterized in that, The instruction decomposition model includes a multi-task learning architecture for simultaneously executing operation classification, object recognition, and attribute detection tasks, wherein: The operation classification task obtains the editing operation by performing verb semantic parsing on the text instruction; The object recognition task obtains the editable object by segmenting the text instruction into object instances; The attribute detection task obtains the editing attributes by performing semantic parsing of attribute words on the text instructions.

3. The image editing method according to claim 1, characterized in that, The editing operations include at least one of the following operations: add, delete, replace, zoom, and move; The editing object includes at least one of the following: a single target object, a global object; The editing attributes include at least one of the following: image color, image texture, and attributes of the object being edited.

4. The image editing method according to claim 1, characterized in that, The step of image inpainting the mask image to obtain the inpainted image includes: The mask region of the mask image is downsampled at multiple levels to obtain multiple downsampled images of the mask region; Based on the resolution from largest to smallest, content recognition, missing content filling, and style transfer are performed on each of the multiple downsampled images to obtain the updated image corresponding to each downsampled image; The multiple downsampled images are fused together with the multiple updated images corresponding to them, and the fused image is upsampled to obtain the repaired image.

5. The image editing method according to claim 1, characterized in that, The step of editing the region image according to the editing operation and / or editing attributes to obtain an edited image includes: If the editing operation is a replacement operation, the editing attributes corresponding to the region image and the edited object are input into the image diffusion model; Based on the editing attributes, a latent feature map corresponding to the region image is generated through the image-generated image diffusion model; The latent feature map is diffused and denoised using a graph-based diffusion model to obtain an edited image that matches the perspective effect of the region image.

6. The image editing method according to claim 1, characterized in that, The process of fusing the repaired image and the edited image to obtain a fused image includes: Perform an edge dilation operation on the edited image, and then perform an edge erosion operation on the edited image after the edge dilation operation to obtain the edge-modified image corresponding to the edited image; The edge-modified image and the repaired image are merged to obtain the fused image.

7. The image editing method according to claim 1, characterized in that, The image optimization of the fused image includes: Edge restoration is performed on the fused image using an edge restoration model; Global optimization is performed on the fused image after edge repair.

8. The image editing method according to claim 7, characterized in that, The global optimization of the fused image after edge repair includes: The edge-repaired fused image is then subjected to tone optimization and style transfer.

9. An image editing device, characterized in that, The image editing device is used to implement the image editing method as described in any one of claims 1 to 8, and the image editing device includes: The data receiving module is used to receive image and text commands input by the user; The instruction decomposition module is used to determine the editing operation, editing object, and editing attributes corresponding to the text instruction through the instruction decomposition model. The region segmentation module is used to determine the region image of the corresponding target region in the input image based on the editing object through a segmentation model, and to generate a mask image of the target region; The image restoration module is used to perform image restoration on the mask image to obtain a restored image; The image editing module is used to perform image editing on the region image according to the editing operation and / or editing attributes to obtain an edited image; The fusion optimization module is used to fuse the repaired image and the edited image to obtain a fused image, and to optimize the fused image to obtain a target image.

10. An electronic device, characterized in that, include: Memory, and A processor that executes computer-readable instructions stored in the memory to implement the image editing method as described in any one of claims 1 to 8.

Images