Image editing method and device, computer device and readable storage medium

Abstract

The application relates to an image editing method and device, computer equipment and a readable storage medium, which can be used in the technical field of image processing. The method comprises the following steps: acquiring an image to be edited; performing feature extraction on color channel data and transparency channel data respectively to obtain initial color features and initial transparency features corresponding to the image to be edited; performing fusion processing on the initial transparency features and the initial color features to obtain fusion features; performing feature generation processing on the fusion features according to an editing instruction for the image to be edited to obtain predicted mixed features; performing fusion processing on the initial transparency features and the predicted transparency features to obtain target transparency features; and generating target image corresponding to the image to be edited according to target transparency channel data corresponding to the target transparency features and target color channel data corresponding to the predicted color features. The method can improve the accuracy of image editing.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image editing method, apparatus, computer device, and readable storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, intelligent image editing technology has been widely applied in fields such as digital entertainment, advertising design, and film and television post-production. Among these applications, the automated and targeted modification of images based on user editing instructions has become an important direction for the development of image processing technology.

[0003] Currently, existing image editing methods typically treat the image as a single, holistic feature. However, when faced with images possessing complex visual elements or multi-dimensional attributes, this holistic approach struggles to capture the detailed information across different dimensions of the image, resulting in lower accuracy in image editing. Summary of the Invention

[0004] Therefore, it is necessary to provide an image editing method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve the accuracy of image editing in response to the above-mentioned technical problems.

[0005] Firstly, this application provides an image editing method. The method includes:

[0006] Obtain the image to be edited; the image to be edited includes color channel data and transparency channel data;

[0007] Feature extraction is performed on the color channel data and the transparency channel data respectively to obtain the initial color features and initial transparency features corresponding to the image to be edited;

[0008] The initial transparency feature and the initial color feature are fused together to obtain a fused feature;

[0009] Based on the editing instructions for the image to be edited, the fusion features are processed to generate predicted blending features; the predicted blending features include predicted transparency features and predicted color features.

[0010] The initial transparency feature and the predicted transparency feature are fused together to obtain the target transparency feature;

[0011] Based on the target transparency channel data corresponding to the target transparency feature and the target color channel data corresponding to the predicted color feature, a target image corresponding to the image to be edited is generated.

[0012] In one embodiment, the step of performing feature generation processing on the fused features according to the editing instructions for the image to be edited, to obtain predicted fused features, includes:

[0013] Extract the geometric features corresponding to the image to be edited, and extract the semantic features corresponding to the editing instructions;

[0014] Based on the geometric features and the semantic features, the fused features are processed to generate the predicted fused features.

[0015] In one embodiment, extracting the geometric features corresponding to the image to be edited includes:

[0016] Edge detection processing is performed on the image to be edited to obtain the edge feature map of the image to be edited;

[0017] The image to be edited is subjected to depth recognition processing to obtain a depth feature map of the image to be edited;

[0018] The geometric features are determined based on the edge feature map and the depth feature map.

[0019] In one embodiment, the process of fusing the initial transparency feature and the initial color feature to obtain a fused feature includes:

[0020] Use the initial transparency feature as the offset feature;

[0021] The offset feature and the initial color feature are added element by element to obtain the fused feature.

[0022] In one embodiment, the process of fusing the initial transparency feature and the predicted transparency feature to obtain the target transparency feature includes:

[0023] The initial transparency feature and the predicted transparency feature are concatenated to obtain a hybrid transparency feature;

[0024] The mixed transparency features are subjected to feature extraction and activation processing to obtain an attention map;

[0025] Based on the attention map, the initial transparency feature and the predicted transparency feature are fused to obtain the target transparency feature.

[0026] In one embodiment, the step of fusing the initial transparency feature and the predicted transparency feature based on the attention map to obtain the target transparency feature includes:

[0027] Based on the attention map, determine the first weight corresponding to the initial transparency feature and the second weight corresponding to the predicted transparency feature;

[0028] Based on the first weight, the initial transparency feature is weighted to obtain the first weighted feature;

[0029] The predicted transparency feature is weighted according to the second weight to obtain the second weighted feature;

[0030] The first weighted feature and the second weighted feature are fused to obtain the target transparency feature.

[0031] Secondly, this application also provides an image editing apparatus. The apparatus includes:

[0032] The image acquisition module is used to acquire the image to be edited; the image to be edited includes color channel data and transparency channel data;

[0033] The feature extraction module is used to extract features from the color channel data and the transparency channel data respectively to obtain the initial color features and initial transparency features corresponding to the image to be edited.

[0034] The feature fusion module is used to fuse the initial transparency feature and the initial color feature to obtain the fused feature;

[0035] The feature generation module is used to perform feature generation processing on the fusion features according to the editing instructions for the image to be edited, to obtain predicted blended features; the predicted blended features include predicted transparency features and predicted color features;

[0036] The feature processing module is used to fuse the initial transparency feature and the predicted transparency feature to obtain the target transparency feature;

[0037] The image generation module is used to generate a target image corresponding to the image to be edited based on the target transparency channel data corresponding to the target transparency feature and the target color channel data corresponding to the predicted color feature.

[0038] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0039] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0040] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the above aspects.

[0041] The aforementioned image editing methods, apparatus, computer devices, computer-readable storage media, and computer program products, by extracting and fusing features from color channel data and transparency channel data respectively, facilitate the simultaneous processing of image color and transparency attributes during the feature generation stage, avoiding inaccurate edges caused by separate processing of color and transparency. Furthermore, by fusing initial transparency features and predicted transparency features, it is beneficial to retain the original transparency features of the image to be edited while incorporating new transparency features from the predicted mixed features. This allows for precise control of transparency when modifying image color or texture, improving the accuracy of image editing. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating an image editing method in one embodiment;

[0044] Figure 2 This is a flowchart illustrating the steps of feature generation processing in one embodiment;

[0045] Figure 3 This is a schematic diagram of the system architecture of an image editing method in one embodiment;

[0046] Figure 4 This is a structural block diagram of an image editing device in one embodiment;

[0047] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0050] In one exemplary embodiment, such as Figure 1 As shown, an image editing method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc.; the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. In this embodiment, the method includes the following steps:

[0051] Step S101: Obtain the image to be edited; the image to be edited includes color channel data and transparency channel data;

[0052] Step S102: Extract features from the color channel data and the transparency channel data respectively to obtain the initial color features and initial transparency features corresponding to the image to be edited;

[0053] Step S103: The initial transparency feature and the initial color feature are fused to obtain the fused feature;

[0054] Step S104: According to the editing instructions for the image to be edited, perform feature generation processing on the fusion features to obtain predicted blended features; the predicted blended features include predicted transparency features and predicted color features;

[0055] Step S105: The initial transparency features and the predicted transparency features are fused to obtain the target transparency features;

[0056] Step S106: Generate the target image corresponding to the image to be edited based on the target transparency channel data corresponding to the target transparency feature and the target color channel data corresponding to the predicted color feature.

[0057] The image to be edited can be an image containing color and transparency information.

[0058] The color channel data can be the data used to indicate the color of pixels in the image to be edited.

[0059] The transparency channel data can be data used in the image to be edited to indicate the degree of pixel transparency.

[0060] The initial color features can be obtained by encoding the color channel data using an image feature extraction model.

[0061] The initial transparency feature can be obtained by encoding the transparency channel data using an image feature extraction model.

[0062] Among them, the fusion feature can be a comprehensive feature that combines the initial color feature and the initial transparency feature.

[0063] Editing instructions can be control information entered by the user to indicate the intention to modify the image.

[0064] Among them, the predicted blended features can be features containing new image content generated after reconstructing the fused features according to the editing instructions.

[0065] Among them, the predicted transparency feature can be a new transparency feature extracted from the predicted mixed features.

[0066] Among them, the predicted color features can be new color features extracted from the predicted mixed features.

[0067] The target transparency feature can be obtained by fusing the initial transparency feature and the predicted transparency feature according to a preset ratio.

[0068] Among them, the target transparency channel data can be transparency image data obtained by reconstructing the target transparency features using a decoding model.

[0069] The target color channel data can be color image data obtained by reconstructing the predicted color features using a decoding model.

[0070] The target image can be an edited image obtained by merging the target color channel data and the target transparency channel data.

[0071] Optionally, the terminal acquires an image to be edited, containing color channel data and alpha channel data; processes the color channel data using a feature extraction model (e.g., a neural network model) to obtain initial color features, and processes the alpha channel data using the same model to obtain initial alpha features; the initial alpha features are superimposed on the initial color features as an offset for fusion processing to obtain fused features; the fused features are input into a feature generation network and iteratively denoised using editing instructions for the image to be edited to obtain predicted mixed features, and predicted alpha features and predicted color features are separated from the predicted mixed features; fusion weights are determined, and the initial alpha features and predicted alpha features are weighted and fused based on these weights to obtain target alpha features; the predicted color features are processed using a decoding model to obtain target color channel data, and the target alpha features are processed using a decoding model to obtain target alpha channel data; the target color channel data and target alpha channel data are concatenated to generate the target image corresponding to the image to be edited.

[0072] In the aforementioned image editing method, by extracting and fusing features from color channel data and transparency channel data separately, it is beneficial to process both the color and transparency attributes of the image simultaneously during the feature generation stage, avoiding the inaccurate edge problems caused by the separate processing of color and transparency. Furthermore, by fusing the initial transparency features and the predicted transparency features, it is beneficial to retain the original transparency features of the image to be edited while combining the new transparency features from the predicted mixed features. This allows for precise control of transparency when modifying image color or texture, thereby improving the accuracy of image editing.

[0073] In one exemplary embodiment, reference is made to Figure 2 Based on the editing instructions for the image to be edited, feature generation processing is performed on the fusion features to obtain predicted fusion features, including:

[0074] Step S201: Extract the geometric features corresponding to the image to be edited, and extract the semantic features corresponding to the editing instructions;

[0075] Step S202: Based on geometric and semantic features, perform feature generation processing on the fused features to obtain the predicted fused features.

[0076] Among them, geometric features can be spatial structural information that reflects the contour gradient of objects in the image to be edited and their relationship with three-dimensional spatial perspective.

[0077] Among them, semantic features can be semantic embedding vectors obtained by encoding editing instructions using a semantic coding model.

[0078] Optionally, after the terminal acquires the image to be edited, it extracts the geometric features corresponding to the image to be edited. These geometric features are used to characterize the contour structure and spatial depth relationship of objects in the image to be edited. It also extracts the semantic features corresponding to the editing instructions. These semantic features are used to characterize the editing intent indicated by the editing instructions. The geometric features and semantic features are used together as conditional information to perform feature generation processing on the fused features. This ensures that the generation process follows the editing intent while being constrained by the original structure of the image to be edited, thereby obtaining the predicted fused features.

[0079] The technical solution provided in this embodiment extracts the geometric features corresponding to the image to be edited and the semantic features corresponding to the editing instructions, and performs feature generation processing on the fused features based on both. This is beneficial because the generation of new features is subject to the dual constraints of structural information and semantic intent, thereby helping to avoid structural collapse or geometric deformation during the generation process and thus improving the accuracy of image editing.

[0080] In an exemplary embodiment, extracting the geometric features corresponding to the image to be edited includes: performing edge detection processing on the image to be edited to obtain an edge feature map of the image to be edited; performing depth recognition processing on the image to be edited to obtain a depth feature map of the image to be edited; and determining the geometric features based on the edge feature map and the depth feature map.

[0081] Edge detection processing can be an operation that uses an edge detection network to identify the contours of objects in an image in order to extract pixel-level edge probabilities.

[0082] Among them, the edge feature map can be a soft edge feature image that preserves image details and light and shadow transitions.

[0083] Depth recognition processing can be an operation that uses a depth estimation model to infer the relative depth of a scene from an image.

[0084] Among them, the depth feature map can be a spatial depth image used to clarify the front and back occlusion relationships of objects in the scene.

[0085] Optionally, the terminal uses an edge detection network to perform edge detection processing on the image to be edited, extracts the values ​​of continuous gradients, and obtains the edge feature map of the image to be edited; it uses a depth estimation model to perform monocular depth recognition processing on the image to be edited, infers the relative depth of the scene, and obtains the depth feature map of the image to be edited; it aligns and standardizes the edge feature map and the depth feature map in the channel dimension, and determines the geometric features based on the processed edge feature map and depth feature map.

[0086] The technical solution provided in this embodiment obtains edge feature maps and depth feature maps by performing edge detection and depth recognition processing on the image to be edited. This is beneficial for extracting contour gradient information and three-dimensional spatial perspective information, thereby facilitating the construction of a multi-view geometric constraint mechanism and avoiding spatial adhesion or perspective errors during the generation process.

[0087] In an exemplary embodiment, the initial transparency feature and the initial color feature are fused to obtain a fused feature, including: using the initial transparency feature as an offset feature; and adding the offset feature and the initial color feature element by element to obtain the fused feature.

[0088] The offset feature can be an offset vector in the latent feature space used to indicate the transparency attribute.

[0089] Element-wise addition can be an operation that adds the values ​​of corresponding elements in two feature tensors.

[0090] Optionally, after obtaining the initial transparency feature and the initial color feature, the terminal uses the initial transparency feature as the offset feature in the latent space; the offset feature and the initial color feature are added element by element according to the values ​​at the corresponding positions, and the transparency information is implicitly injected into the latent space where the initial color feature is located to obtain the fused feature.

[0091] The technical solution provided in this embodiment uses the initial transparency feature as an offset feature and adds it to the initial color feature element by element. This allows transparency information to be implicitly injected into the color feature without changing the original physical input dimension. This helps the subsequent generative network to perceive the transparency distribution and prevents incorrect background color penetration.

[0092] In an exemplary embodiment, the initial transparency feature and the predicted transparency feature are fused to obtain the target transparency feature, including: concatenating the initial transparency feature and the predicted transparency feature to obtain a hybrid transparency feature; performing feature extraction and activation processing on the hybrid transparency feature to obtain an attention map; and fusing the initial transparency feature and the predicted transparency feature according to the attention map to obtain the target transparency feature.

[0093] The splicing process can be a process of stacking and combining the initial transparency features and the predicted transparency features in the channel dimension.

[0094] Among them, the hybrid transparency feature can be the feature data obtained by splicing the initial transparency feature and the predicted transparency feature.

[0095] Feature extraction and activation processing can involve sequentially performing convolutional dimensionality reduction and nonlinear activation on the mixed transparency features. For example, the number of channels in the mixed transparency features can be compressed first through a convolutional layer, and then the compressed values ​​can be mapped to a preset value range through an activation function.

[0096] The attention map can be a single-channel feature map obtained after feature extraction and activation of the hybrid transparency features. The value at each position in the attention map represents the fusion weight ratio between the initial transparency feature and the predicted transparency feature at the corresponding position.

[0097] Optionally, the terminal concatenates the initial transparency features and the predicted transparency features along the channel dimension to obtain a hybrid transparency feature; the hybrid transparency feature is then subjected to convolutional dimensionality reduction and activation processing in sequence, and the intermediate features output by the convolutional dimensionality reduction processing are mapped to a single-channel fractional map, and then the fractional map is activated to constrain the values ​​in the fractional map to a preset numerical range to obtain an attention map; the initial transparency features and the predicted transparency features are then subjected to pixel-wise weighted fusion processing based on the attention map to obtain the target transparency feature.

[0098] The technical solution provided in this embodiment, by splicing the initial transparency features and the predicted transparency features, and performing feature extraction and activation processing to obtain an attention map, is beneficial for adaptively measuring the ratio of retaining the original transparency structure to adopting the newly generated transparency structure at different positions of the image to be edited. This facilitates pixel-level dynamic adjustment of the transparency structure and improves the accuracy of image editing.

[0099] In an exemplary embodiment, the initial transparency feature and the predicted transparency feature are fused according to the attention map to obtain the target transparency feature, including: determining a first weight corresponding to the initial transparency feature and a second weight corresponding to the predicted transparency feature according to the attention map; weighting the initial transparency feature according to the first weight to obtain a first weighted feature; weighting the predicted transparency feature according to the second weight to obtain a second weighted feature; and fusing the first weighted feature and the second weighted feature to obtain the target transparency feature.

[0100] The first weight can be a value in the attention map, used to represent the proportion of the initial transparency feature that is retained.

[0101] The second weight can be a complementary value derived from the attention map, used to characterize the proportion of the adoption prediction transparency feature.

[0102] The first weighted feature can be obtained by multiplying the initial transparency feature element by element based on the first weight.

[0103] The second weighted feature can be obtained by multiplying the predicted transparency feature element by element based on the second weight.

[0104] Optionally, the terminal determines a first weight corresponding to the initial transparency feature at each position and a second weight corresponding to the predicted transparency feature based on the value at each position in the attention map. The first weight and the second weight are complementary. The initial transparency feature is weighted pixel by pixel according to the first weight to obtain a first weighted feature. The predicted transparency feature is weighted pixel by pixel according to the second weight to obtain a second weighted feature. The first weighted feature and the second weighted feature are added element by element to obtain the target transparency feature.

[0105] The technical solution provided in this embodiment determines the first and second weights corresponding to the initial transparency features and the predicted transparency features based on the attention map, and then performs weighted processing before fusing to obtain the target transparency features. This facilitates pixel-level control between the original transparency structure and the newly generated transparency structure, thereby improving the accuracy of image editing by adopting the new structure in the editing area while preserving the original details in the non-editing area.

[0106] The following embodiment illustrates the image editing method provided in this application. This embodiment details an RGBA (red, green, blue, and transparency) image editing method and system based on dual-stream latent feature fusion and gating mechanisms.

[0107] Image editing task introduction:

[0108] RGB image editing is a core task at the intersection of computer vision and computer graphics, encompassing everything from traditional signal processing to modern generative artificial intelligence techniques.

[0109] RGB image editing, defined by its tasks and technological layers, refers to the process of modifying, enhancing, or reconstructing digital images composed of red, green, and blue color channels using algorithms. Its core objective is to strictly maintain the naturalness and realism of the image in unmodified areas while altering its visual content or attributes according to the user's specific intent (such as through text commands, mask guidance, reference images, or interactive operations). From a technical perspective, this task can be divided into two main layers: first, "low-level visual editing," which focuses on signal processing and includes denoising, deblurring, super-resolution reconstruction, and color correction, aiming to improve the physical quality of the image; and second, "high-level semantic editing," which focuses on content understanding and reconstruction, including inpainting, style transfer, object attribute modification (such as changing color or shape), and text-based content generation for compositing. These two types of technologies together form the foundation of modern digital image processing.

[0110] Challenges of Existing Technologies: Despite significant progress in image editing from deep learning techniques such as Generative Adversarial Networks (GANs) and Diffusion Models, numerous challenges remain when handling high-quality, high-fidelity editing tasks. First, there's the challenge of semantic consistency and background fusion. Especially when removing objects or drastically modifying the foreground, algorithms often struggle to perfectly reconstruct occluded backgrounds, leading to artifacts at editing boundaries or unnatural lighting and perspective – a classic ill-posed problem. Second, there's the difficulty of attribute decoupling. Since RGB pixel values ​​are highly coupled with lighting, texture, and geometry, traditional editing methods often struggle to change an object's color without destroying its original shadow and three-dimensionality. Furthermore, existing generative models frequently suffer from high-frequency detail loss during image reconstruction, tending to generate smooth, low-frequency textures, resulting in edited areas lacking the graininess and subtle details expected in realistic photographs.

[0111] RGB image editing technology has penetrated deeply into several key industries, demonstrating immense commercial value. In the field of digital creativity and design, it has significantly lowered the barrier to entry for photographers and designers through intelligent image retouching (such as one-click sky replacement and portrait beautification) and auxiliary creative tools, reshaping the content production workflow. In the e-commerce and advertising industries, this technology is widely used to automate the generation of product display images, virtual try-on, and background replacement, significantly reducing the cost of commercial shooting and post-production. Furthermore, in augmented reality (AR) and virtual production, real-time removal of obscuring objects in real-world scenes and film special effects compositing also heavily rely on high-precision image editing algorithms. Even in the field of medical image processing, RGB editing technology is used to enhance the clarity of endoscopic images and remove visual artifacts in lesion analysis, assisting doctors in making more accurate diagnoses.

[0112] The academic and social significance of this research: In-depth research on RGB image editing technology not only has theoretical significance but also profound socio-economic benefits. First, it promotes the democratization of AIGC technology. Powerful algorithmic tools allow non-professional users to create professional-level works through simple language descriptions or smearing operations, greatly unleashing public creativity. Second, it serves as a touchstone for testing whether artificial intelligence possesses high-order visual cognitive abilities. Perfect editing requires computers not only to process pixels but also to deeply understand the physical laws and semantic logic of a scene. Finally, in terms of data security and privacy protection, the de-identification and replacement scheme for sensitive information (such as faces and license plates) based on editing technology protects user privacy while preserving data usability, providing a new technological path for privacy protection in the era of big data.

[0113] The special characteristics of RGBA image editing tasks:

[0114] Compared to traditional image editing that only contains color channel data (hereinafter referred to as RGB images), RGBA image editing tasks (images that simultaneously contain color channel data and alpha channel data, where R, G, and B represent the red, green, and blue color channels respectively, and A stands for Alpha, referring to the alpha channel) have stronger physical coupling characteristics. In RGBA data, the alpha channel is not simply a fourth color dimension, but rather simultaneously represents the degree of light transmission (i.e., opacity) and the geometric existence information of objects. This gives RGBA image editing tasks two attributes: on the one hand, it belongs to image generation tasks, which need to be responsible for the texture and color reconstruction of color channels; on the other hand, it belongs to high-precision image matting tasks, which need to be responsible for the edge and semi-transparency prediction of the alpha channel. Its special feature is that the color intensity values ​​of the color channel data and the alpha values ​​of the alpha channel data must follow a strict physical product relationship, namely the pre-multiplication alpha relationship (a way of expressing color channel data by weighting it according to the alpha value during storage or processing). If modifications to color content (such as changing the color of an object) are not synchronized with the adjustment of the transparency channel to match the new material properties (such as changing from an opaque material to a translucent material), or vice versa, visual artifacts that violate physical laws will occur. Therefore, RGBA image editing requires processing algorithms to have the ability to understand the compositional relationship between foreground, background, and transparency, rather than simply mapping pixel values.

[0115] Current RGBA image editing technology faces the following technical challenges:

[0116] Inter-channel spatial alignment and color bleeding issues. One of the technical challenges in RGBA image editing is ensuring strict pixel-level alignment between color channel information and alpha channel information. When editing semi-transparent objects (such as smoke, water, and hair), the alpha channel often exhibits drastic gradient characteristics. If the generated color texture exceeds the range covered by the alpha channel, the background color will seep into the foreground area, resulting in color bleeding; conversely, if the alpha channel's coverage exceeds the effective pixel range of the color channel, unnatural black or white edges will appear at the object's edges. Existing image editing solutions typically treat the alpha channel as a normal channel, failing to adequately consider this edge alignment requirement, making it difficult to cleanly composite the edited image with a new background.

[0117] Background independence and compositing robustness are key challenges. Unlike color image editing, which only needs to ensure the current image appears visually natural, RGBA image editing results are typically used as source material and require background independence. This means that the edited semi-transparent areas (such as shadows or glass reflections) should contain accurate physical transparency information to ensure proper blending when overlaid on backgrounds of different colors or complex textures. Existing generation methods tend to embed background color information into the semi-transparent pixels of foreground objects (e.g., a transparent container generated on a light-colored background will have difficult-to-remove light-colored reflections). Decoupling ambient light information from the object's inherent color information to generate relatively pure transparency data is one of the technical challenges in this field.

[0118] The semantic decoupling of material properties and transparent structures is a challenge. In editing scenarios involving high-level semantics, it is often necessary to independently control the shape and material properties of objects. For example, editing a circular transparent object into a square transparent object (changing the shape while maintaining the transparent material), or editing a transparent object into an opaque object (maintaining the shape while changing the material and transparency). Since color and texture features and transparency features often exhibit strong correlation and coupling in the depth feature space, existing solutions struggle to achieve fine decoupling of these attributes: changing the object's shape easily destroys its original transparent texture, while changing the material properties easily loses the original fine edge structure. Achieving controllable and independent editing of color and transparency information requires designing corresponding decoupling architectures and constraint mechanisms.

[0119] Traditional RGBA image editing schemes and their limitations:

[0120] Before the application of deep learning-based generative models, editing RGBA images primarily employed a channel separation approach, which involves separating color channel data and alpha channel data into two relatively independent parts for separate processing. This approach typically includes the following steps: First, extracting the alpha mask of the foreground object using image matting techniques or manual methods; then, treating the alpha channel as a regular grayscale image and applying pixel-based texture synthesis algorithms or image inpainting algorithms in parallel with the color channels for pixel filling or modification; finally, recombining the processed channels through a simple overlay operation. While this method is usable in relatively simple image editing scenarios, it is essentially a shallow signal processing operation, lacking the ability to semantically understand the image content.

[0121] This type of traditional approach has the following limitations:

[0122] The fragmented modeling of color and alpha channels leads to physical inconsistencies and visual artifacts. Traditional methods typically model and process color and alpha channels separately, neglecting the inherent physical coupling of RGBA data. Since the color and alpha information of a semi-transparent object are closely related, the forced separation of these two aspects in traditional algorithms often results in visual artifacts during image compositing. For example, when editing object edges, the algorithm's failure to coordinate color and alpha updates can cause background colors to inadvertently penetrate the foreground area, or create unnatural black or white edges along the object's contours. This approach essentially degenerates into a post-processing operation similar to image matting, rather than true image content reconstruction, making it difficult to guarantee pixel-level fidelity in the composite.

[0123] The system suffers from redundant structure and a lack of high-dimensional generation capabilities. To compensate for the accuracy loss caused by channel separation processing, traditional or early solutions often require the introduction of additional encoder and decoder branches to specifically handle transparency information. This not only increases the number of model parameters and inference costs but also makes the overall processing pipeline quite redundant. Furthermore, due to the lack of a unified representation capability for high-dimensional latent feature spaces, such solutions struggle to achieve truly generative editing. When faced with complex editing instructions that require changing the shape or material of an object (e.g., editing a transparent circular object into an opaque square object), traditional solutions typically rely on pixel duplication or interpolation, making it difficult to generate new transparent structures that conform to physical laws. Consequently, the final editing effect fails to meet the high-requirement requirements of application scenarios in terms of both semantic logic and texture detail.

[0124] To address the aforementioned technical issues, this embodiment proposes an image editing method and system based on dual-stream latent feature fusion and gating mechanism, aiming to solve the technical problem that existing generative image editing schemes struggle to balance physical consistency and structural controllability when handling transparent image editing tasks.

[0125] In terms of architecture design, this embodiment employs a dual-stream coding strategy, extracting color and transparency features through a pre-trained color variational autoencoder and a lightweight transparency encoder, respectively. Implicit fusion of the two feature streams is achieved through a latent offset mechanism (a fusion method that injects one latent feature stream as an offset into another), enabling the used diffusion network (UNet) to perform denoising and generation processing while being aware of transparency information. In the decoding stage, this embodiment introduces a feature gate module. This module dynamically adjusts the fusion weights between the original and generated transparency features based on the semantics of the editing instructions (e.g., whether only texture modification or shape reconstruction is required). Finally, the target image is reconstructed through two independent decoding processes: a color decoder and a transparency decoder.

[0126] This embodiment has the following advantages over the prior art:

[0127] This approach overcomes the limitations of traditional channel separation processing methods, effectively addressing issues of color bleeding and edge artifacts. Traditional RGBA image editing schemes typically treat color and transparency channels as two independent signals, processing them separately (e.g., performing image inpainting or filtering separately) before simply superimposing them. This ignores the physical product relationship between color and opacity information in semi-transparent objects, easily leading to color bleeding or unnatural black-and-white edges in the edited image. This embodiment introduces a latent offset mechanism to implicitly couple color and transparency features within the feature space. This allows the diffusion network to perceive the corresponding transparency attribute at each location when generating color information, meaning the generated result is not a simple color value but a color expression weighted and correlated with transparency information. This helps reduce edge artifacts, allowing the generated semi-transparent objects (e.g., smoke, glass) to blend better into different new backgrounds.

[0128] This invention represents a technological leap from simple pixel patching to semantic structure reconstruction, enabling the processing of complex editing instructions. Traditional editing methods primarily rely on sample block-based texture synthesis or shallow convolutional networks, essentially involving pixel-level manipulation and splicing. They struggle to understand the high-level semantic information of images, thus only capable of handling relatively simple tasks such as watermark removal or background patching, and unable to respond to complex editing instructions involving geometric deformation and material reasoning. This embodiment deeply integrates a latent diffusion model with a multimodal guidance approach including a language-image joint coding model and a structural control network. It not only understands the semantic information of editing instructions but also generates reasonable details not present in the original image based on learned physical prior information (such as light refraction and transmission laws), enabling editing operations to possess a certain degree of spatial perspective understanding and light and shadow reconstruction capabilities.

[0129] This implementation achieves decoupled control between texture and structural information, which helps address the issue of insufficient control granularity during editing. In some existing depth generation schemes, color and transparency information are tightly integrated, making it easy for changes in one area to affect others. For example, users might accidentally disrupt the object's original transparent outline when only changing its color, or lose the original material texture when modifying its shape. This embodiment designs a feature gating module that dynamically adjusts the mixing weight between the original and generated transparency features based on the editing intent. This allows users to perform texture recoloring while keeping the transparency channel structure essentially unchanged, or to perform significant shape reshaping edits while opening the transparency channel, achieving on-demand decoupled editing control.

[0130] I. Overall Architecture Description:

[0131] In one exemplary embodiment, this embodiment constructs an end-to-end image editing system, whose core feature generation network is an improvement and extension of a pre-trained latent diffusion image inpainting network architecture. This original network architecture typically only supports processing three-channel color data. To overcome this limitation, this embodiment designs a dual-stream coding-latent offset fusion-gated dual decoding processing pipeline: retaining the original network architecture's pre-trained generation capability for color channel data, while adding a lightweight transparency processing branch. Through feature coupling and decoupling within the latent space, it achieves refined editing of image texture and structural information.

[0132] Figure 3 This is a schematic diagram of the system architecture, including: an input RGBA image (an image containing both color channel data and alpha channel data); an RGB Encoder (used to extract features from the color channel data); an alpha Encoder (used to extract features from the alpha channel data, where alpha refers to the alpha channel); a Mask (used to indicate the areas to be edited in the image); Concat (a process of stacking and combining features along the channel dimension); Latent Offset ADD (an operation of adding one latent feature as an offset to another latent feature element-wise); a Soft-edge / depth extractor (used for edge detection and depth recognition of the image); a ControlNet (a control network used to transform geometric features into multi-scale control features and inject them into a diffusion network); CLIP (a language-image joint pre-trained model used to encode text or reference images into semantic vectors); a UNet diffusion network (a neural network used for iterative denoising of latent features); and a Feature Gate (used for dynamic weighted fusion of the original alpha features and the predicted alpha features); alpha Decoder (used to decode the fused transparency latent features into transparency channel data); RGB Decoder (used to decode the predicted color features into color channel data); Channel merge (used to combine the decoded color channel data and transparency channel data into the final image); Edited RGBA image (the target image obtained after editing).

[0133] II. Two-stream feature coding and input adaptation:

[0134] At the input end, the terminal employs a parallel dual-encoder design: On one hand, the color components in the input RGBA image are fed into a color encoder with parameters kept frozen. This color encoder can reuse a pre-trained variational autoencoder (a neural network structure that maps high-dimensional data to a low-dimensional latent space through an encoding-sampling-decoding method) to compress high-dimensional pixel space data into standard latent features (Latents, referring to the low-dimensional feature expressions obtained after encoding by the variational autoencoder), thereby ensuring compatibility with the backbone network. On the other hand, the transparency component is fed into a specially designed transparency encoder, which is a lightweight convolutional network used to extract gradient and geometric information of transparency and output transparency latent features aligned with the dimensions of the color latent features. In addition, the terminal performs downsampling on the editing mask. The downsampled mask is used for subsequent feature stitching and is also combined with the original image to be edited to extract soft-edge (pixel-level edge probability features) or depth (features representing the relative distances of scenes) feature maps, in preparation for geometric structure control.

[0135] III. Potential Offset Injection and Input Construction:

[0136] To enable the denoising backbone (UNet) of the adopted diffusion network to perceive transparency information, this embodiment modifies the feature construction of the input layer: First, the color latent features and the downsampled mask are concatenated (i.e., stacked combination operation in the channel dimension) to meet the input format requirements of the adopted image inpainting diffusion network for noise latent variables, mask, and latent variables of the masked image; then, a core latent offset addition process is introduced, where the transparency latent features from the transparency encoder are used as an offset and injected into the color latent features through element-wise addition. Through this process, the physical properties of transparency are implicitly encoded into the color features, enabling the subsequent denoising process to generate a matching background blending color based on the transparency distribution, thereby preventing color penetration beyond the boundaries.

[0137] IV. Feature generation processing guided by multimodal approaches:

[0138] The core inference engine is implemented using the UNet diffusion network, which is the denoising backbone network in the aforementioned image inpainting-type diffusion networks. This network receives the fused latent features as its main input, and is also guided by the following two conditions:

[0139] Structure-guided: The extracted geometric feature maps (including soft edge feature maps and depth feature maps) are transformed into multi-scale residual features through ControlNet and injected into various layers of the UNet diffusion network to lock the object contour and spatial perspective relationship.

[0140] Semantic guidance: The input image or text editing instructions are transformed into semantic vectors by the CLIP encoder and injected into the UNet diffusion network through a cross-attention mechanism (an attention computation method that associates features generated during the process with external conditional vectors) to guide the generation of texture details that conform to the description. The UNet diffusion network undergoes multi-step iterative denoising processing, and the output contains hybrid latent features that simultaneously contain new color texture information and implicit transparency structure information.

[0141] V. Feature Gating Mechanism and Dual-Path Independent Decoding:

[0142] In the decoding output stage, this embodiment uses feature gating to decouple the editing process. The feature gating module simultaneously receives two signals: one is the original transparency feature from the transparency encoder, representing the original structural information before editing; the other is the transparency prediction feature extracted from the hybrid features output by the UNet diffusion network, representing the new structural information generated according to the semantics of the editing instruction. Based on the semantics of the editing instruction (e.g., whether to change color or shape), the feature gating module dynamically adjusts the fusion weights of the two features. The fused transparency feature is fed into the alphaDecoder to reconstruct the transparency channel; the main feature output by the UNet diffusion network is fed into the RGB Decoder (a color decoder that can reuse a pre-trained decoding network matching the aforementioned color encoder) to reconstruct the color channel; the two are combined in the channel merging process to output the final edited RGBA image.

[0143] VI. Overall Architecture and Condition Injection Mechanism Description:

[0144] The core processing architecture of this embodiment is based on a pre-trained latent diffusion image inpainting network architecture, combined with a control network based on zero convolution initialization for guided generation.

[0145] The core foundation of this latent diffusion image inpainting network architecture is the Latent Diffusion Model (LDM), a generative model that performs noise addition and denoising in a low-dimensional latent feature space. Unlike traditional inpainting models that directly compute in pixel space, this architecture utilizes a pre-trained variational autoencoder to compress the image into a low-dimensional latent space. The diffusion process (noise addition and denoising) is performed entirely on this latent feature space, thereby reducing computational complexity and enabling the model to generate high-resolution, high-fidelity inpainting results with limited computational resources.

[0146] In terms of channel input strategy, the input layer of the denoising network in a standard text generation image model typically only receives 4 channels of noise latent features. To enable the model to perceive the missing regions and reference background information, the image inpainting diffusion network used in this embodiment modifies the input layer, adopting a 9-channel concatenation input strategy: noise latent features (4 channels), representing the generation target to be denoised at the current time step; masked image latent features (4 channels), which are latent features obtained by zeroing the masked parts of the original image and encoding them through a variational autoencoder, used to provide the model with contextual information of known regions (such as background texture, lighting direction, etc.); and mask latent features (1 channel), which are data obtained by downsampling the input binary mask image to the latent space size, used to clearly indicate which regions need to remain unchanged and which regions need to be regenerated. The above three parts of data are concatenated in the channel dimension to form a tensor with 4+4+1 channels, which is input into the first convolutional layer of the denoising network.

[0147] In terms of semantic guidance, the user's text prompt (i.e., text-based editing instructions) is transformed into semantic embedding vectors by the CLIP text encoder. These vectors are injected into various layers of the denoising network through a cross-attention mechanism. During the generation process, the network is guided to generate content that conforms to the semantics of the text within the masked area. At the same time, the aforementioned contextual features are combined to ensure that the generated content and the unmasked background are visually integrated.

[0148] During the iterative denoising process, the network starts with random noise and gradually predicts and removes noise. Unlike the conventional generation process, this architecture performs an alignment operation after each denoising step, replacing the predicted results outside the masked area with the known latent features of the original image at the corresponding noisy time step. This mask-guided diffusion ensures that the background area in the final output image remains consistent with the original image, while the edited area achieves a natural transition and reconstruction from texture to semantics.

[0149] This embodiment adds a transparency processing branch to the above-mentioned color image editing framework. It extracts features through a transparency encoder, fuses features through a latent offset fusion mechanism, and decodes features through a transparency decoder, thereby obtaining a transparency processing effect that is consistent with the semantics and details of the color channels.

[0150] VII. Specific implementation of the transparency encoder and transparency decoder:

[0151] The transparency encoder and decoder follow the standard variational autoencoder's encoder-sampling-decoder processing paradigm, but with targeted simplifications in network depth and width. Since the semantic complexity of the transparency channel data is lower than that of the three-channel color data, this embodiment omits the deep self-attention module used to capture complex textures in the network structure of the transparency encoder and decoder, and reduces the number of stacked residual blocks. The overall design is a fully convolutional network, maintaining the same downsampling factor as the backbone color encoder (e.g., an 8x downsampling factor) to ensure that the generated transparency latent features are perfectly aligned spatially with the color latent features (e.g., a 64×64 spatial resolution), thereby supporting subsequent element-wise latent offset operations.

[0152] In one implementation, the transparency encoder includes: an input layer for receiving a single-channel transparency image (e.g., represented as a data dimension of height x width x 1), which expands the number of channels to a base width (e.g., 32 or 64) through a first-layer convolution; a downsampling backbone containing multiple consecutive downsampling stages, each stage being dimensionality-reduced by a stride-2 convolutional layer (Stride-2 Conv, i.e., convolution with a stride of 2), followed by a lightweight residual block, where the standard convolution can be replaced with a depthwise separable convolution (a convolution method that splits the standard convolution into channel-wise convolution and pointwise convolution to reduce computational cost); a latent mapping layer, where, after multiple downsampling processes, the feature map resolution is reduced to a preset ratio of the original image (e.g., one-eighth of the original image), and at this point, a convolutional layer maps the number of channels to multiple values ​​to construct a diagonal Gaussian distribution; and an output layer, which uses reparameterization techniques. Trick, a technique that transforms a random sampling process into a differentiable computation process, samples the latent transparency features (e.g., 4 channels) whose dimensions match the dimensions of the latent color features (e.g., 64×64×4).

[0153] In one implementation, the transparency decoder receives latent features (e.g., 4 channels) output from the feature gating module. The decoding process includes multiple upsampling stages. To avoid checkerboard artifacts common in deconvolution (a processing method that achieves upsampling by transposing convolution kernels), this embodiment uses a combination of nearest neighbor upsampling (an upsampling method that fills the positions of new pixels with the values ​​of neighboring pixels to achieve resolution improvement) and convolution processing to restore resolution. After each upsampling stage, a lightweight residual block is set to restore edge details. The number of channels in the decoder decreases layer by layer as the resolution increases (e.g., it can be reduced from 128 to 64, and then to 32). The last layer compresses the number of channels to 1 through a 1×1 convolution and uses a sigmoid activation function (an activation function that can map any real number to a value range between 0 and 1) to constrain the output value within a preset value range of 0 to 1, representing physical opacity, ensuring that the generated transparency mask is numerically valid.

[0154] During the training phase, the training objectives of the transparency encoder and transparency decoder may include: employing a reconstruction loss that measures the difference between the input transparency data and the reconstructed transparency data, such as at least one of L1 loss (a loss function based on absolute error) or binary cross-entropy loss (BCE Loss, a loss function that measures the difference between two probability distributions), focusing on gradient accuracy in edge regions; and calculating a KL divergence (Kullback-Leibler Divergence, a measure of the difference between two probability distributions) regularization term between the transparency latent distribution and the standard normal distribution N(0,1), i.e., a standard normal distribution with a mean of 0 and a variance of 1, to prevent excessive numerical fluctuations in the latent space and ensure that the transparency latent features can be smoothly superimposed on the color features as offsets without compromising the numerical stability of the backbone diffusion network.

[0155] VIII. Detailed Explanation of Dual-Stream Feature Fusion Method:

[0156] The Latent Offset Fusion method proposed in this embodiment is an implicit fusion approach performed in the feature space. Its core idea is not to simply treat the alpha channel as an additional channel for physical concatenation, but rather to view the latent alpha feature as an offset vector in the vector space representing the latent color feature. By injecting the latent alpha feature into the latent color feature in an element-wise additive manner, the color feature points are shifted on the latent manifold (a low-dimensional curved subspace structure within the high-dimensional space of the latent features) to the region where the feature points representing alpha information also reside. This design allows the pre-trained diffusion network, with its parameters frozen, to perceive the object's alpha attribute (e.g., light transmittance) through the offset in feature values, without modifying the original input dimension of the diffusion network.

[0157] Dimensionality Alignment and Preprocessing: To achieve pixel-level accurate fusion, it is essential to ensure strict alignment of the RGB and Alpha streams in their latent spatial dimensions. Spatial Dimension Alignment: Assume the input image resolution is H×W. The RGB encoder has a downsampling factor f=8, and the output RGB latent features Z... rgb The dimensions are H / 8×W / 8×4. The lightweight alpha encoder designed in this embodiment strictly adheres to the same downsampling structure, ensuring that the output alpha latent features Z... alpha It has a completely consistent spatial resolution of H / 8 × W / 8. Channel dimension alignment: Although the original input to the alpha channel is a single channel, the alpha encoder is designed to output a 4-channel feature map. This makes Z... alpha With Z rgb A perfect match in the number of channels creates the conditions for subsequent direct addition operations. The specific operation flow of offset injection: In the data flow of the Inpainting task, the fusion operation occurs after feature concatenation (Concat) and during the construction of the UNet input tensor. The specific operation is as follows: Let Z... in To prepare the composite tensor for input to UNet (typically 9 channels in SD 1.5 Inpainting, containing latent features to be denoised, mask features, and background latent features), this embodiment will use Z... alpha Targeted injection into Z in The middle part represents the four channel components of the "image to be generated". Mathematically, it is expressed as:

[0158] =Z target + Z alpha .

[0159] Among them, Z target It is the original latent feature of RGB. It is a composite feature resulting from fusion. It is a learnable or preset adjustment coefficient (usually initially set to 1.0). This step is logically equivalent to telling the model: the pixel at this location is not only red (RGB feature), but also semi-transparent (Alpha offset).

[0160] IX. Detailed description of the condition control generation module:

[0161] To address the structural collapse and geometric deformation issues that easily occur in diffusion networks during image editing, this embodiment designs an independent conditional control generation module. This module extracts explicit geometric prior information from the input image and constructs structural constraint information to constrain the diffusion network's generation process. Instead of using only single edge or depth information, this embodiment introduces two complementary visual features in parallel: one is soft edge features obtained from Holistically-Nested Edge Detection (HED, a deep neural network model capable of outputting pixel-level edge probability maps); the other is depth features obtained from Monocular Depth Estimation (a deep neural network model capable of inferring relative depth information of a scene from a single image). This multi-view geometric constraint mechanism enables the model to maintain the contour gradient and spatial perspective relationship of the original image when reconstructing textures or changing object materials, thereby generating RGBA images with good structural fidelity.

[0162] In the edge feature extraction branch, this embodiment employs a globally nested edge detection model. Unlike traditional edge detection operators that only output binary edge results, this model possesses multi-scale feature fusion capabilities, enabling it to output pixel-level edge probability maps, i.e., soft edge features. For semi-transparent objects (such as smoke, water flow, and glass), their boundaries typically exhibit gradient characteristics rather than truncation characteristics. The continuous gradient values ​​extracted by this model can accurately describe the attenuation trend of transparency. This branch maps the input image into a soft edge feature map that preserves contour details and lighting transition information, providing constraint information about the object's contour for subsequent generation processes.

[0163] In the depth feature extraction branch, this embodiment integrates a pre-trained monocular depth estimation model, which can infer the relative depth map of a scene from a single color image. This feature map maps image pixels to depth values ​​from near to far, representing the occlusion relationship of objects in the scene. Depth features play a spatial constraint role in the editing task, enabling the diffusion network to understand which pixels in the image belong to foreground objects and which belong to the background environment, thereby avoiding artifacts such as spatial adhesion between foreground and background or perspective errors during the generation process.

[0164] The extracted soft edge feature maps and depth feature maps are aligned and normalized along the channel dimension before being fed into ControlNet. This control network transforms the aforementioned geometric conditions into multi-scale latent control features through a series of residual blocks initialized with zero convolutions (a type of convolution with weights initialized to zero). These features are then injected into the encoder and decoder layers of the backbone UNet diffusion network. Through this mechanism, the contour constraints provided by the soft edge features and the spatial constraints provided by the depth features work together to guide the UNet diffusion network to adhere to the geometric structure of the original image at each step of the denoising generation process, enabling texture and material editing while maintaining the basic physical structure.

[0165] Detailed network structure description of the feature gating module:

[0166] The feature gating module is constructed as a differentiable fusion layer in the latent space, receiving two feature inputs with the same spatial resolution (e.g., 64×64) in parallel: the first input is the raw transparency latent feature (denoted as Z) from the output of the transparency encoder. origin The first feature fully encodes the geometric contours and transparency distribution of the object in the original image, representing the structural information before editing. The second input is the edited feature from the output of the UNet diffusion network. Since this output is usually a mixed tensor containing both color and transparency information, this embodiment uses a feature projection layer (e.g., a 1×1 convolution or linear segregating structure) to extract the prediction feature (denoted as Z) specifically corresponding to the transparency change from this mixed tensor. pred This feature represents new structural information generated by the model based on text or image conditions.

[0167] Attention-based feature gating module network architecture design:

[0168] In this embodiment, the feature gating module adopts a Convolutional Spatial Attention Network architecture. The module receives two latent feature tensors with the same spatial resolution (H×W) and number of channels (C) as input: the original structural features F from the Alpha Encoder. origin , representing the inherent transparency geometry of the input image; F, the predicted structural features F from the projection layer output of the diffusion model (UNet). pred This represents the transparency change information generated based on the editing intent. The core task of the module is to compute a single-channel Soft Score Map M and use this map to perform pixel-by-pixel weighted fusion of the two inputs.

[0169] 1. Attention Map Generation Subnetwork: To generate high-precision attention maps, a lightweight convolutional subnetwork is built within the module. The specific process is as follows:

[0170] Feature stacking: First, F origin With F pred The features are concatenated along the channel dimension to form a hybrid feature tensor with dimensions H×W×2C; this step aims to enable the network to simultaneously perceive the differences and commonalities between the original structure and the newly generated structure.

[0171] Feature extraction and compression: The concatenated tensors are passed sequentially through a dimensionality reduction convolutional layer and a feature fusion layer; the dimensionality reduction convolutional layer typically uses a 3×3 convolutional kernel with a ReLU (rectified linear function) activation function to compress the number of channels from 2C to an intermediate dimension (such as C / 2) to extract local spatial context associations; the feature fusion layer further maps the features to a single-channel original score map through a 1×1 convolution, and its value represents the network's confidence in "preserving the original structure".

[0172] Softening activation: The original score map is processed by the Sigmoid activation function, which strictly maps all values ​​to the interval [0, 1], outputting the final soft attention map M∈R. H×W×1 The closer the value in the image is to 1, the more likely the original image structure should be retained at that pixel location; the closer the value is to 0, the more likely the generated new structure should be adopted at that location.

[0173] 2. After obtaining the soft attention map M through spatial adaptive weighted fusion, the module executes a broadcast mechanism, copying M C times along the channel dimension to match the dimension of the input features, and then performs an element-wise weighted summation operation. The fusion formula is defined as:

[0174] F fused =M F origin +(1-M) Fpred .

[0175] in This represents the Hadamard Product (element-wise multiplication), where 1 represents an all-1 matrix.

[0176] This operation achieves pixel-level decoupled control: in the image background or unedited areas, the attention mechanism automatically predicts high scores (M≈1), thus preserving the original transparency details without loss; while in the editing target area (such as the edge where the object shape changes), the attention mechanism predicts low scores (M≈0), allowing the features generated by the diffusion model to dominate, thereby achieving a natural transition between texture and structure.

[0177] The technical solution provided in this embodiment has the following beneficial effects:

[0178] This approach achieves more precise decoupling control between texture and structural information, improving the fineness of editing control. Existing image editing solutions often treat color and transparency information as a tightly integrated whole, which can easily lead to accidental destruction of the original transparent contours when only the object's material needs to be modified, or loss of the original material texture when modifying the shape. This embodiment, through a feature gating mechanism, can dynamically adjust the mixing weight between the original transparency features and the generated transparency features according to the semantics of the editing instructions. This allows the system to perform texture replacement while keeping the transparency channel structure essentially unchanged, and also to perform significant geometric reconstruction when the transparency channel is open, achieving a finer control granularity and improving the accuracy of editing operations.

[0179] A latent feature representation with relatively consistent physical relationships is constructed, which helps reduce color penetration and edge artifacts. Targeting the unique semi-transparent mixing physical properties of RGBA images, this embodiment employs a latent offset technique to implicitly inject transparency information into the deep feature space where color features reside. This allows the diffusion network to perceive the current transparency distribution information at each step of the denoising generation process. Compared to traditional channel separation processing methods, this embodiment helps reduce the penetration of background color into the semi-transparent foreground area, resulting in a cleaner and sharper transition effect in semi-transparent edge areas (such as hair strands, smoke, and glass reflections). It also possesses physical properties that better conform to the pre-multiplied transparency relationship, facilitating compositing with backgrounds of varying complexity.

[0180] It possesses good ecosystem compatibility and training efficiency, which helps reduce model training costs. The dual-stream coding architecture of this embodiment can directly reuse the mature standard color variational autoencoder. Through the design of backbone freezing and side adaptation, the training cost is reduced on the one hand, requiring only fine-tuning of the lightweight transparency encoder and feature gating module; on the other hand, ecosystem compatibility is improved, and the system can be directly adapted to various stylized models derived from the same pre-trained network architecture (such as models with different art styles or rendering styles). This means that this embodiment can be easily transferred and applied to transparent image editing tasks in different vertical fields without retraining the backbone network.

[0181] A Leap in Productivity from "Manual Refinement" to "Intelligent Automation": In commercial applications, this implementation significantly reduces the marginal cost of producing professional transparent materials. In e-commerce design, game asset creation, and film post-production, processing the cutout and repair of semi-transparent objects typically requires experienced designers to spend considerable time manually creating masks and repairing channels. This implementation transforms this process into a "one-line command" or "simple reference image guidance," automatically generating commercial-grade high-fidelity RGBA materials within seconds. This not only increases production efficiency by several orders of magnitude but also ensures the stability of material quality through standardized algorithm output, providing a crucial technological foundation for building a large-scale automated AIGC content production pipeline.

[0182] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0183] Based on the same inventive concept, this application also provides an image editing apparatus for implementing the image editing method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more image editing apparatus embodiments provided below can be found in the limitations of the image editing method described above, and will not be repeated here.

[0184] In one exemplary embodiment, such as Figure 4As shown, an image editing apparatus 400 is provided, which may include:

[0185] Image acquisition module 401 is used to acquire an image to be edited; the image to be edited includes color channel data and transparency channel data.

[0186] The feature extraction module 402 is used to extract features from the color channel data and the transparency channel data respectively, so as to obtain the initial color features and initial transparency features corresponding to the image to be edited.

[0187] The feature fusion module 403 is used to fuse the initial transparency feature and the initial color feature to obtain the fused feature;

[0188] The feature generation module 404 is used to perform feature generation processing on the fusion features according to the editing instructions for the image to be edited, so as to obtain the predicted fusion features; the predicted fusion features include predicted transparency features and predicted color features;

[0189] The feature processing module 405 is used to fuse the initial transparency features and the predicted transparency features to obtain the target transparency features;

[0190] The image generation module 406 is used to generate a target image corresponding to the image to be edited based on the target transparency channel data corresponding to the target transparency features and the target color channel data corresponding to the predicted color features.

[0191] In an exemplary embodiment, the feature generation module 404 is further configured to extract geometric features corresponding to the image to be edited and extract semantic features corresponding to the editing instructions; based on the geometric features and semantic features, the fused features are processed to generate predicted fused features.

[0192] In an exemplary embodiment, the feature generation module 404 is further configured to perform edge detection processing on the image to be edited to obtain an edge feature map of the image to be edited; perform depth recognition processing on the image to be edited to obtain a depth feature map of the image to be edited; and determine geometric features based on the edge feature map and the depth feature map.

[0193] In an exemplary embodiment, the feature fusion module 403 is further configured to use the initial transparency feature as an offset feature; and to perform element-wise addition of the offset feature and the initial color feature to obtain the fused feature.

[0194] In an exemplary embodiment, the feature processing module 405 is further configured to concatenate the initial transparency feature and the predicted transparency feature to obtain a mixed transparency feature; perform feature extraction and activation processing on the mixed transparency feature to obtain an attention map; and perform fusion processing on the initial transparency feature and the predicted transparency feature based on the attention map to obtain the target transparency feature.

[0195] In an exemplary embodiment, the feature processing module 405 is further configured to determine, based on the attention map, a first weight corresponding to the initial transparency feature and a second weight corresponding to the predicted transparency feature; perform weighted processing on the initial transparency feature according to the first weight to obtain a first weighted feature; perform weighted processing on the predicted transparency feature according to the second weight to obtain a second weighted feature; and perform fusion processing on the first weighted feature and the second weighted feature to obtain the target transparency feature.

[0196] Each module in the aforementioned image editing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0197] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image editing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0198] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0199] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0200] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above-described method embodiments.

[0201] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0202] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0203] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0204] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image editing method, characterized in that, The method includes: Obtain the image to be edited; the image to be edited includes color channel data and transparency channel data; Feature extraction is performed on the color channel data and the transparency channel data respectively to obtain the initial color features and initial transparency features corresponding to the image to be edited; The initial transparency feature and the initial color feature are fused together to obtain a fused feature; Based on the editing instructions for the image to be edited, the fusion features are processed to generate predicted blending features; the predicted blending features include predicted transparency features and predicted color features. The initial transparency feature and the predicted transparency feature are fused together to obtain the target transparency feature; Based on the target transparency channel data corresponding to the target transparency feature and the target color channel data corresponding to the predicted color feature, a target image corresponding to the image to be edited is generated.

2. The method according to claim 1, characterized in that, The step of performing feature generation processing on the fused features according to the editing instructions for the image to be edited, to obtain predicted fused features, includes: Extract the geometric features corresponding to the image to be edited, and extract the semantic features corresponding to the editing instructions; Based on the geometric features and the semantic features, the fused features are processed to generate the predicted fused features.

3. The method according to claim 2, characterized in that, The step of extracting the geometric features corresponding to the image to be edited includes: Edge detection processing is performed on the image to be edited to obtain the edge feature map of the image to be edited; The image to be edited is subjected to depth recognition processing to obtain a depth feature map of the image to be edited; The geometric features are determined based on the edge feature map and the depth feature map.

4. The method according to claim 1, characterized in that, The process of fusing the initial transparency feature and the initial color feature to obtain the fused feature includes: Use the initial transparency feature as the offset feature; The offset feature and the initial color feature are added element by element to obtain the fused feature.

5. The method according to claim 1, characterized in that, The process of fusing the initial transparency feature and the predicted transparency feature to obtain the target transparency feature includes: The initial transparency feature and the predicted transparency feature are concatenated to obtain a hybrid transparency feature; The mixed transparency features are subjected to feature extraction and activation processing to obtain an attention map; Based on the attention map, the initial transparency feature and the predicted transparency feature are fused to obtain the target transparency feature.

6. The method according to claim 5, characterized in that, The step of fusing the initial transparency feature and the predicted transparency feature based on the attention map to obtain the target transparency feature includes: Based on the attention map, determine the first weight corresponding to the initial transparency feature and the second weight corresponding to the predicted transparency feature; Based on the first weight, the initial transparency feature is weighted to obtain the first weighted feature; The predicted transparency feature is weighted according to the second weight to obtain the second weighted feature; The first weighted feature and the second weighted feature are fused to obtain the target transparency feature.

7. An image editing device, characterized in that, The device includes: The image acquisition module is used to acquire the image to be edited; the image to be edited includes color channel data and transparency channel data; The feature extraction module is used to extract features from the color channel data and the transparency channel data respectively to obtain the initial color features and initial transparency features corresponding to the image to be edited. The feature fusion module is used to fuse the initial transparency feature and the initial color feature to obtain the fused feature; The feature generation module is used to perform feature generation processing on the fusion features according to the editing instructions for the image to be edited, to obtain predicted blended features; the predicted blended features include predicted transparency features and predicted color features; The feature processing module is used to fuse the initial transparency feature and the predicted transparency feature to obtain the target transparency feature; The image generation module is used to generate a target image corresponding to the image to be edited based on the target transparency channel data corresponding to the target transparency feature and the target color channel data corresponding to the predicted color feature.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images