Method, device and electronic equipment for local editing of a render map

Abstract

This invention provides a method, apparatus, and electronic device for local editing of a rendered image. The method includes: mapping mask information representing the editing intention to the latent space of a diffusion model to generate a latent space mask; performing image encoding on the rendered image to be edited to obtain latent image features; determining conditional information based on the latent image features and editing instructions; and inputting the conditional information and the latent space mask into the diffusion model to generate a target rendered image. This invention generates a latent space mask that can define the boundary between the edited region and the preserved region in the rendered image at the latent space level by accurately mapping the mask information of the editing intention to the latent space of the diffusion model. The latent space mask applies differentiated processing to the edited region and the preserved region during the denoising generation process of the diffusion model. The generation of the edited region strictly follows the guidance of the editing instructions, effectively maintaining the original style and details of the preserved region, and ultimately ensuring the overall visual consistency of the target rendered image.

Description

Technical Field

[0001] This invention relates to the field of image rendering technology, and in particular to a method, apparatus, and electronic device for local editing of rendered images. Background Technology

[0002] Industrial product design is a highly iterative and complex process. Designers need to continuously adjust and optimize their designs based on market feedback, user research, engineering constraints, and other factors. In this process, localized modifications are among the most frequent operations. For example, in automotive styling, designers may need to experiment with different wheel styles, headlight designs, waistline designs, and grille styles; in consumer electronics design, they may need to adjust details such as button layouts, interface positions, and indicator light styles. Although these localized modifications involve a limited scope, they often have a significant impact on the overall visual appeal and market acceptance of the product.

[0003] In traditional industrial product design processes, local modifications face significant efficiency bottlenecks. When a designer needs to adjust a component based on a rendered image, the traditional process requires: first, modifying the 3D model in Computer-Aided Design (CAD) software; then, re-meshing and topology optimizing; next, configuring material and lighting parameters; and finally, starting the rendering engine to generate a new rendering. This process is time-consuming, with a single iteration potentially taking hours or even days, severely limiting the speed and depth of design exploration. Furthermore, the traditional process demands high levels of expertise; ordinary designers cannot independently complete complex editing operations and must rely on the collaboration of professional 3D modelers and rendering engineers. In recent years, deep learning technology has made groundbreaking progress in image generation and editing. Generative artificial intelligence technologies, represented by diffusion models, have demonstrated powerful image synthesis and editing capabilities, while image editing models such as InstructPix2Pix and Qwen-Image-Edit can modify images locally or globally based on natural language commands or user interaction. The emergence of these technologies has brought new possibilities to partial editing of industrial design. Designers can now quickly replace, modify, or add or remove parts through simple interactive operations, without the need for cumbersome 3D modeling and rendering processes.

[0004] However, when applying general image editing models to industrial design scenarios, existing methods struggle to accurately define the boundaries between the edited and preserved areas, leading to inconsistencies in style between the edited area and surrounding pixels. During the denoising process of diffusion models, complex information interactions exist between different regions of the image, and information from the edited area often diffuses outwards, resulting in noticeable seams, color jumps, or abrupt style changes at mask edges. Industrial product renderings demand high detail quality, and these boundary defects severely impact the overall visual appeal and professionalism of the image. Summary of the Invention

[0005] This invention provides a method, apparatus, and electronic device for local editing of rendered images, which solves the defects in the prior art when using diffusion models to edit industrial design images. Due to uncontrollable information diffusion between the editing area and the retained area during the denoising process, defects such as seams and color jumps are produced at the editing boundary, which cannot meet the requirements of industrial rendered images for detail consistency and professionalism.

[0006] This invention provides a method for local editing of a rendered image, comprising the following steps.

[0007] Obtain the rendered image to be edited, the editing instructions representing the editing intent, and the mask information of the editing intent; The mask information is mapped to the latent space of the diffusion model to generate a latent space mask aligned with the feature dimensions of the latent space; the latent space mask is used to define the boundary between the editable region and the retained region in the rendering image to be edited at the latent space level. The image to be edited is encoded to obtain potential image features. Based on the potential image features and the editing instructions, condition information is determined. The conditional information and the latent space mask are input into the diffusion model, which then performs denoising based on the conditional information and the latent space mask to generate the target rendering image of the image to be edited.

[0008] According to a method for local editing of a rendered image provided by the present invention, the diffusion model is obtained by iteratively performing the following training steps until a preset iteration cutoff condition is met: Obtain training samples; the training samples include the sample rendering image to be edited, sample editing instructions, sample mask information, and the real rendering image of the sample rendering image; Image encoding is performed on the sample rendering image to be edited to obtain sample latent image features. Sample condition information is constructed based on the sample latent image features and the sample editing instructions. The sample mask information is mapped to the initial latent space of the initial diffusion model to generate a sample latent space mask aligned with the feature dimensions of the initial latent space. The sample condition information and the sample latent space mask are input into the initial diffusion model, and the initial diffusion model outputs the prediction noise. Based on the difference between the predicted noise and the actual noise of the actual rendered image, an element-level loss map is determined; Based on the sample latent space mask, a first weight within the edit region of the element-level loss map and a second weight within the retain region of the element-level loss map are determined; the first weight is higher than the second weight. Based on the first weight and the second weight, a weight mask is determined. Based on the weight mask and the element-level loss map, a target loss is determined. Based on the target loss, the model parameters of the initial diffusion model are updated.

[0009] According to a rendering image local editing method provided by the present invention, the step of mapping the mask information to the latent space of a diffusion model to generate a latent space mask aligned with the feature dimensions of the latent space includes: The mask information is downsampled to obtain an intermediate state mask, and the intermediate state mask is divided into multiple mask patches; the size of the mask patches is the same as the size of the feature patches operated by the attention processing unit of the diffusion model. If there are pixel values ​​belonging to the editing area within the mask block, the state value of the spatial position of the mask block in the potential space is marked as edited; if there are no pixel values ​​belonging to the editing area within the mask block, the state value of the spatial position of the mask block in the potential space is marked as reserved. The latent space mask is generated based on the state values ​​of the spatial locations of all the mask tiles in the latent space.

[0010] According to a method for local editing of a rendered image provided by the present invention, determining condition information based on the latent image features and the editing instructions includes: The potential image features are determined as the main control channel features; Acquire at least one auxiliary control image, and perform image encoding on the auxiliary control image to obtain auxiliary control channel features; Based on the main control channel features and the auxiliary control channel features, feature fusion is performed to obtain multi-source fusion features; The condition information is determined based on the multi-source fusion features and the text embedding features corresponding to the editing instructions.

[0011] According to a rendering image local editing method provided by the present invention, the step of image encoding the auxiliary control image to obtain auxiliary latent features includes: If it is determined that the image size of the auxiliary control image is inconsistent with the image size of the rendering image to be edited, the image size of the auxiliary control image is adjusted based on the image size of the rendering image to be edited. The adjusted auxiliary control image is image encoded to obtain the auxiliary latent features; If the auxiliary control image and the image to be edited are of the same size, the auxiliary control image is image encoded to obtain the auxiliary latent features.

[0012] According to the present invention, a method for local editing of a rendered image is provided, wherein the auxiliary control image includes a first auxiliary control image and a second auxiliary control image; The first auxiliary control image is used to guide the target shape and structure of the editing area; The second auxiliary control image is used to guide the color and texture style of the editing area; The feature fusion based on the main control channel features and the auxiliary control channel features to obtain multi-source fused features includes: The latent image features, the first auxiliary latent features corresponding to the first auxiliary control image, and the second auxiliary latent features corresponding to the second auxiliary control image are concatenated along the channel dimension to obtain the multi-source fusion features.

[0013] According to a method for partial editing of a rendered image provided by the present invention, the step of determining the mask information includes: Receive normalized bounding box coordinates input by the user, map the normalized bounding box coordinates to pixel coordinates, and obtain the first mask information; Receive the free-form outline drawn by the user, fill the internal area of ​​the free-form outline, and obtain the second mask information; Identify the component regions in the rendering image to be edited, and generate third mask information based on the component regions; The user's natural language description is parsed, the image region corresponding to the natural language description is located, and a fourth mask information is generated based on the image region; The mask information is determined based on at least one of the first mask information, the second mask information, the third mask information, and the fourth mask information.

[0014] According to a method for partial editing of a rendered image provided by the present invention, the step of obtaining the rendered image to be edited includes: Obtain the original rendered image to be edited, and determine the aspect ratio of the original rendered image to be edited; An initial resolution is determined based on a preset target area value and the aspect ratio value. The initial resolution is then normalized to obtain a normalized resolution. The width and height values ​​of the normalized resolution are both integer multiples of the preset values. The resolution of the original rendering image to be edited is adjusted to the normalized resolution to obtain the rendering image to be edited.

[0015] The present invention also provides a rendering image partial editing device, comprising the following units: The acquisition unit is used to acquire the rendering image to be edited, the editing instructions representing the editing intention, and the mask information of the editing intention; A generation unit is used to map the mask information to the latent space of the diffusion model and generate a latent space mask that is aligned with the feature dimensions of the latent space; the latent space mask is used to define the boundary between the editable region and the retained region in the rendering image to be edited at the latent space level. The encoding unit is used to encode the rendered image to be edited to obtain potential image features, and to determine condition information based on the potential image features and the editing instructions; The input unit is used to input the condition information and the latent space mask into the diffusion model, and the diffusion model performs denoising processing based on the condition information and the latent space mask to generate the target rendering image of the rendering image to be edited.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the rendering image partial editing method as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the rendering partial editing method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the rendering image partial editing method as described above.

[0019] This invention provides a method, apparatus, and electronic device for local editing of rendered images. The method involves acquiring the rendered image to be edited, editing instructions representing the editing intent, and mask information representing the editing intent. The mask information is mapped to the latent space of a diffusion model to generate a latent space mask aligned with the feature dimensions of the latent space. Image encoding is performed on the rendered image to obtain latent image features. Based on the latent image features and editing instructions, conditional information is determined. The conditional information and the latent space mask are input into the diffusion model, which performs denoising processing to generate the target rendered image. This invention generates a latent space mask that can define the boundary between the edited and preserved regions in the rendered image at the latent space level by precisely mapping the mask information of the editing intent to the latent space of the diffusion model. The latent space mask can apply differentiated processing to the edited and preserved regions during the denoising generation process of the diffusion model. The generation of the edited region strictly follows the guidance of the editing instructions, while effectively maintaining the original style and details of the preserved region. This overcomes the technical limitations of general editing models in accurately controlling the editing boundary, avoiding problems such as seams, color jumps, or style abrupt changes at the mask edges, ultimately ensuring the overall visual consistency of the target rendered image and improving the user's viewing experience. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the local editing method for rendered images provided by the present invention.

[0022] Figure 2 This is a flowchart illustrating the training steps of the diffusion model provided by the present invention.

[0023] Figure 3 This is a schematic diagram of the process for generating a latent space mask provided by the present invention.

[0024] Figure 4 This is a flowchart illustrating the process of determining condition information provided by the present invention.

[0025] Figure 5 This is a schematic diagram of the structure of the rendering image local editing device provided by the present invention.

[0026] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and that the objects distinguished by "first," "second," etc., are generally of the same class.

[0029] This invention provides a method for partial editing of rendered images, which can be applied to computer-aided design systems, image editing software, or cloud image processing platforms in fields such as industrial design, game development, and virtual reality. Figure 1 This is a flowchart illustrating the local editing method for rendered images provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 110: Obtain the rendered image to be edited, the editing instructions representing the editing intent, and the mask information of the editing intent.

[0030] Specifically, firstly, it can obtain the rendered image to be edited, the editing instructions representing the editing intent, and the mask information of the editing intent.

[0031] The image to be edited refers to the initial image that needs to be modified locally. The image to be edited is usually the 3D rendering result of industrial products, such as the side view of a car, the appearance of consumer electronics products, and the rendering of architectural interior design. This embodiment of the invention does not make specific limitations on this.

[0032] The image to be edited and rendered can be a common image format, such as JPG or PNG, or a special format image containing depth or normal information.

[0033] Here, the editing instruction representing the editing intent refers to the instruction entered by the user to describe the content to be modified. Editing instructions are usually in the form of natural language text, such as "change the wheel rims to a five-spoke sporty style," "change the headlights to a slim design," or "change the body color to matte black." The editing instruction specifies what operation to perform on the selected area, such as replacing, modifying attributes, adding or removing components, etc. This embodiment of the invention does not specifically limit this.

[0034] Here, the mask information for editing intent is a binarized image or data matrix used to spatially indicate the area to be edited in the rendered image. The mask information can have the same dimensions as the rendered image to be edited, or it can have a corresponding proportional relationship with the rendered image to be edited. In the mask information, pixel values, such as 1 or 255, are typically displayed as white areas representing the editable area, while pixel values, such as 0, are typically displayed as black areas representing the reserved area.

[0035] The mask information can be drawn manually by user interaction, automatically generated by object detection algorithm, or generated based on text description by semantic segmentation model. This embodiment of the invention does not make specific limitations on this.

[0036] For example, in a car exterior design scenario, the rendered image to be edited is a side view photo of a car, and the user wants to change the wheel style. In this case, the editing instruction is "use carbon fiber double five-spoke wheels," and the mask information is a white pixel area that only covers the front and rear wheel areas, while the rest of the car body and background are black.

[0037] Step 120: Map the mask information to the latent space of the diffusion model to generate a latent space mask aligned with the feature dimensions of the latent space; the latent space mask is used to define the boundary between the editable region and the reserved region in the rendering image to be edited at the latent space level.

[0038] Specifically, after obtaining the mask information, the mask information can be mapped to the latent space of the diffusion model to generate a latent space mask aligned with the feature dimensions of the latent space. The latent space mask is used to define the boundary between the editable region and the reserved region in the rendering image to be edited at the latent space level.

[0039] Here, the editable area refers to the region in the rendered image where content is expected to change. The preserved area refers to the region in the rendered image where content is expected to remain unchanged.

[0040] To reduce computational complexity, mainstream generative diffusion models, such as Stable Diffusion (SD) and Latent Diffusion Models (LDMs), typically perform denoising and generation processes in a compressed latent space, rather than in the original pixel space. Therefore, the input pixel-level mask information must accurately map to the feature distribution in the latent space to ensure the accuracy of the editing boundaries.

[0041] Here, the latent space refers to the feature space in which the rendered image to be edited is located after being downsampled by an encoder such as a Variational Autoencoder (VAE). For example, if the resolution of the rendered image to be edited is 512×512, after being downsampled by 8 times by a VAE encoder, the resolution of its latent image features in the latent space may become 64×64.

[0042] To map mask information to the latent space of the diffusion model, coarse scaling methods such as nearest neighbor interpolation cannot be used simply, as this will cause jagged or misaligned mask boundaries in the latent space, leading to artifacts or unnatural stitching in the generated image at editing boundaries. The latent space mask generated in this step has a resolution similar to the features processed by the diffusion model. Figure 1 Furthermore, it can accurately reflect, numerically, which potential feature points belong to the editing area and which belong to the reserved area.

[0043] Here, the mapping process may include downsampling and special boundary handling mechanisms, such as max pooling or specific convolution operations, to ensure that edge information belonging to the edited region is not lost during feature compression, thereby clearly defining the boundary between the edited region and the retained region at the latent space level. This alignment operation ensures that the subsequent diffusion model can apply different processing logic to different latent feature points during denoising.

[0044] Step 130: Image encoding is performed on the rendering image to be edited to obtain potential image features. Based on the potential image features and the editing instructions, condition information is determined.

[0045] Specifically, the image to be edited can be image encoded to obtain potential image features, and condition information can be determined based on the potential image features and editing instructions.

[0046] Here, a pre-trained image encoder, such as the Encoder part of VAE, can be used to encode the rendered image to be edited. The image encoding process compresses high-dimensional pixel data into low-dimensional, high-density latent image features.

[0047] Among them, the latent image features retain the main structural and semantic information of the image to be edited and rendered, serving as the base map or reference basis for subsequent generation.

[0048] Here, the generation process of the diffusion model is controlled by conditional information, which acts as a guiding signal to drive the entire generation process. The latent image features contain the basic structure of the image to be edited and the texture details of the non-edited areas, ensuring that the final generated content is highly consistent with the original image in the areas to be retained. In specific implementations, latent image features can be used as one of the direct inputs to the diffusion model, or embedded in the model's forward pass as control signals.

[0049] Here, the editing instructions are in text form, which can be converted into text embedding features by a text encoder. The text embedding features are further integrated into multiple levels of the diffusion model through a cross-attention mechanism, thereby guiding the diffusion model to generate new content that conforms to the semantic description, such as realizing the shape and texture transformation of a wheel hub from its original style to a five-spoke style.

[0050] In summary, conditional information is a composite feature representation that integrates visual reference and semantic intent. The role of conditional information is to explicitly instruct the diffusion model how to generate target content that conforms to editing instructions based on given latent image features.

[0051] Step 140: Input the condition information and the latent space mask into the diffusion model, and the diffusion model performs denoising processing based on the condition information and the latent space mask to generate the target rendering image of the rendering image to be edited.

[0052] Specifically, after obtaining the conditional information, the conditional information and the latent space mask can be input into the diffusion model. The diffusion model then performs denoising based on the conditional information and the latent space mask to generate the target rendering image of the image to be edited.

[0053] Here, the diffusion model is a trained neural network whose core objective is to reconstruct a target rendering image that meets editing requirements from Gaussian noise through a stepwise reverse denoising process.

[0054] The diffusion model guides the semantic direction of the generated content based on the text embedding features of the editing instructions in the conditional information, such as generating car lights of a specific shape; at the same time, it uses the latent image features in the conditional information to maintain the structural framework of the generated content and the visual consistency of the unedited areas.

[0055] In each denoising step, a latent space mask is used to distinguish region processing strategies. The diffusion model uses the mask to identify the editable regions that need to be redrawn and the regions that must be preserved. For editable regions, the diffusion model mainly combines the semantic guidance of editing instructions with initial noise to generate content; for preserved regions, it uses strong constraints to preserve original features to ensure that the preserved regions are highly consistent with the rendered images to be edited in the target rendering image, thereby achieving a natural fusion between the editable and preserved regions.

[0056] After multiple iterations of denoising, the diffusion model outputs optimized latent features, which are then mapped to pixel space by the image decoder to obtain the final target rendering image.

[0057] In this process, the target rendering image remains consistent with the original rendering image to be edited within the reserved area, while the new content described by the editing instructions is presented within the editing area. Furthermore, the transition between the reserved area and the editing area is coordinated, and the relationship between light and shadow is reasonable and unified.

[0058] The method provided in this invention involves obtaining a rendering image to be edited, editing instructions representing the editing intent, and mask information of the editing intent; mapping the mask information to the latent space of a diffusion model to generate a latent space mask aligned with the feature dimensions of the latent space; image encoding the rendering image to be edited to obtain latent image features; determining conditional information based on the latent image features and the editing instructions; and inputting the conditional information and the latent space mask into the diffusion model for denoising to generate the target rendering image. This invention generates a latent space mask that can define the boundary between the edited and retained regions in the rendering image at the latent space level by precisely mapping the mask information of the editing intent to the latent space of the diffusion model. The latent space mask can apply differentiated processing to the edited and retained regions during the denoising generation process of the diffusion model. The generation of the edited region strictly follows the guidance of the editing instructions, while effectively maintaining the original style and details of the retained region. This overcomes the technical limitations of general editing models in accurately controlling the editing boundary, avoiding problems such as seams, color jumps, or style abrupt changes at the mask edges, ultimately ensuring the overall visual consistency of the target rendering image and improving the user's viewing experience.

[0059] In related technologies, maintaining style consistency in unmodified areas is a key challenge during the editing process. The style of a rendered image involves multiple dimensions such as lighting, materials, color, and contrast, and these elements need to be consistent across the entire image. Editing operations may disrupt this consistency, resulting in a mismatch between the style of unmodified areas and the edited areas. Existing methods typically employ simple weighted blending strategies, which struggle to preserve the overall style of the original image while maintaining the new features of the edited areas.

[0060] Based on the above embodiments, the diffusion model is obtained by iteratively executing the following training steps until a preset iteration cutoff condition is met: Step 210: Obtain training samples; the training samples include the sample rendering image to be edited, sample editing instructions, sample mask information, and the real rendering image of the sample rendering image; Step 220: Image encoding is performed on the sample rendering image to be edited to obtain sample latent image features. Sample condition information is constructed based on the sample latent image features and the sample editing instructions. The sample mask information is mapped to the initial latent space of the initial diffusion model to generate a sample latent space mask aligned with the feature dimensions of the initial latent space. Step 230: Input the sample condition information and the sample latent space mask into the initial diffusion model, and output the prediction noise from the initial diffusion model; Step 240: Determine an element-level loss map based on the difference between the predicted noise and the actual noise of the actual rendered image; Step 250: Based on the sample latent space mask, determine a first weight within the edit region of the element-level loss map and a second weight within the retention region of the element-level loss map; the first weight is higher than the second weight. Step 260: Based on the first weight and the second weight, determine a weight mask; based on the weight mask and the element-level loss map, determine a target loss; and update the model parameters of the initial diffusion model based on the target loss.

[0061] Specifically, the training process in this embodiment of the invention aims to enable the diffusion model to learn how to accurately modify the edited region according to the editing instructions while maintaining style consistency in the preserved region.

[0062] Specifically Figure 2 This is a flowchart illustrating the training steps of the diffusion model provided by the present invention, as shown below. Figure 2 As shown, the diffusion model in this embodiment is obtained by iteratively executing training steps until a preset iteration cutoff condition is met. The preset iteration cutoff condition can be reaching a preset number of training rounds, or the convergence of the target loss, etc., and this embodiment of the invention does not specifically limit it.

[0063] The training steps specifically include: First, training samples can be obtained. These samples include the sample rendering image to be edited, sample editing instructions, sample mask information, and the real rendering image of the sample rendering image. The real rendering image refers to the final image corresponding to the sample rendering image to be edited, after the desired editing has been completed. The sample editing instructions are text describing the changes from the sample rendering image to the real rendering image. The sample mask information is a binary image that precisely marks the regions where changes occur between the sample rendering image to be edited and the real rendering image.

[0064] Then, the sample rendering image to be edited can be image encoded to obtain the sample latent image features. Based on the sample latent image features and sample editing instructions, sample condition information can be constructed. The sample mask information can be mapped to the initial latent space of the initial diffusion model to generate a sample latent space mask that is aligned with the feature dimensions of the initial latent space.

[0065] Here, the parameters of the initial diffusion model can be preset or randomly generated, and the embodiments of the present invention do not impose specific limitations on this.

[0066] Furthermore, sample condition information and sample latent space masks can be input into the initial diffusion model, which then outputs the predicted noise. In the standard diffusion model training framework, known real noise is first added to the real rendered image or its latent features to construct noisy samples; then, the noisy samples and their corresponding sample condition information, as well as the latent space mask, are input into the initial diffusion model, which predicts the added real noise, and its output is the predicted noise.

[0067] After obtaining the predicted noise, an element-level loss map can be determined based on the difference between the predicted noise and the actual noise of the rendered image, as shown in the following formula: ; in, Represents the element-level loss plot. Indicates prediction noise, This represents the actual noise in a realistic rendering.

[0068] It's important to clarify that the diffusion model operates within the latent space. Therefore, the true noise in the real rendered image does not refer to pixel-space noise, but rather to a known noise tensor added to the latent features of the real rendered image in the latent space. Correspondingly, the predicted noise from the initial diffusion model is also a latent space noise tensor with the exact same size as the true noise tensor.

[0069] Here, the element-level loss map occurs at the latent space level. It can be obtained by calculating the mean squared error (MSE) between the predicted noise and the real noise of the real rendered map at each latent feature point. The element-level loss map has the same size as the feature map in the latent space. The value of each element in the element-level loss map represents the accuracy of the initial diffusion model prediction at that location.

[0070] For example, if the dimensions of the latent image features obtained after encoding the image to be edited are (number of channels C, height H, width W), such as (4, 64, 64), then the dimensions of both the real noise and the predicted noise are also (4, 64, 64). The dimensions of the final calculated element-level loss map are also (4, 64, 64). Each element value in the element-level loss map represents the accuracy of the initial diffusion model's prediction on a specific channel (c) at the corresponding position (h, w) in the latent space.

[0071] Based on the sample latent space mask, the first weight within the edit region of the element-level loss map and the second weight within the retain region of the element-level loss map are determined, wherein the first weight is higher than the second weight.

[0072] Here, the edit region refers to the area in the sample rendering image where content changes are expected, and its spatial location is defined by the sample latent space mask. The retainable region refers to the area in the sample rendering image where content is expected to remain unchanged. To achieve differentiated supervision, the first weight is set higher than the second weight. For example, the first weight can be set to 2.0, while the second weight can be set to 1.0. This means that the initial diffusion model's prediction error in the edit region will be penalized more severely.

[0073] To illustrate the differential weighting process more specifically, this invention proposes a mask-aware differential loss function, which can be called MaskEditLoss. Its detailed algorithm flow is as follows: Optionally, a weighting can be assigned based on the time step t of the current denoising process. This time step weight can be designed to decrease with t, that is, a smaller weight is given in the early stages of denoising (when t is large and the noise level is high), focusing on the generation of the overall image contour; and a larger weight is given in the later stages of denoising (when t is small and the noise level is low), focusing on the fine depiction of image details. This progressive learning strategy helps to improve the quality of the generated image.

[0074] Finally, based on the first and second weights, a weight mask is determined. Based on the weight mask and the element-level loss map, the target loss MaskEditLoss is determined. Based on the target loss, the model parameters of the initial diffusion model are updated.

[0075] Specifically, it can be based on editing the mask. Generate region weights and obtain the weight mask ( Specifically, for the edit region (mask value m=1), the first weight, also known as the foreground weight, is used. ),For example =2.0; For the reserved region (mask value m=0), the second weight, also known as the background weight, is used. ),For example =1.0. The formula for calculating the weight mask is: ; in, Indicates the weight mask, Indicates the editing area. Indicates the first weight. Indicates the reserved area. This indicates the second weight.

[0076] Apply weights and aggregate: Combine the generated weight mask ( ) Applied to element-level loss maps ( The weighted loss is obtained by calculating the weighted loss, and the formula for the weighted loss is as follows: ; in, Indicates weighted loss. Represents the element-level loss plot. This represents the weight mask.

[0077] Finally, the weighted losses are aggregated according to the preset aggregation method (reduction parameter) to obtain the final target loss. For example, aggregation can be done by summing or by calculating the mean, as shown in the following formula: ; in, Indicates target loss. This indicates the weighted loss.

[0078] The method provided in this invention determines a first weight within the edit region and a second weight within the retain region of the element-level loss map based on a sample latent space mask. The first weight is higher than the second weight. Based on the first and second weights, the edit region and retain region in the element-level loss map are weighted respectively to obtain the target loss. The model parameters of the initial diffusion model are then updated based on the target loss. This method introduces mask-based differentiated weights, namely the first and second weights, into the target loss. This forces the initial diffusion model to pay more attention to the generation quality of the edit region during training, while suppressing unnecessary modifications to the retain region. This allows the model to learn the ability to maintain clear boundaries and consistent style in local editing tasks, solving the problem of inconsistent style between the edit region and surrounding pixels in existing technologies, and improving the rendering quality of the target rendering map.

[0079] In related technologies, diffusion models typically operate in the latent space of a Visual Image Encoder (VAE). The input image is first compressed to a low-dimensional representation by a VAE encoder. VAE encoding uses 8x downsampling to compress the 512×512 input image into a 64×64 latent feature map. This downsampling process leads to a loss of spatial information, making it impossible for a simple pixel space mask to accurately correspond to the edit region in the latent space. Existing methods typically apply the pixel space mask directly to the latent space after downsampling, ignoring details such as patching, resulting in a discrepancy between the mask boundaries and the actual edit region.

[0080] Based on the above embodiments, step 120 includes: Step 1201: Downsample the mask information to obtain an intermediate state mask, and divide the intermediate state mask into multiple mask patches; the size of the mask patches is the same as the size of the feature patches operated by the attention processing unit of the diffusion model; Step 1202: If there are pixel values ​​belonging to the editing area in the mask block, mark the state value of the spatial position of the mask block in the potential space as edited; if there are no pixel values ​​belonging to the editing area in the mask block, mark the state value of the spatial position of the mask block in the potential space as reserved. Step 1203: Generate the latent space mask based on the state values ​​of the spatial positions of all the mask blocks in the latent space.

[0081] Specifically, if the resolution of the image to be edited is 512×512, after being downsampled 8 times by the VAE encoder, the resolution of its latent image features in the latent space may become 64×64. This downsampling process leads to the loss of spatial information, making it impossible for a simple pixel space mask to accurately correspond to the editing area in the latent space. Existing methods typically apply the pixel space mask directly to the latent space after downsampling, ignoring details such as patching, resulting in a deviation between the mask boundaries and the actual editing area, leading to insufficient mask mapping accuracy.

[0082] Figure 3 This is a schematic diagram of the process for generating a latent space mask provided by the present invention, as shown below. Figure 3 As shown, to address this issue, this embodiment provides a specific implementation method for accurately mapping mask information from the pixel space to the latent space of the diffusion model, ensuring that editing instructions can be precisely executed at the feature processing level of the diffusion model. Since the diffusion model, especially the Transformer-based model, internally processes compressed and reconstructed latent feature sequences, it is essential to ensure that the mask information also undergoes the same transformation process to achieve strict alignment with the feature sequence.

[0083] The specific steps of this mapping method are as follows: First, the mask information is downsampled to obtain an intermediate state mask. This intermediate state mask is then divided into multiple mask patches, where the size of each patch is the same as the size of the feature patches operated on by the attention processing unit of the diffusion model. Here, the mask information in the pixel space is downsampled in alignment with the variational autoencoder encoding process to obtain the intermediate state mask. For example, if the VAE encoder performs 8x downsampling on the input image, this step also uses 8x downsampling. For instance, using an average pooling operation with a kernel size of 8 and a stride of 8 to process the mask information (size [H, W]), an intermediate state mask of size [H / 8, W / 8] is obtained, as shown in the following formula: .

[0084] Understandably, the advantage of average pooling is that it can preserve the overall information of the mask and avoid the loss of boundary information that may be caused by max pooling.

[0085] Here, the intermediate state mask refers to a mask whose resolution is reduced from the original pixel-level mask information to match the resolution of the potential image features processed by the diffusion model. For example, average pooling can be used to downsample the mask information by a factor of 8 to match the compression ratio of the VAE encoder. The mask patch refers to a grid partition of the intermediate state mask according to a specific size. The size of the mask patch is the same as the size of the feature map patches operated on by the attention processing unit of the diffusion model. For example, if the Transformer structure inside the diffusion model packages feature maps into 2×2 patches for processing, then the size of the mask patch should also be set to 2×2.

[0086] Secondly, the patching operation in the Transformer architecture is simulated to reshape and adjust the dimensions of the intermediate state mask. In a typical implementation, a latent feature of size [H / 8, W / 8] undergoes a 2×2 patching operation before being fed into the Transformer, halving the spatial dimensions again and stacking the 2×2=4 spatial information pieces onto the channel dimension. To maintain alignment, the same operation is performed on the intermediate state mask: its size is reshaped from [B, H / 8, W / 8] and the dimension order is adjusted to obtain a tensor of size [B, H / 16, W / 16, 4]. Here, B is the batch size, H / 16 and W / 16 are the new spatial dimensions after packing, and 4 is the new channel dimension generated by the packing operation.

[0087] Then, a patch-wise max pooling operation is performed to determine the state of each mask patch. This step involves performing max pooling on the four generated channels. The tensor [B, H / 16, W / 16, 4] is compressed to [B, H / 16, W / 16]. The core idea of ​​this design is that for any 2×2 mask patch, if any pixel (or feature point) in the mask patch belongs to the edit region (its mask value is 1), after max pooling, the final mask value representing the mask patch will also be 1, i.e., it is marked as edited. Conversely, the mask patch is only marked as preserved when all pixels in the mask patch do not belong to the edit region.

[0088] Here, the edit state indicates that the mask tile corresponding to the potential spatial location needs content editing; the retain state indicates that the mask tile corresponding to the potential spatial location should retain its original content. This design ensures that even tiny editable areas are not ignored during dimensionality reduction and packaging, thus avoiding jagged edges or holes at the edit boundaries. In other words, this step ensures that if any small part of a mask tile belongs to the editable area, the entire mask tile is considered to need editing, thus avoiding the loss of subtle editing details during downsampling.

[0089] Finally, based on the state values ​​of all mask patches in the latent space, a latent space mask is generated. The latent space mask can be a new two-dimensional feature map composed of all state values, or it can be a flattened one-dimensional sequence whose dimensions are perfectly aligned with the dimensions of the feature sequence processed internally by the diffusion model. For example, by flattening the sequence, the final latent space mask is generated. The two-dimensional mask map of size [B, H / 16, W / 16] obtained in the previous step is flattened along the spatial dimensions to form a one-dimensional sequence. The length of this one-dimensional sequence is... The length of this sequence is exactly the same as the length of the latent sequence actually processed by the Transformer model. The resulting one-dimensional sequence is the latent space mask.

[0090] The method provided in this invention maps mask information from pixel space to latent space by using mask patch state marking. In particular, it marks the state of the entire mask patch by judging whether there are editable pixels within the mask patch. This can simulate the subsequent patching processing details of the diffusion model during dimensionality compression, so that the generated latent space mask is spatially precisely aligned with the feature patches processed inside the diffusion model. This method breaks through the limitations of traditional methods that only perform simple downsampling on the mask and ignore the internal data structure of the model, such as patch packaging. It avoids the spatial information loss caused by downsampling and the resulting deviation between the latent space mask boundary and the actual editable area. Ultimately, it ensures that the diffusion model can obtain accurate spatial guidance during the generation process, thereby achieving a clear and seamless editing effect.

[0091] In related technologies, practical industrial design scenarios often require referencing multiple images simultaneously. For example, a designer might want to retain the overall shape and style of the original car body, only replacing the headlights; or they might want to apply the color scheme of car A to the styling of car B. These requirements involve the fusion of multi-source control signals—requiring simultaneous reference to the structural information of the original image, the style information of the reference image, and the semantic information of the editing instructions. Existing models typically only support single control signals, making it difficult to effectively fuse multi-source control information.

[0092] Based on the above embodiments, step 130, determining the condition information based on the potential image features and the editing instructions, includes: Step 1301: Determine the potential image features as the main control channel features; Step 1302: Obtain at least one auxiliary control image, and perform image encoding on the auxiliary control image to obtain auxiliary control channel features; Step 1303: Based on the main control channel features and the auxiliary control channel features, feature fusion is performed to obtain multi-source fusion features; Step 1304: Determine the condition information based on the multi-source fusion features and the text embedding features corresponding to the editing instructions.

[0093] Specifically, Figure 4 This is a flowchart illustrating the process of determining condition information provided by the present invention, such as... Figure 4 As shown, firstly, the latent image features are determined as the main control channel features. These main control channel features provide the diffusion model with the overall structural layout and global contextual information of the original image to be edited, thus maintaining the consistency between the target rendered image and the image to be edited. Figure 1 The foundation of consistency.

[0094] Secondly, acquire at least one auxiliary control image and perform image encoding on the auxiliary control image to obtain the auxiliary control channel features.

[0095] In this context, the auxiliary control image refers to a user-provided reference image used to provide additional guidance for editing. The auxiliary control channel features are the representation of the auxiliary control image in the latent space, used to inject style and / or structural information into the diffusion model.

[0096] After obtaining the main control channel features and the auxiliary control channel features, feature fusion can be performed based on these features to obtain multi-source fused features. For example, the main control channel features and the auxiliary control channel features can be concatenated along the channel dimension to obtain multi-source fused features.

[0097] Here, multi-source fusion features are a richer latent feature representation that includes image information of the original render image to be edited and information of all auxiliary reference images in the channel dimension.

[0098] Finally, conditional information is determined based on the multi-source fusion features and the text embedding features corresponding to the editing instructions. This means that the final conditional information includes the structure from the original rendering image to be edited, style information and structural information from the auxiliary images, as well as the semantic intent from the text instructions.

[0099] The method provided in this invention constructs multi-source fusion features based on the main control channel features representing the global structure of the rendered image to be edited and at least one auxiliary control channel feature. This allows for the unified integration of visual control signals from different image sources before generating the target rendered image. This enables the subsequent diffusion model to perform denoising under a unified and more informative conditional information. This overcomes the limitations of existing technologies where models only support a single control signal and struggle to effectively fuse multi-source control information. It avoids the problem of editing failure or unsatisfactory results due to the inability to simultaneously satisfy multiple visual constraints. Ultimately, this method ensures accurate response to and realization of composite editing intentions from multiple reference sources, significantly improving editing capabilities and flexibility in professional scenarios such as industrial design.

[0100] Based on the above embodiments, step 1302, which involves image encoding the auxiliary control image to obtain auxiliary latent features, includes: Step 310: If it is determined that the image size of the auxiliary control image is inconsistent with the image size of the rendering image to be edited, adjust the image size of the auxiliary control image based on the image size of the rendering image to be edited; Step 320: Image encoding is performed on the adjusted auxiliary control image to obtain the auxiliary latent features; Step 330: If it is determined that the auxiliary control image and the image to be edited have the same image size, the auxiliary control image is image encoded to obtain the auxiliary latent features.

[0101] Specifically, a crucial preprocessing step is required before encoding the auxiliary control images to ensure that the format of all input images conforms to the diffusion model requirements. Specifically, the main control image (i.e., the image to be rendered) and all auxiliary control images are processed separately by a preprocessor. The preprocessor's operations include: When the auxiliary control image and the rendered image to be edited have different image sizes, the size of the auxiliary control image is adjusted based on the size of the rendered image to be edited. For example, bilinear interpolation or bicubic interpolation algorithms can be used to scale the auxiliary control image to the exact same resolution as the rendered image to be edited. This step is a prerequisite for ensuring that the subsequently generated potential features have the same spatial dimension and is a necessary condition for effective feature fusion.

[0102] The preprocessor also normalizes the pixel values ​​of all control images. For example, it converts pixel values ​​from an integer range of [0, 255] to a floating-point range of [-1, 1] or [0, 1] to meet the input requirements of the image encoder. After preprocessing, the adjusted auxiliary control images are image encoded to obtain auxiliary latent features.

[0103] It is understandable that if the image size of the auxiliary control image is consistent with that of the image to be edited and rendered, then there is no need to adjust the image size of the auxiliary control image, and the auxiliary control image is encoded to obtain auxiliary latent features.

[0104] Based on the above embodiments, the auxiliary control image includes a first auxiliary control image and a second auxiliary control image; The first auxiliary control image is used to guide the target shape and structure of the editing area; The second auxiliary control image is used to guide the color and texture style of the editing area; Step 1303 includes: The latent image features, the first auxiliary latent features corresponding to the first auxiliary control image, and the second auxiliary latent features corresponding to the second auxiliary control image are concatenated along the channel dimension to obtain the multi-source fusion features.

[0105] Specifically, the auxiliary control image includes a first auxiliary control image and a second auxiliary control image. The first auxiliary control image is used to guide the target shape and structure of the editing area, and the second auxiliary control image is used to guide the color and texture style of the editing area.

[0106] Accordingly, the latent image features, the first auxiliary latent features corresponding to the first auxiliary control image, and the second auxiliary latent features corresponding to the second auxiliary control image can be concatenated along the channel dimension to obtain the multi-source fusion features, as shown in the following formula: ; in, Indicates multi-source fusion characteristics, Represents latent image features, Indicates the first auxiliary latent feature, Indicates the second auxiliary latent feature, This indicates a splicing operation.

[0107] It is understood that this invention can process a variety of control images of different sizes and from different sources, and allows users to control the structure and style of the generated content separately by providing different auxiliary images. This enables more refined and intuitive composite control of the editing process, solves the difficulties in integrating multi-source control signals and the stringent requirements for input formats in the prior art, and greatly expands the application scenarios and editing capabilities of the method.

[0108] This method leverages the powerful generative capabilities of the diffusion model, combined with mask-aware differential loss and multi-control image mechanisms, to achieve precise and natural editing of specific components or regions in the rendered image. It solves the problems of blurred editing boundaries, inconsistent styles, and difficulty in fusing multi-source control signals in existing technologies.

[0109] In related technologies, the general-purpose editing model lacks sufficient understanding of industrial product terminology, making it difficult to accurately execute editing commands. The field of industrial design contains numerous specialized terms, such as "sporty front end," "aerodynamic kit," "floating roof," and "continuous taillights," which describe specific combinations of design elements and styling features. The general-purpose model's training data primarily comes from generic images on the internet, resulting in a weak understanding of these specialized concepts.

[0110] Based on the above embodiments, the step of determining the mask information includes: Step 410: Receive the normalized bounding box coordinates input by the user, map the normalized bounding box coordinates to pixel coordinates, and obtain the first mask information; Step 420: Receive the freeform shape outline drawn by the user, fill the internal area of ​​the freeform shape outline, and obtain the second mask information; Step 430: Identify the component regions in the rendering image to be edited, and generate third mask information based on the component regions; Step 440: Parse the user's natural language description, locate the image region corresponding to the natural language description, and generate fourth mask information based on the image region; Step 450: Determine the mask information based on at least one of the first mask information, the second mask information, the third mask information, and the fourth mask information.

[0111] Specifically, the system receives normalized bounding box coordinates input by the user and maps these coordinates to pixel coordinates to obtain the first mask information.

[0112] In this method, the user specifies the editing area by inputting one or more rectangles through the interactive interface. The normalized bounding box coordinates refer to a list of coordinate values ​​between 0 and 1 relative to the overall image size, for example, in the format [(x1, y1, x2, y2), ...], where (x1, y1) and (x2, y2) represent the relative coordinates of the top-left and bottom-right corners of the rectangle, respectively.

[0113] After receiving the normalized bounding box coordinates, the system multiplies the normalized bounding box coordinates by the actual width and height of the image to convert them into pixel coordinates, which are the absolute positions in the image pixel matrix.

[0114] Here, the first mask information is a binary image generated based on pixel coordinates. The pixel value of the first mask information is 1 (or 255) within the specified rectangular area, and 0 in the remaining areas. This mode supports specifying multiple editing areas simultaneously, and is particularly suitable for scenarios involving batch editing of standardized parts.

[0115] In addition, the system can receive freeform contours drawn by the user, fill the internal areas of the freeform contours, and obtain second mask information. In this mode, users can use input devices such as a mouse, stylus, or finger to directly draw irregular closed curves on the image to be edited; these irregular closed curves constitute the freeform contour. The system captures the trajectory of the freeform contour and automatically executes a filling algorithm, such as a scanline fill algorithm, setting the pixel value of the internal area enclosed by the freeform contour to 1 (or 255), thereby generating the second mask information. This mode grants users a high degree of freedom and is suitable for precisely outlining components with complex or irregular shapes, such as car wheel rims, headlights, and rearview mirrors.

[0116] In one alternative embodiment, component regions in the rendering to be edited can be identified, and third mask information can be generated based on these component regions.

[0117] In this approach, the system integrates one or more pre-trained object detection or instance segmentation models, such as YOLO and Mask R-CNN. These models, trained on a large number of industrial product images, can automatically identify common standardized component regions in the image to be edited, such as wheels, windows, doors, and lights, and output their precise pixel-level segmentation masks. These pixel-level segmentation masks are directly used as third-party mask information. This mode fully automates the selection process of the editing region, significantly reducing the user's interaction costs, and is particularly suitable for rapid design iterations of standardized products.

[0118] In one optional embodiment, the system can parse the user's natural language description, locate the image region corresponding to the natural language description, and generate a fourth mask information based on the image region. In this approach, the user only needs to input text describing the desired editing location, such as "headlight area," "driver's side door," or "left side mirror." The system uses a Vision-Language Model (VLM) to understand the semantics of this natural language description and associates it with the content of the rendered image to be edited, thereby automatically locating the image region pointed to by the text. Subsequently, the system generates a corresponding mask based on this location result, serving as the fourth mask information. This mode provides the most intuitive and convenient interaction method, further lowering the barrier to entry for using professional software.

[0119] Ultimately, the system can determine the final mask information used to guide the editing process based on at least one of the first mask information, the second mask information, the third mask information, and the fourth mask information, or a combination thereof, for example, by merging multiple masks through a Boolean "OR" operation.

[0120] It should be noted that the rendering image local editing method in this embodiment of the invention offers flexible interaction, supporting multiple region selection methods such as bounding boxes, free drawing, automatic detection, and semantic specification, meeting the needs of different scenarios. The interactive interface is intuitive and easy to use, lowering the barrier to entry. Furthermore, it has broad format compatibility, automatically recognizing various input formats without requiring manual conversion by the user. Batch processing is supported, improving processing efficiency.

[0121] The method provided by this invention reduces the barrier to user operation and improves the flexibility and efficiency of interaction through diverse mask generation methods. It can meet the usage needs of different scenarios, from rapid prototyping to fine-tuning, thereby enhancing the practicality and user experience of the entire editing method.

[0122] Based on the above embodiments, obtaining the rendering image to be edited in step 110 includes: Step 110-1: Obtain the original rendering image to be edited and determine the aspect ratio of the original rendering image to be edited; Step 110-2: Determine the initial resolution based on the preset target area value and the aspect ratio value, and perform numerical normalization on the initial resolution to obtain the normalized resolution; the width and height values ​​of the normalized resolution are both integer multiples of the preset values. Step 110-3: Adjust the resolution of the original rendering image to be edited to the normalized resolution to obtain the rendering image to be edited.

[0123] Specifically, in practical applications, the resolution of the original images to be edited and rendered uploaded by users can vary greatly. Furthermore, the performance and efficiency of diffusion models, especially those based on the Transformer architecture, often have specific requirements regarding the size of the input feature maps; for example, the size must be an integer multiple of a certain preset value. To address this issue, this embodiment provides a resolution adaptive adjustment method, specifically including: First, obtain the original image to be edited and determine its aspect ratio. The original image to be edited refers to the initial image uploaded by the user without any processing. The aspect ratio is the ratio of the image's height to its width.

[0124] Secondly, the initial resolution is determined based on the preset target area and aspect ratio values. After determining the initial resolution, it can be numerically normalized to obtain the normalized resolution. This step aims to calculate an optimal resolution that maintains the original image proportions while meeting the input requirements of the diffusion model. The specific calculation process is as follows: The system can set a target area value ( The target area value represents the desired total number of pixels in the output image, which can be preset based on computing resources or application scenarios. Combined with the aspect ratio value obtained from the original image to be edited, the initial width and height can be calculated using the following formula: ; in, This represents the square root operation. This step allows us to obtain an initial resolution close to the target processing scale while maintaining the original image aspect ratio.

[0125] Here, the normalized resolution refers to a resolution where both the width and height values ​​are integer multiples of a preset value (e.g., 32). This preset value typically depends on the partition size of the feature map patches in the diffusion model architecture. The normalization operation can be performed according to the following formula: ； The `round` function performs a rounding operation. This step ensures that the final resolution size is divisible by 32, thus perfectly matching the internal architecture of the diffusion model.

[0126] In some cases, to control video memory usage or computation time, the system can set a maximum resolution limit. When the overall resolution exceeds the maximum resolution limit, the system will scale it based on the area ratio while maintaining the aspect ratio. For example, if the calculated width or height exceeds the maximum limit, the width and height will be scaled down proportionally so that the adjusted total area does not exceed the area corresponding to the maximum limit, while keeping the height / width ratio unchanged.

[0127] Finally, the resolution of the original image to be edited is adjusted to the final normalized resolution calculated in the above process using image scaling algorithms, such as bilinear interpolation or bicubic interpolation, so as to obtain an image to be edited that can be directly input into subsequent processes.

[0128] The method provided in this embodiment of the invention, through the above-described resolution adaptive adjustment method, can automatically process input images of any resolution without requiring manual preprocessing by the user. This ensures the compatibility between the input image to be edited and the internal processing architecture of the diffusion model, such as the tile partitioning of the attention mechanism, thereby avoiding calculation errors or performance degradation that may be caused by resolution mismatch and improving the robustness and ease of use of local editing.

[0129] Based on any of the above embodiments, the rendering image partial editing method of the present invention can be deployed in an intelligent generation system running on a terminal device, such as a personal computer, workstation, or cloud server, providing industrial designers with an interactive rendering image partial editing platform. Designers first upload an original rendering image of an industrial product to be edited, such as a side view of a car. To address the issue of diverse image formats and resolutions uploaded by users, the system first performs a series of automated preprocessing steps. The system integrates an automatic image layout inference mechanism, which intelligently identifies the data layout based on the dimension (nd), number of channels, and numerical range of the input data tensor. For example, it can determine whether the input is in two-dimensional HW (height × width) format, three-dimensional CHW (channels × height × width) or HWC (height × width × channels) format, or even four-dimensional BCHW (batch × channels × height × width) format, and automatically convert the format based on the inference result, ensuring that the layout and value range of all input data (including auxiliary control images) meet the processing requirements of the model. At the same time, the system will also perform resolution adaptive calculation to adjust the image to a regular resolution that maintains the original aspect ratio and meets the model alignment requirements, such as a width and height that are both integer multiples of 32, to obtain the image to be edited and rendered.

[0130] Next, the designer selects the area to be edited through the interactive interface. The system provides several flexible area selection methods, including bounding box input, free drawing, automatic detection, and semantic specification. After determining the editing area, the system executes a detailed mask generation algorithm: first, it creates a zero matrix with the same size as the rendered image to be edited as the mask base; then, for each user-specified editing area, it calculates its specific range in pixel space and sets the pixel value of the base matrix within that range to 255 (white); finally, morphological processing (such as opening or closing operations) is optionally used to optimize the mask boundaries to remove isolated noise or fill small internal holes, ultimately outputting a high-quality single-channel binary mask. Subsequently, this pixel-space mask information is precisely mapped to the latent space of the diffusion model through the mask space mapping mechanism proposed in this invention (including 8x downsampling and 2×2 patch-wise max pooling operations), generating a latent space mask. If a designer uploads one or more auxiliary control images (such as reference styling images or style example images), the system will execute a multi-control image fusion mechanism. After preprocessing all control images (including size alignment and normalization), VAE encoding is performed on each image, and these latent features are concatenated along the channel dimension to form multi-source fusion features. Finally, the designer inputs editing instructions in text form. These instructions, together with the multi-source fusion features, constitute the final conditional information. This conditional information, along with the latent space mask, is input into the diffusion model, and the target rendering image is generated through denoising. During the training phase of the diffusion model, a differential loss calculation method based on MaskEditLoss is used to optimize the model parameters. The entire system supports recording the editing history, enabling undo and redo of multiple editing operations. The completed editing results can be exported to standard image formats such as PNG / JPEG for direct use in design reviews.

[0131] To more clearly illustrate this invention, two specific application scenarios are described below. Scenario 1: Replacement of car wheel rim style. The input configuration includes: a side view of a car as the original rendering image; a left front wheel rim area specified by a bounding box input method as an editing mask; a five-spoke sporty wheel rim image as an auxiliary control image; and "change the wheel rim to a five-spoke sporty style" as the control command. When the system executes, it first extracts and maps the mask, then loads the auxiliary control image as the first auxiliary control image (control_1), and finally performs inference under the differential supervision of MaskEditLoss. The generated result is: the left front wheel rim is accurately replaced with a five-spoke sporty style, the transition between the wheel rim and the body and tire is natural, other areas of the image (such as the body and background) are completely consistent with the original image, and there are no obvious seams at the editing boundary. Scenario 2: Modification of car headlight design. The input configuration includes: an image of a car's front end as the original rendering; a circular headlight area outlined using freehand drawing as an editing mask; a reference image containing slender headlights as an auxiliary control image; and the control command "change the circular headlights to a slender shape". The final output is: the car's headlights are successfully changed from circular to slender, maintaining overall vehicle style consistency with the original image. The modified headlight light effect is natural and realistic, fully meeting the design review's requirements.

[0132] In summary, this invention offers significant advantages over existing technologies. Regarding editing boundaries, it achieves precise definition and seamless integration of the editing area through a latent space mask mapping mechanism. In terms of style consistency, relying on the MaskEditLoss differential loss mechanism, it effectively maintains the original style of the retained area and allows the modified area to blend naturally with surrounding pixels. At the functional application level, through a multi-control image fusion mechanism, it not only supports various editing operations such as component replacement, movement, and erasure, but also allows for comprehensive adjustments based on multiple images simultaneously, fully meeting the complex needs of actual design scenarios. Furthermore, this method offers high flexibility in its interaction methods, supporting multiple mask generation paths and possessing automated input format and resolution adaptation capabilities, significantly reducing the operational threshold and substantially improving overall processing efficiency.

[0133] The rendering image local editing device provided by the present invention is described below. The rendering image local editing device described below and the rendering image local editing method described above can be referred to in correspondence.

[0134] Based on any of the above embodiments, the present invention provides a rendering image partial editing device. Figure 5 This is a schematic diagram of the structure of the rendering image partial editing device provided by the present invention, as shown below. Figure 5 As shown, the device includes: The acquisition unit 510 is used to acquire the rendering image to be edited, the editing instructions representing the editing intention, and the mask information of the editing intention; The generation unit 520 is used to map the mask information to the latent space of the diffusion model and generate a latent space mask that is aligned with the feature dimensions of the latent space; the latent space mask is used to define the boundary between the editable area and the retained area in the rendering image to be edited at the latent space level. The encoding unit 530 is used to perform image encoding on the rendering image to be edited to obtain potential image features, and determine condition information based on the potential image features and the editing instructions; The input unit 540 is used to input the condition information and the latent space mask into the diffusion model, and the diffusion model performs denoising processing based on the condition information and the latent space mask to generate the target rendering image of the rendering image to be edited.

[0135] The apparatus provided in this invention acquires a rendering image to be edited, editing instructions representing the editing intent, and mask information of the editing intent; maps the mask information to the latent space of a diffusion model to generate a latent space mask aligned with the feature dimensions of the latent space; performs image encoding on the rendering image to be edited to obtain latent image features; determines conditional information based on the latent image features and the editing instructions; and inputs the conditional information and the latent space mask into the diffusion model for denoising processing to generate the target rendering image. This invention generates a latent space mask that can define the boundary between the edited region and the retained region in the rendering image at the latent space level by accurately mapping the mask information of the editing intent to the latent space of the diffusion model. The latent space mask can apply differentiated processing to the edited region and the retained region during the denoising generation process of the diffusion model. The generation of the edited region strictly follows the guidance of the editing instructions, while effectively maintaining the original style and details of the retained region. This overcomes the technical limitations of general editing models in accurately controlling the editing boundary, avoiding problems such as seams, color jumps, or style abrupt changes at the mask edges, ultimately ensuring the overall visual consistency of the target rendering image and improving the user's viewing experience.

[0136] Based on any of the above embodiments, the diffusion model is obtained by iteratively performing the following training steps until a preset iteration cutoff condition is met: Obtain training samples; the training samples include the sample rendering image to be edited, sample editing instructions, sample mask information, and the real rendering image of the sample rendering image; Image encoding is performed on the sample rendering image to be edited to obtain sample latent image features. Sample condition information is constructed based on the sample latent image features and the sample editing instructions. The sample mask information is mapped to the initial latent space of the initial diffusion model to generate a sample latent space mask aligned with the feature dimensions of the initial latent space. The sample condition information and the sample latent space mask are input into the initial diffusion model, and the initial diffusion model outputs the prediction noise. Based on the difference between the predicted noise and the actual noise of the actual rendered image, an element-level loss map is determined; Based on the sample latent space mask, a first weight within the edit region of the element-level loss map and a second weight within the retain region of the element-level loss map are determined; the first weight is higher than the second weight. Based on the first weight and the second weight, a weight mask is determined. Based on the weight mask and the element-level loss map, a target loss is determined. Based on the target loss, the model parameters of the initial diffusion model are updated.

[0137] Based on any of the above embodiments, the generation unit 520 is specifically used for: The mask information is downsampled to obtain an intermediate state mask, and the intermediate state mask is divided into multiple mask patches; the size of the mask patches is the same as the size of the feature patches operated by the attention processing unit of the diffusion model. If there are pixel values ​​belonging to the editing area within the mask block, the state value of the spatial position of the mask block in the potential space is marked as edited; if there are no pixel values ​​belonging to the editing area within the mask block, the state value of the spatial position of the mask block in the potential space is marked as reserved. The latent space mask is generated based on the state values ​​of the spatial locations of all the mask tiles in the latent space.

[0138] Based on any of the above embodiments, the encoding unit 530 specifically includes: A main control channel feature unit is determined to identify the potential image features as main control channel features; A feature unit for the auxiliary control channel is determined, which is used to acquire at least one auxiliary control image, and the auxiliary control image is image encoded to obtain the auxiliary control channel features; The feature fusion unit is used to perform feature fusion based on the main control channel features and the auxiliary control channel features to obtain multi-source fused features; The condition information determination unit is used to determine the condition information based on the multi-source fusion features and the text embedding features corresponding to the editing instructions.

[0139] Based on any of the above embodiments, the unit for determining the auxiliary control channel features is specifically used for: If it is determined that the image size of the auxiliary control image is inconsistent with the image size of the rendering image to be edited, the image size of the auxiliary control image is adjusted based on the image size of the rendering image to be edited. The adjusted auxiliary control image is image encoded to obtain the auxiliary latent features; If the auxiliary control image and the image to be edited are of the same size, the auxiliary control image is image encoded to obtain the auxiliary latent features.

[0140] Based on any of the above embodiments, the auxiliary control image includes a first auxiliary control image and a second auxiliary control image; The first auxiliary control image is used to guide the target shape and structure of the editing area; The second auxiliary control image is used to guide the color and texture style of the editing area; The feature fusion unit is specifically used for: The latent image features, the first auxiliary latent features corresponding to the first auxiliary control image, and the second auxiliary latent features corresponding to the second auxiliary control image are concatenated along the channel dimension to obtain the multi-source fusion features.

[0141] Based on any of the above embodiments, a mask information determination unit is further included, wherein the mask information determination unit is specifically used for: Receive normalized bounding box coordinates input by the user, map the normalized bounding box coordinates to pixel coordinates, and obtain the first mask information; Receive the free-form outline drawn by the user, fill the internal area of ​​the free-form outline, and obtain the second mask information; Identify the component regions in the rendering image to be edited, and generate third mask information based on the component regions; The user's natural language description is parsed, the image region corresponding to the natural language description is located, and a fourth mask information is generated based on the image region; The mask information is determined based on at least one of the first mask information, the second mask information, the third mask information, and the fourth mask information.

[0142] Based on any of the above embodiments, the acquisition unit 510 is specifically used for: Obtain the original rendered image to be edited, and determine the aspect ratio of the original rendered image to be edited; An initial resolution is determined based on a preset target area value and the aspect ratio value. The initial resolution is then normalized to obtain a normalized resolution. The width and height values ​​of the normalized resolution are both integer multiples of the preset values. The resolution of the original rendering image to be edited is adjusted to the normalized resolution to obtain the rendering image to be edited.

[0143] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a local editing method for a rendered image. This method includes: acquiring a rendered image to be edited, editing instructions representing the editing intent, and mask information of the editing intent; mapping the mask information to the latent space of a diffusion model to generate a latent space mask aligned with the feature dimensions of the latent space; the latent space mask is used to define the boundary between the edited region and the preserved region in the rendered image to be edited at the latent space level; performing image encoding on the rendered image to be edited to obtain latent image features; determining condition information based on the latent image features and the editing instructions; inputting the condition information and the latent space mask into the diffusion model, whereby the diffusion model performs denoising processing based on the condition information and the latent space mask to generate a target rendered image of the rendered image to be edited.

[0144] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the rendering image local editing method provided by the above methods. The method includes: acquiring a rendering image to be edited, editing instructions representing editing intentions, and mask information of the editing intentions; mapping the mask information to the latent space of a diffusion model to generate a latent space mask aligned with the feature dimensions of the latent space; the latent space mask is used to define the boundary between the editing area and the reserved area in the rendering image to be edited at the latent space level; performing image encoding on the rendering image to be edited to obtain latent image features; determining condition information based on the latent image features and the editing instructions; inputting the condition information and the latent space mask into the diffusion model, and having the diffusion model perform denoising processing based on the condition information and the latent space mask to generate a target rendering image of the rendering image to be edited.

[0146] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for local editing of a rendered image provided by the methods described above. This method includes: acquiring a rendered image to be edited, editing instructions representing the editing intent, and mask information of the editing intent; mapping the mask information to the latent space of a diffusion model to generate a latent space mask aligned with the feature dimensions of the latent space; the latent space mask being used to define the boundary between the edited region and the retained region in the rendered image to be edited at the latent space level; performing image encoding on the rendered image to be edited to obtain latent image features; determining conditional information based on the latent image features and the editing instructions; inputting the conditional information and the latent space mask into the diffusion model, whereby the diffusion model performs denoising processing based on the conditional information and the latent space mask to generate a target rendered image of the rendered image to be edited.

[0147] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for rendering map local editing, characterized in that, include: Obtain the rendered image to be edited, the editing instructions representing the editing intent, and the mask information of the editing intent; The mask information is mapped to the latent space of the diffusion model to generate a latent space mask that is aligned with the feature dimensions of the latent space. The latent space mask is used to define the boundary between the editable area and the reserved area in the rendering image to be edited at the latent space level; The image to be edited is encoded to obtain potential image features. Based on the potential image features and the editing instructions, condition information is determined. The conditional information and the latent space mask are input into the diffusion model, which then performs denoising based on the conditional information and the latent space mask to generate the target rendering image of the image to be edited.

2. The method for local editing of a rendered image according to claim 1, characterized in that, The diffusion model is obtained by iteratively executing the following training steps until a preset iteration cutoff condition is met: Obtain training samples; the training samples include the sample rendering image to be edited, sample editing instructions, sample mask information, and the real rendering image of the sample rendering image; Image encoding is performed on the sample rendering image to be edited to obtain sample latent image features. Sample condition information is constructed based on the sample latent image features and the sample editing instructions. The sample mask information is mapped to the initial latent space of the initial diffusion model to generate a sample latent space mask aligned with the feature dimensions of the initial latent space. The sample condition information and the sample latent space mask are input into the initial diffusion model, and the initial diffusion model outputs the prediction noise. Based on the difference between the predicted noise and the actual noise of the actual rendered image, an element-level loss map is determined; Based on the sample latent space mask, a first weight within the edit region of the element-level loss map and a second weight within the retention region of the element-level loss map are determined. The first weight is higher than the second weight; Based on the first weight and the second weight, a weight mask is determined. Based on the weight mask and the element-level loss map, a target loss is determined. Based on the target loss, the model parameters of the initial diffusion model are updated.

3. The method for local editing of a rendered image according to claim 1, characterized in that, The step of mapping the mask information to the latent space of the diffusion model to generate a latent space mask aligned with the feature dimensions of the latent space includes: The mask information is downsampled to obtain an intermediate state mask, and the intermediate state mask is divided into multiple mask patches; the size of the mask patches is the same as the size of the feature patches operated by the attention processing unit of the diffusion model. If there are pixel values ​​belonging to the editing area within the mask block, the state value of the spatial position of the mask block in the potential space is marked as edited; if there are no pixel values ​​belonging to the editing area within the mask block, the state value of the spatial position of the mask block in the potential space is marked as reserved. The latent space mask is generated based on the state values ​​of the spatial locations of all the mask tiles in the latent space.

4. The method for partial editing of a rendered image according to any one of claims 1 to 3, characterized in that, The determination of condition information based on the potential image features and the editing instructions includes: The potential image features are determined as the main control channel features; Acquire at least one auxiliary control image, and perform image encoding on the auxiliary control image to obtain auxiliary control channel features; Based on the main control channel features and the auxiliary control channel features, feature fusion is performed to obtain multi-source fusion features; The condition information is determined based on the multi-source fusion features and the text embedding features corresponding to the editing instructions.

5. The method for local editing of a rendered image according to claim 4, characterized in that, The step of encoding the auxiliary control image to obtain auxiliary latent features includes: If it is determined that the image size of the auxiliary control image is inconsistent with the image size of the rendering image to be edited, the image size of the auxiliary control image is adjusted based on the image size of the rendering image to be edited. The adjusted auxiliary control image is image encoded to obtain the auxiliary latent features; If the auxiliary control image and the image to be edited are of the same size, the auxiliary control image is image encoded to obtain the auxiliary latent features.

6. The method for local editing of a rendered image according to claim 4, characterized in that, The auxiliary control image includes a first auxiliary control image and a second auxiliary control image; The first auxiliary control image is used to guide the target shape and structure of the editing area; The second auxiliary control image is used to guide the color and texture style of the editing area; The feature fusion based on the main control channel features and the auxiliary control channel features to obtain multi-source fused features includes: The latent image features, the first auxiliary latent features corresponding to the first auxiliary control image, and the second auxiliary latent features corresponding to the second auxiliary control image are concatenated along the channel dimension to obtain the multi-source fusion features.

7. The method for partial editing of a rendered image according to any one of claims 1 to 3, characterized in that, The steps for determining the mask information include: Receive normalized bounding box coordinates input by the user, map the normalized bounding box coordinates to pixel coordinates, and obtain the first mask information; Receive the free-form outline drawn by the user, fill the internal area of ​​the free-form outline, and obtain the second mask information; Identify the component regions in the rendering image to be edited, and generate third mask information based on the component regions; The user's natural language description is parsed, the image region corresponding to the natural language description is located, and a fourth mask information is generated based on the image region; The mask information is determined based on at least one of the first mask information, the second mask information, the third mask information, and the fourth mask information.

8. The method for partial editing of a rendered image according to any one of claims 1 to 3, characterized in that, The process of obtaining the rendered image to be edited includes: Obtain the original rendered image to be edited, and determine the aspect ratio of the original rendered image to be edited; An initial resolution is determined based on a preset target area value and the aspect ratio value. The initial resolution is then normalized to obtain a normalized resolution. The width and height values ​​of the normalized resolution are both integer multiples of the preset values. The resolution of the original rendering image to be edited is adjusted to the normalized resolution to obtain the rendering image to be edited.

9. A device for partial editing of a rendered image, characterized in that, include: The acquisition unit is used to acquire the rendering image to be edited, the editing instructions representing the editing intention, and the mask information of the editing intention; The generation unit is used to map the mask information to the latent space of the diffusion model and generate a latent space mask that is aligned with the feature dimensions of the latent space. The latent space mask is used to define the boundary between the editable area and the reserved area in the rendering image to be edited at the latent space level; The encoding unit is used to encode the rendered image to be edited to obtain potential image features, and to determine condition information based on the potential image features and the editing instructions; The input unit is used to input the condition information and the latent space mask into the diffusion model, and the diffusion model performs denoising processing based on the condition information and the latent space mask to generate the target rendering image of the rendering image to be edited.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the rendering map local editing method as described in any one of claims 1 to 8.

Images