Method and apparatus for rendering map update based on local editing of line drawing

Abstract

This invention provides a method and apparatus for updating a rendered image based on local line art editing. The method includes: acquiring an initial rendered image and a corresponding initial line art; generating an edited line art in response to a user's editing operation on the initial line art; generating an editing mask for the modified area of ​​the initial rendered image based on the initial line art and the edited line art, wherein the modified area corresponds to the editing operation; and inputting the initial rendered image, the edited line art, the editing mask, and instruction data into a fusion editing model to obtain an updated rendered image output by the fusion editing model. The fusion editing model is obtained through reinforcement training based on initial rendered image samples, edited line art samples, editing mask samples, instruction data samples, and a multi-dimensional reward function. The method and apparatus for updating a rendered image based on local line art editing provided by this invention, by fusing line art editing, mask constraints, and multi-dimensional rewards, can achieve efficient closed-loop updating of the rendered image, ensuring local consistency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for updating rendered images based on partial editing of line drawings. Background Technology

[0002] Industrial product design is a highly iterative and complex process, typically involving multiple rounds of modification and rendering. In traditional workflows, modifying a rendered design requires adjusting the model, repairing the mesh, and re-rendering in CAD software, involving switching between multiple software programs and data conversion, resulting in a long cycle and low efficiency.

[0003] In recent years, generative artificial intelligence has demonstrated powerful image generation capabilities, potentially accelerating design. However, existing technologies treat generation and editing as independent tasks, resulting in a disconnect between the two. When existing methods modify line art locally, the redrawn areas often differ from the original image in lighting and style. Furthermore, the evaluation of generated results heavily relies on subjective human judgment, making it difficult for design iterations to meet the high fidelity and consistency requirements of industrial-grade designs. Therefore, there is an urgent need for a rendering update method that can achieve efficient closed-loop updates while ensuring local consistency. Summary of the Invention

[0004] This invention provides a method and apparatus for updating rendered images based on local editing of line art, which solves the defects of existing rendered image updating methods that are difficult to achieve efficient closed-loop updates and difficult to guarantee local consistency.

[0005] This invention provides a method for updating a rendered image based on partial editing of line art, comprising: Obtain the initial rendered image and the corresponding initial line art, and generate the edited line art in response to the user's editing operation on the initial line art; Based on the initial line drawing and the edited line drawing, an edit mask is generated for the modified area of ​​the initial rendered image, and the modified area corresponds to the edit operation; The initial rendered image, the edited line art, the edit mask, and the instruction data are input into the fusion editing model to obtain the updated rendered image output by the fusion editing model. The fusion editing model is obtained by reinforcement training based on initial rendered image samples, edited line art samples, edit mask samples, and instruction data samples, as well as a multi-dimensional reward function.

[0006] In some embodiments, the fusion editing model includes: A feature extraction layer is used to extract latent image features from the initial rendered image and extract instruction semantic features from the instruction data; A control branch layer is used to extract the structural constraint features of the edited line art; A mask processing layer is used to generate edit weights based on the edit mask. The edit weights include: a first weight of the modified area and a second weight of the non-modified area of ​​the initial rendered image, wherein the first weight is greater than the second weight. The fusion generation layer is used to fuse the latent features of the image, the semantic features of the instructions, and the structural constraint features to obtain fused features, and to generate an updated rendering image based on the fused features and the editing weights.

[0007] In some embodiments, generating the updated rendering image based on the fusion features and the editing weights includes: Based on the fusion features and the first weight, the modified region is updated to obtain the updated modified region; Based on the fusion features and the second weight, the unmodified region is updated to obtain the updated unmodified region; Based on the updated modified area and the updated unmodified area, the updated rendering image is reconstructed and generated.

[0008] In some embodiments, fusing the image latent features, instruction semantic features, and structural constraint features to obtain fused features includes: Determine the weights of the latent features of the image, the semantic features of the instruction, and the structural constraint features; Based on the weights of the latent image features, the semantic instruction features, and the structural constraint features, the latent image features, the semantic instruction features, and the structural constraint features are fused to obtain the fused features.

[0009] In some embodiments, the initial rendering image includes a layout image from multiple perspectives; Correspondingly, the step of generating an edit mask for the modified area of ​​the initial rendered image based on the initial line drawing and the edited line drawing includes: Identify and segment the various view regions in the layout diagram; Based on the initial line art and the edited line art, a target view area is determined from all the view areas, a modification area is determined from the target view area, and an editing mask for the modification area is generated.

[0010] In some embodiments, the instruction data includes style selection instruction data; the fusion editing model further includes a style adaptation layer, used to assign a specific design style to the updated rendering image based on the style selection instruction data during the generation of the updated rendering image.

[0011] In some embodiments, the multi-dimensional reward function is constructed based on multiple reward metrics, including: text rendering accuracy, aesthetic quality, text-image alignment, and target structural integrity.

[0012] In some embodiments, the fusion editing model is trained based on the following steps: Obtain the initial rendered image sample and the corresponding initial line art sample, and obtain the instruction data sample; generate the edited line art sample based on the initial line art sample. Based on the initial line drawing sample and the edited line drawing sample, an edit mask sample for the modified area of ​​the initial rendered image sample is generated; The initial rendered image sample, the edited line drawing sample, the edited mask sample, and the instruction data sample are input into the initial fusion editing model to obtain the prediction result of the updated rendered image output by the initial fusion editing model; Based on the multi-dimensional reward function, the reward value of the prediction result of the updated rendered image is calculated, and the parameters of the initial fusion editing model are iteratively updated based on the reward value to obtain the fusion editing model.

[0013] In some embodiments, the instruction data sample includes multiple prompt word samples, and the training process of the fusion editing model includes: Maintain a reward history buffer for each prompt word sample, and record the reward score of the prediction result corresponding to each prompt word sample; Calculate the mean and variance of the reward score for each cue word sample; If the mean of the reward score of any prompt word sample is greater than a first preset threshold and the variance is less than a second preset threshold, then the prompt word sample is determined to be an abnormal prompt word sample. During the iterative update of the parameters of the initial fusion editing model, the instruction data samples corresponding to the abnormal prompt word samples are filtered.

[0014] The present invention also provides a rendering update device based on partial editing of line art, comprising: The line art editing unit is used to obtain the initial rendering image and the corresponding initial line art, and in response to the user's editing operation on the initial line art, generate the edited line art; A mask generation unit is used to generate an editing mask for the modified area of ​​the initial rendering image based on the initial line drawing and the edited line drawing, wherein the modified area corresponds to the editing operation; The rendering update unit is used to input the initial rendering image, the edited line drawing, the editing mask and instruction data into the fusion editing model to obtain the updated rendering image output by the fusion editing model; The fusion editing model is obtained by reinforcement training based on initial rendered image samples, edited line art samples, edit mask samples, and instruction data samples, as well as a multi-dimensional reward function.

[0015] The present invention provides a rendering image update method and apparatus based on local line art editing. It acquires an initial rendering image and corresponding initial line art, and generates an edited line art in response to user editing operations on the initial line art. Based on the initial and edited line art, it generates an editing mask for the modified areas of the initial rendering image, with the modified areas corresponding to the editing operations. The initial rendering image, edited line art, editing mask, and instruction data are input into a fusion editing model to obtain the updated rendering image output by the fusion editing model. The fusion editing model is obtained through reinforcement training based on initial rendering image samples, edited line art samples, editing mask samples, and instruction data samples, as well as a multi-dimensional reward function. This enables efficient closed-loop updating of the rendering image, ensuring local consistency. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the rendering update method based on partial editing of line art provided in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the process for generating an edit mask for the modification area of ​​the initial rendering image, provided in an embodiment of the present invention.

[0019] Figure 3 This is a flowchart illustrating the training process of the fusion editing model provided in this embodiment of the invention.

[0020] Figure 4 This is a schematic diagram of the structure of the rendering update device based on partial editing of line drawings provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] Figure 1 This is a flowchart illustrating a rendering update method based on partial line art editing provided in an embodiment of the present invention. Figure 1 As shown, a method for updating a rendered image based on partial editing of line art is provided, including the following steps: step 110, step 120, and step 130. This method's steps are merely one possible implementation of the present invention.

[0023] Step 110: Obtain the initial rendering image and the corresponding initial line art. In response to the user's editing operation on the initial line art, generate the edited line art.

[0024] The initial rendering refers to the industrial design rendering to be modified or iterated. It can be a high-quality image rendered from a 3D model or an image generated from the previous design iteration. The initial rendering typically includes the product's materials, lighting, color, and specific shape details. The initial line drawing refers to a line image that strictly corresponds to the initial rendering in structure and outline, used to represent the product's geometric features and edge information. Initial line drawings can be obtained in various ways; for example, they can be extracted from the initial rendering through edge detection algorithms, or they can be line drawings directly uploaded by the user that match the initial rendering, or intermediate line drawing files associated with the initial rendering saved in the design system.

[0025] It should be noted that users can edit the initial line art using drawing tools through the human-computer interaction interface. These drawing tools can include brushes, erasers, shape tools, drag-and-drop transformation tools, etc. Specific forms of editing include, but are not limited to: using the brush tool to draw new lines to add design elements, such as adding a car waistline or modifying the headlight outline; using the eraser tool to erase unwanted lines to remove redundant features; or performing geometric transformations on existing lines, such as scaling, rotating, and translating. The system responds to these user operations in real time, updating the line art status on the canvas. Once the user completes the editing and confirms, the edited line art is generated. The edited line art retains the original structure of the unmodified parts while also incorporating the user's newly defined design intent.

[0026] Step 120: Based on the initial line art and the edited line art, generate an edit mask for the modification area of ​​the initial rendering image. The modification area corresponds to the editing operation.

[0027] It's important to note that the purpose of generating an edit mask is to accurately locate areas that have changed, allowing only those areas to be redrawn during subsequent generation, while protecting other areas from being affected. The process of generating an edit mask based on the initial and edited line art is essentially a difference detection and region localization process. Specifically, it compares the differences between the initial and edited line art at the pixel level. Any area where pixel values ​​have changed is identified as the modified area.

[0028] Furthermore, to ensure continuity and natural transitions in the generated data, morphological processing is typically performed after identifying the differing pixels. For example, dilation operations are used to appropriately expand the differing regions outwards to cover the potential influence around the modified lines, thus forming the final edit mask. Alternatively, opening operations are used to remove isolated noise, and closing operations are used to fill small holes. The edit mask is usually a binary or grayscale image, where white areas or areas with specific numerical values ​​represent the modified areas, and black areas or areas with another specific numerical value represent the unmodified areas. This edit mask is spatially aligned strictly with the initial render and the line art, ensuring a correspondence between the editing operations and the final image update.

[0029] Step 130: Input the initial rendering image, the edited line art, the editing mask, and the instruction data into the fusion editing model to obtain the updated rendering image output by the fusion editing model.

[0030] The fusion editing model is obtained through reinforcement training based on initial rendered image samples, edited line art samples, edit mask samples, and instruction data samples, as well as a multi-dimensional reward function.

[0031] Instruction data refers to text descriptions or parameter settings used to guide the generation process, which contain semantic definitions of the user's design style, material properties, or modification intentions. For example, changing the headlights to a slimmer shape.

[0032] The fusion editing model is a specially trained deep learning generative model with multimodal input processing capabilities. This model can simultaneously receive visual image information, structural constraint information, region control information, and semantic control information. During inference, the fusion editing model uses the edited line art as a strong structural constraint to ensure that the generated object shape conforms to the user's modification intention; it uses an editing mask as an attention control mechanism, instructing the model to only perform denoising generation or redrawing within the modified areas marked by the mask, while preserving the non-modified areas marked by the mask using information from the initial render, thus achieving pixel-level consistency; and it uses instruction data to adjust the style and semantic attributes of the generated content. Finally, the model outputs an updated render image, which exhibits new design features in the modified areas, maintains a high degree of consistency with the initial render image in the non-modified areas, and has naturally blended overall lighting and materials.

[0033] In this embodiment of the invention, to address the problems of traditional generative models' difficulty in objectively evaluating design quality and the tendency for local inconsistencies, a reinforcement learning mechanism is introduced into the training of this fusion editing model. Training data includes paired or grouped sample data, namely, initial rendered image samples, corresponding line art samples that have been manually or automatically edited, corresponding edit mask samples, and corresponding instruction data samples. During training, the predicted images generated by the model are evaluated using a multi-dimensional reward function.

[0034] In some embodiments, the multi-dimensional reward function is constructed based on multiple reward metrics, including: text rendering accuracy, aesthetic quality, text-image alignment, and target structural integrity.

[0035] The multi-dimensional reward function is a composite function that integrates multiple evaluation dimensions. These dimensions aim to quantitatively evaluate the quality of the generated results in different aspects, such as the semantic consistency between the generated image and the instruction text, the visual aesthetic quality of the image, the accuracy of line structure reproduction, and the clarity of text or specific symbols. By maximizing the expected value of this multi-dimensional reward function, the model can learn generation strategies that meet aesthetic standards and engineering requirements, thereby producing higher-quality and more expected design solutions in practical applications.

[0036] In this embodiment of the invention, by constructing an end-to-end closed loop from line drawing editing to mask generation and then to model generation, rapid iteration of industrial design solutions can be achieved. Users only need to modify the line drawing to obtain a rendering with realistic lighting and rich details, greatly shortening the design cycle. At the same time, the mask mechanism ensures the consistency of the overall image when making local modifications, and the model optimized by a multi-dimensional reward function ensures the objective quality and usability of the generated results, effectively solving the problems of low efficiency in traditional processes and poor controllability of existing generation tools.

[0037] In some embodiments, the fusion editing model includes: a feature extraction layer, a control branch layer, a mask processing layer, and a fusion generation layer.

[0038] The feature extraction layer is used to extract the latent image features of the initial rendered image and the instruction semantic features of the instruction data.

[0039] Specifically, for the initial rendered image, the feature extraction layer can use a variational autoencoder to compress and map the high-resolution initial rendered image in pixel space to the latent space, obtaining image latent features. Image latent features retain the main semantic and structural information of the initial rendered image, but the data dimensionality is significantly reduced, which is beneficial for subsequent efficient computation. For instruction data, the feature extraction layer can use a pre-trained text encoder to convert natural language instructions into corresponding word embedding vectors, i.e., instruction semantic features. Instruction semantic features capture semantic information such as design intent, style description, and object category in the instruction data.

[0040] Control the branch layer, which is used to extract the structural constraint features of the edited line art.

[0041] The control branch layer aims to extract strong geometric constraint information from the edited line art to guide the contour and structure of the generated image. This layer can employ a neural network structure similar to ControlNet. It receives the edited line art as input and extracts geometric features such as edges, line directions, and closed regions from the edited line art layer by layer through a series of convolutional layers, residual blocks, or attention modules. These features are encoded as structural constraint features. This layer ensures that the generated result strictly adheres to the user's line design.

[0042] The mask processing layer is used to generate edit weights based on the edit mask. The edit weights include: a first weight for the modified area and a second weight for the non-modified area of ​​the initial rendered image, with the first weight being greater than the second weight.

[0043] The masking layer transforms the edit mask into attention or weight adjustment parameters during model generation. The edit mask is first downsampled or mapped to the same resolution as the latent space features. Then, this layer generates a corresponding edit weight matrix based on the numerical distribution of the edit mask. In this matrix, locations corresponding to modified regions are assigned a first weight, and locations corresponding to unmodified regions are assigned a second weight.

[0044] The fusion generation layer is used to fuse latent image features, instruction semantic features, and structural constraint features to obtain fused features. Based on the fused features and editing weights, an updated rendering image is generated.

[0045] The fusion generation layer is the core generation module of the model, typically employing a diffusion model based on the Transformer or U-Net architecture. It receives latent image features and instruction semantic features from the feature extraction layer, as well as structural constraint features from the control branch layer. These multi-source features undergo deep interaction and fusion within the network to form fused features.

[0046] Through the collaborative work of the above layers, the fusion editing model in this embodiment can simultaneously take into account semantic understanding, structural fidelity and local controllability, so that the generated result not only conforms to the user's modified line structure, but also has high-quality rendering details and seamlessly integrates into the original background.

[0047] In some embodiments, generating an updated rendered image based on fusion features and edit weights includes: Based on the fusion features and the first weight, the modified region is updated to obtain the updated modified region. Based on the fusion features and the second weight, the unmodified region is updated to obtain the updated unmodified region. Based on the updated modified area and the updated unmodified area, the updated rendering map is reconstructed.

[0048] During image generation, the model spatially modulates feature updates using the aforementioned edit weights. Specifically, based on the edit weights, the model primarily updates modified regions using new noise predictions calculated from fused features, while in non-modified regions, it mainly uses latent features from the initial rendered image for backfilling or preservation. Finally, the latent features, after multiple denoising iterations, are fed into the decoder to reconstruct and generate the final updated rendered image.

[0049] To achieve the effect of local redrawing and global preservation, the first weight is set to be significantly greater than the second weight. For example, the first weight can be close to 1, while the second weight can be close to 0 or a small value. In some more refined implementations, the mask processing layer can also soften the edges of the edit mask, creating a smooth transition area between the first and second weights to prevent obvious mosaic artifacts in the generated image.

[0050] In some embodiments, latent image features, instruction semantic features, and structural constraint features are fused to obtain fused features, including: Determine the weights of latent image features, instruction semantic features, and structural constraint features; Based on the weights of latent image features, instruction semantic features, and structural constraint features, the latent image features, instruction semantic features, and structural constraint features are fused to obtain fused features.

[0051] The weights of latent image features, instruction semantic features, and structural constraint features can be determined in various ways. For example, these weights can be preset fixed hyperparameters, dynamically adjusted variables based on the current generation step, or control parameters in response to user input.

[0052] Specifically, the weights of the instruction semantic features typically correspond to the proportion coefficients of the unclassifier-guided model, used to control the degree to which the generated image fits the text description; the weights of the structural constraint features correspond to the control strength of ControlNet, and users can adjust this parameter to determine the degree to which the generated result conforms to the line drawing; the weights of the image latent features are related to the denoising strength, determining how much information from the original image is retained. An optimal weight combination can be automatically recommended based on the task type; for example, assigning higher weights to image latent features when making fine adjustments, and higher weights to structural constraint features when making significant shape changes.

[0053] In this embodiment of the invention, fine-grained control over the generation process is achieved by explicitly determining and applying the weights of each feature. This allows the model to find the optimal balance between maintaining structural accuracy, stylistic consistency, and visual coherence, avoiding image degradation due to an overly strong single condition or generation failure due to conflicting conditions, thereby significantly improving the quality and usability of the final rendered image.

[0054] Figure 2 This is a schematic diagram illustrating the process of generating an edit mask for the modification area of ​​the initial rendered image, as provided in an embodiment of the present invention. Figure 2 As shown, in some embodiments, the initial rendering image includes layout images from multiple perspectives; correspondingly, step 120 generates an edit mask for the modified areas of the initial rendering image based on the initial line art and the edited line art, including: Step 121: Identify and segment the various view regions in the layout diagram.

[0055] The initial rendering image can be a combination layout image containing different perspectives of the same product object, such as front view, side view, rear view, top view, etc.

[0056] Identifying and segmenting the various viewpoint regions in the layout image aims to define the boundaries of each viewpoint, thereby enabling independent processing or collaborative updates of specific viewpoints. This process typically employs image content-based segmentation algorithms.

[0057] Specifically, background color estimation and content density analysis techniques can be used. First, pixels in the image edge regions are statistically analyzed to robustly estimate the background color. Then, based on the background color, the color difference distance of all pixels in the image is calculated to identify pixels belonging to the product foreground. Next, by statistically analyzing the foreground pixel density in the horizontal and vertical directions of the image, i.e., the projected contour, a content density curve is drawn. After smoothing this curve, density troughs or zero-value intervals are identified; these intervals correspond to the gaps between viewpoints. Based on the location of these gaps, bounding boxes surrounding the content of each viewpoint are generated, thereby accurately segmenting each viewpoint region from the overall layout image.

[0058] Step 122: Based on the initial line art and the edited line art, determine the target view area from all view areas, determine the modification area from the target view area, and generate the editing mask of the modification area.

[0059] Determining the target viewpoint region from all viewpoint regions refers to identifying the specific viewpoint corresponding to the editing operation. This can be achieved by detecting the spatial coordinates of the editing operation. When the user's stroke is detected to fall within the boundary of a segmented viewpoint region, that region is marked as the target viewpoint region.

[0060] After identifying the target viewpoint region, pixel-level difference detection is performed within that viewpoint. The initial and edited line art within the target viewpoint region are compared to extract the differing pixels, generating a local difference mask. This local mask is then mapped back to the coordinate system of the overall layout image, or a mask of a specific size is generated only for the target viewpoint region. In some embodiments, if multi-view collaboration is required, the modified regions in other related viewpoint regions can be inferred based on the modifications made to the target viewpoint region using 3D projection relationships or feature matching techniques. Editing masks for these related viewpoint regions are then generated synchronously, allowing multiple viewpoint images to be updated simultaneously in a single generation process.

[0061] In this embodiment of the invention, the problem of traditional image generation models struggling to handle complex layout images is solved by automatically identifying and segmenting multi-view layout images. This allows designers to directly make local modifications to a complete proposal image containing multiple perspectives without first cropping the image into individual pieces, making changes, and then stitching them together. Simultaneously, precise region positioning ensures that modifications do not interfere with the image content of adjacent perspectives, improving the consistency and editing efficiency of multi-view design schemes.

[0062] In some embodiments, the instruction data includes style selection instruction data; the fusion editing model also includes a style adaptation layer, which is used to assign a specific design style to the updated rendering image based on the style selection instruction data during the generation of the updated rendering image.

[0063] Style selection instruction data refers to control signals issued by the user explicitly specifying the desired design style. These signals can be specific options selected by the user in the interface menu, such as minimalist, streamlined, or atmospheric, or text prompts entered by the user containing specific style descriptive words.

[0064] Optionally, style tags can be automatically injected based on a prompt template by filling in parameters. Complete style prompts can be automatically generated by combining parameters such as base template, shape selection, complexity selection, surface selection, rendering style, and LoRA selection.

[0065] The style adaptation layer is a pluggable module in the fusion editing model used to dynamically adjust the generated style. Specifically, this layer typically employs Low-Rank Adaptation (LoRA) technology. LoRA achieves efficient fine-tuning of the model's style with a minimal number of parameters by injecting a low-rank decomposition matrix into the key weight matrix of the pre-trained model. Multiple LoRA adapter weight files corresponding to different design styles are pre-trained and stored in a style library.

[0066] When style selection instructions are received, the style adaptation layer can dynamically load the corresponding LoRA weights and merge them into the backbone network of the fusion editing model based on the instructions. This process does not require reloading the entire large main model, so the switching can be completed in milliseconds. During the generation process, the style adaptation layer loaded with specific LoRA weights can guide the direction of feature transformation in the model, so that the generated rendered image, while maintaining the constraints of the line art structure, presents the specific visual style required by the instructions, such as specific color tendencies, lighting textures, and line curvature characteristics.

[0067] In this embodiment of the invention, a style adaptation layer is introduced to enable flexible and instant switching of design styles. This allows designers to quickly preview the visual appearance of the same design scheme under different market styles without altering the geometric shape, greatly enhancing the adaptability and decision-making efficiency of the design scheme. It also solves the pain point of traditional methods where style transfer requires retraining the model or cumbersome parameter adjustments.

[0068] Figure 3 This is a flowchart illustrating the training process of the fusion editing model provided in an embodiment of the present invention. Figure 3 As shown, in some embodiments, the fusion editing model is trained based on the following steps: Step 310: Obtain the initial rendering image sample and the corresponding initial line art sample, and obtain the instruction data sample. Generate the edited line art sample based on the initial line art sample.

[0069] The methods for obtaining the initial rendered image samples and initial line art samples are similar to those in step 110, but they are used here as training data and are more extensive and diverse. Methods for obtaining initial line art samples typically include extraction from rendered image samples or manual annotation. The initial line art samples should have a high degree of spatial correspondence with the initial rendered image samples. Obtaining instruction data samples refers to collecting text descriptions containing various design intentions, such as changing the headlights to round shapes or adding a dynamic waistline.

[0070] This method generates edited line art samples based on initial line art samples, aiming to simulate real user modifications to line art. This generation can be based on rule-based random perturbations, such as randomly erasing lines or adding random shapes, or it can utilize specialized line art editing models to generate modifications with specific semantics, such as changing straight lines to curves.

[0071] Step 320: Based on the initial line art sample and the edited line art sample, generate an edit mask sample for the modified area of ​​the initial rendered image sample.

[0072] In this step, the process of generating the edit mask sample is consistent with the principle of mask generation in step 120, that is, it is generated by comparing the difference areas between the initial line art sample and the edited line art sample. The mask sample is used to train the model to identify and focus on the modified areas, and learn to generate only the mask-covered areas that conform to the line art constraints while keeping the non-modified areas unchanged.

[0073] Step 330: Input the initial rendering sample, the edited line drawing sample, the edited mask sample, and the instruction data sample into the initial fusion editing model to obtain the prediction result of the updated rendering output by the initial fusion editing model.

[0074] In this step, the initial fusion editing model refers to a pre-trained diffusion model that has not yet been fine-tuned for a specific task, or a model that has undergone preliminary training but has not yet converged. After inputting the above sample data into the model, the model can generate the predicted updated rendering map based on the input samples.

[0075] Step 340: Based on the multi-dimensional reward function, calculate the reward value of the prediction result of the updated rendered image, and iteratively update the parameters of the initial fusion editing model based on the reward value to obtain the fusion editing model.

[0076] In this step, the multi-dimensional reward function is a comprehensive evaluation index system used to quantitatively assess the quality of the generated results. This function typically includes multiple sub-items, each corresponding to a different design quality dimension, such as structural accuracy, style consistency, and textual relevance. The process of calculating the reward value involves inputting the generated predicted image into various evaluation models for scoring, and then weighted summing to obtain the total score.

[0077] A reinforcement learning algorithm is employed to iteratively update the model parameters based on the reward value. The goal of model training is to maximize this reward value. By continuously adjusting the model parameters, the generated images achieve higher multi-dimensional scores while maintaining structural constraints and consistency in local modifications. This process is repeated until the model converges, ultimately resulting in a fusion editing model with high-quality generation capabilities.

[0078] In this embodiment of the invention, by constructing a closed-loop reinforcement learning training process and optimizing model parameters using a multi-dimensional reward function, the model can learn generation strategies that conform to design standards and engineering constraints, significantly improving the usability and professionalism of the generated results.

[0079] In some embodiments, the instruction data sample includes multiple prompt word samples, and the training process of the fusion editing model includes: Maintain a reward history buffer for each prompt word sample, and record the reward score of the prediction result corresponding to each prompt word sample; Calculate the mean and variance of the reward score for each cue word sample; If the mean of the reward score of any prompt word sample is greater than the first preset threshold and the variance is less than the second preset threshold, then any prompt word sample is determined to be an abnormal prompt word sample. During the iterative update of the parameters of the initial fusion editing model, the instruction data samples corresponding to the abnormal prompt words are filtered.

[0080] Optionally, statistical analysis can be performed on the data in each buffer periodically or in real time. Specifically, the arithmetic mean of all reward scores in the buffer is calculated to reflect the average performance level of the prompt word samples under the current model; at the same time, the variance or standard deviation is calculated to reflect the degree of fluctuation or dispersion of the reward scores corresponding to the prompt word samples.

[0081] Optionally, the first preset threshold is a relatively high value, such as 0.9, and the second preset threshold is a relatively low value, such as 0.1. If a cue word sample consistently receives extremely high scores with minimal fluctuations, this often indicates that the model has found a specific pattern to deceive the reward model, rather than truly learning to generate diverse and high-quality images based on that cue word sample. Alternatively, the cue word sample itself may be problematic, causing the reward function to fail. Therefore, cue word samples that meet this condition are marked as anomalous cue word samples.

[0082] Once a prompt word sample is identified as anomalous, it is filtered out in subsequent training iterations. Specifically, this can be done by skipping training samples containing that prompt word when building the training batch, or by setting the gradient of that training sample to zero when calculating the loss function, preventing it from participating in backpropagation to update model parameters. This filtering mechanism forces the model to explore and learn from samples that haven't yet reached high scores and truly need optimization, thus ensuring the health of the training process and the robustness of the resulting model.

[0083] In this embodiment of the invention, by introducing a dynamic filtering mechanism based on statistical features, abnormal data is automatically identified and removed, ensuring that the multi-dimensional reward function can continuously and effectively guide the model to evolve towards generating truly high-quality design solutions, significantly improving training efficiency and the reliability of the final model.

[0084] The following describes the rendering image updating device based on partial line drawing editing provided by the embodiments of the present invention. The rendering image updating device based on partial line drawing editing described below can be referred to in correspondence with the rendering image updating method based on partial line drawing editing described above.

[0085] Figure 4 This is a schematic diagram of the rendering update device based on partial line drawing editing provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the rendering update device 400 based on partial line drawing editing includes: The line art editing unit 410 is used to obtain the initial rendering image and the corresponding initial line art, and in response to the user's editing operation on the initial line art, to generate the edited line art; The mask generation unit 420 is used to generate an editing mask for the modification area of ​​the initial rendering image based on the initial line drawing and the edited line drawing. The modification area corresponds to the editing operation. The rendering update unit 430 is used to input the initial rendering image, the edited line art, the editing mask and the instruction data into the fusion editing model to obtain the updated rendering image output by the fusion editing model; The fusion editing model is obtained through reinforcement training based on initial rendered image samples, edited line art samples, edit mask samples, and instruction data samples, as well as a multi-dimensional reward function.

[0086] Optionally, the fusion editing model includes: The feature extraction layer is used to extract the latent image features of the initial rendered image and the instruction semantic features of the instruction data; The control branch layer is used to extract the structural constraint features of the edited line art; The mask processing layer is used to generate edit weights based on the edit mask. The edit weights include: a first weight for the modified area and a second weight for the non-modified area of ​​the initial rendered image, with the first weight being greater than the second weight. The fusion generation layer is used to fuse latent image features, instruction semantic features, and structural constraint features to obtain fused features. Based on the fused features and editing weights, an updated rendering image is generated.

[0087] Optionally, an updated rendered image is generated based on the fusion features and edit weights, including: Based on the fusion features and the first weight, the modified region is updated to obtain the updated modified region. Based on the fusion features and the second weight, the unmodified region is updated to obtain the updated unmodified region. Based on the updated modified area and the updated unmodified area, the updated rendering map is reconstructed.

[0088] Optionally, latent image features, instruction semantic features, and structural constraint features are fused to obtain fused features, including: Determine the weights of latent image features, instruction semantic features, and structural constraint features; Based on the weights of latent image features, instruction semantic features, and structural constraint features, the latent image features, instruction semantic features, and structural constraint features are fused to obtain fused features.

[0089] Optionally, the initial render includes layout diagrams from multiple perspectives; Correspondingly, based on the initial line art and the edited line art, an edit mask is generated for the modified areas of the initial rendered image, including: Identify and segment the various viewpoint regions in the layout diagram; Based on the initial and edited line art, the target view area is determined from all view areas, the modification area is determined from the target view area, and an editing mask for the modification area is generated.

[0090] Optionally, the instruction data includes style selection instruction data; the fusion editing model also includes a style adaptation layer, which is used to assign a specific design style to the updated rendering based on the style selection instruction data during the generation of the updated rendering.

[0091] Optionally, the multi-dimensional reward function is constructed based on multiple reward metrics, including: text rendering accuracy, aesthetic quality, text-image alignment, and target structural integrity.

[0092] Optionally, the fusion editing model is trained based on the following steps: Obtain the initial rendered image sample and the corresponding initial line art sample, and obtain the instruction data sample. Generate the edited line art sample based on the initial line art sample. Based on the initial line art sample and the edited line art sample, generate an edit mask sample for the modified area of ​​the initial rendered image sample; The initial rendered image sample, the edited line drawing sample, the edited mask sample, and the instruction data sample are input into the initial fusion editing model to obtain the prediction result of the updated rendered image output by the initial fusion editing model; Based on a multi-dimensional reward function, the reward value of the prediction result of the updated rendered image is calculated. The parameters of the initial fusion editing model are iteratively updated based on the reward value to obtain the fusion editing model.

[0093] Optionally, the instruction data samples include multiple prompt word samples, and the training process of the fusion editing model includes: Maintain a reward history buffer for each prompt word sample, and record the reward score of the prediction result corresponding to each prompt word sample; Calculate the mean and variance of the reward score for each cue word sample; If the mean of the reward score of any prompt word sample is greater than the first preset threshold and the variance is less than the second preset threshold, then any prompt word sample is determined to be an abnormal prompt word sample. During the iterative update of the parameters of the initial fusion editing model, the instruction data samples corresponding to the abnormal prompt words are filtered.

[0094] It should be noted that the rendering image update device based on partial line drawing editing provided in this embodiment of the invention can implement all the method steps implemented in the above-mentioned rendering image update method embodiment based on partial line drawing editing, and can achieve the same technical effect. Here, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail.

[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for updating a rendered image based on partial editing of line art, characterized in that, include: Obtain the initial rendered image and the corresponding initial line art, and generate the edited line art in response to the user's editing operation on the initial line art; Based on the initial line drawing and the edited line drawing, an edit mask is generated for the modified area of ​​the initial rendered image, and the modified area corresponds to the edit operation; The initial rendered image, the edited line art, the edit mask, and the instruction data are input into the fusion editing model to obtain the updated rendered image output by the fusion editing model. The fusion editing model is obtained by reinforcement training based on initial rendered image samples, edited line art samples, edit mask samples, and instruction data samples, as well as a multi-dimensional reward function.

2. The method for updating a rendered image based on partial line art editing according to claim 1, characterized in that, The fusion editing model includes: A feature extraction layer is used to extract the latent image features of the initial rendered image and the instruction semantic features of the instruction data; A control branch layer is used to extract the structural constraint features of the edited line art; A mask processing layer is used to generate edit weights based on the edit mask. The edit weights include: a first weight of the modified area and a second weight of the non-modified area of ​​the initial rendered image, wherein the first weight is greater than the second weight. The fusion generation layer is used to fuse the latent features of the image, the semantic features of the instructions, and the structural constraint features to obtain fused features, and to generate an updated rendering image based on the fused features and the editing weights.

3. The method for updating a rendered image based on partial line art editing according to claim 2, characterized in that, The process of generating the updated rendering image based on the fusion features and the editing weights includes: Based on the fusion features and the first weight, the modified region is updated to obtain the updated modified region; Based on the fusion features and the second weight, the unmodified region is updated to obtain the updated unmodified region; Based on the updated modified area and the updated unmodified area, the updated rendering image is reconstructed and generated.

4. The method for updating a rendered image based on partial line art editing according to claim 2, characterized in that, The fusion of the image latent features, instruction semantic features, and structural constraint features yields fused features, including: Determine the weights of the latent features of the image, the semantic features of the instruction, and the structural constraint features; Based on the weights of the latent image features, the semantic instruction features, and the structural constraint features, the latent image features, the semantic instruction features, and the structural constraint features are fused to obtain the fused features.

5. The method for updating a rendered image based on partial line art editing according to claim 1, characterized in that, The initial rendering includes layout diagrams from multiple perspectives; Correspondingly, the step of generating an edit mask for the modified area of ​​the initial rendered image based on the initial line drawing and the edited line drawing includes: Identify and segment the various view regions in the layout diagram; Based on the initial line art and the edited line art, a target view area is determined from all the view areas, a modification area is determined from the target view area, and an editing mask for the modification area is generated.

6. The method for updating a rendered image based on partial line art editing according to claim 1, characterized in that, The instruction data includes style selection instruction data; the fusion editing model also includes a style adaptation layer, which is used to assign a specific design style to the updated rendering image based on the style selection instruction data during the generation of the updated rendering image.

7. The method for updating a rendered image based on partial line art editing according to claim 1, characterized in that, The multi-dimensional reward function is constructed based on multiple reward metrics, including: text rendering accuracy, aesthetic quality, text-image alignment, and target structural integrity.

8. The method for updating a rendered image based on partial line art editing according to claim 1, characterized in that, The fusion editing model was trained based on the following steps: Obtain the initial rendered image sample and the corresponding initial line art sample, and obtain the instruction data sample; generate the edited line art sample based on the initial line art sample. Based on the initial line drawing sample and the edited line drawing sample, an edit mask sample for the modified area of ​​the initial rendered image sample is generated; The initial rendered image sample, the edited line drawing sample, the edited mask sample, and the instruction data sample are input into the initial fusion editing model to obtain the prediction result of the updated rendered image output by the initial fusion editing model; Based on the multi-dimensional reward function, the reward value of the prediction result of the updated rendered image is calculated, and the parameters of the initial fusion editing model are iteratively updated based on the reward value to obtain the fusion editing model.

9. The method for updating a rendered image based on partial line art editing according to claim 8, characterized in that, The instruction data sample includes multiple prompt word samples, and the training process of the fusion editing model includes: Maintain a reward history buffer for each prompt word sample, and record the reward score of the prediction result corresponding to each prompt word sample; Calculate the mean and variance of the reward score for each cue word sample; If the mean of the reward score of any prompt word sample is greater than a first preset threshold and the variance is less than a second preset threshold, then the prompt word sample is determined to be an abnormal prompt word sample. During the iterative update of the parameters of the initial fusion editing model, the instruction data samples corresponding to the abnormal prompt word samples are filtered.

10. A rendering update device based on partial editing of line art, characterized in that, include: The line art editing unit is used to obtain the initial rendering image and the corresponding initial line art, and in response to the user's editing operation on the initial line art, generate the edited line art; A mask generation unit is used to generate an editing mask for the modified area of ​​the initial rendering image based on the initial line drawing and the edited line drawing, wherein the modified area corresponds to the editing operation; The rendering update unit is used to input the initial rendering image, the edited line drawing, the editing mask and instruction data into the fusion editing model to obtain the updated rendering image output by the fusion editing model; The fusion editing model is obtained by reinforcement training based on initial rendered image samples, edited line art samples, edit mask samples, and instruction data samples, as well as a multi-dimensional reward function.

Images