Picture enhancement method and device in rendering scene, electronic equipment and storage medium

Abstract

The present disclosure provides a picture enhancement method and device in a rendering scene, electronic equipment and storage medium, relates to the technical field of image processing, in particular to the fields of computer vision, artificial intelligence, etc., and can be applied to application scenarios such as rendering picture detail enhancement. The specific implementation scheme is: generating a mask corresponding to the to-be-enhanced object in the target image according to the structured layer information; extracting a target region image from the target image based on the mask; inputting the target region image and preset prompt word information for enhancing image details into a pre-trained image generation model to generate a region enhancement image by the image generation model; and fusing the region enhancement image and the target image based on the mask to obtain a to-be-displayed image. The present scheme can effectively enhance the rendering result, improve the realism and usability of the distorted image, and at the same time reduce resource consumption and processing time, thereby realizing the real-time generation capability of the enhanced image.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, particularly to the fields of computer vision and artificial intelligence, and can be used in application scenarios such as post-rendering image detail enhancement. Specifically, it relates to image enhancement methods, devices, electronic devices, and storage media in rendering scenarios. Background Technology

[0002] In related technologies, real-time rendering is limited by computing resources, making it difficult to generate richly detailed images while maintaining a high frame rate. Existing post-processing enhancement methods are mostly offline processing or uniform filtering of the entire frame, which cannot utilize the structured information such as layers and object identifiers generated internally by the rendering engine to perform targeted detail enhancement on different objects, and it is also difficult to integrate the generated model into the rendering pipeline in real time and achieve seamless fusion. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for image enhancement in a rendering scene.

[0004] According to a first aspect of this disclosure, an image enhancement method for a rendering scene is provided, comprising: generating a mask corresponding to an object to be enhanced in a target image based on structured layer information; the structured layer information is generated by a rendering engine during the rendering of the target image; extracting a target region image from the target image based on the mask; inputting the target region image and preset prompt words for enhancing image details into a pre-trained image generation model, so that the image generation model generates a region-enhanced image; the region-enhanced image is an image after detail enhancement of the object to be enhanced; fusing the region-enhanced image with the target image based on the mask to obtain an image to be displayed; wherein the image to be displayed is generated within a preset time interval after the rendering engine outputs the target image.

[0005] According to a second aspect of this disclosure, an image enhancement device for a rendering scene is provided, comprising: a mask generation module for generating a mask corresponding to an object to be enhanced in a target image based on structured layer information; the structured layer information is generated by a rendering engine during the rendering of the target image; a region extraction module for extracting a target region image from the target image based on the mask; a detail enhancement module for inputting the target region image and preset prompt words for enhancing image details into a pre-trained image generation model, so that the image generation model generates a region enhancement image; the region enhancement image is an image after detail enhancement of the object to be enhanced; and a fusion processing module for fusing the region enhancement image with the target image based on the mask to obtain an image to be displayed; wherein the image to be displayed is generated within a preset time interval after the rendering engine outputs the target image.

[0006] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the methods described in the embodiments of this disclosure.

[0007] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0008] The solution disclosed herein can effectively enhance rendering results, improve the realism and usability of distorted images, and reduce resource consumption and processing time, thereby achieving the ability to generate enhanced images in real time.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating an image enhancement method in a rendering scene according to an embodiment of the present disclosure; Figure 2 This is another schematic flowchart of an image enhancement method in a rendering scene according to an embodiment of the present disclosure; Figure 3 This is a schematic diagram of the structure of an image enhancement device in a rendering scene according to an embodiment of the present disclosure; Figure 4 This is a scene diagram illustrating the image enhancement method in a rendering scene according to an embodiment of the present disclosure; Figure 5 This is a structural diagram of an electronic device used to implement the image enhancement method in the rendering scene of the embodiments of this disclosure. Detailed Implementation

[0011] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0012] In this document, the term "and / or" merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The term "at least one" in this document indicates any combination of at least two of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this document refer to and distinguish between multiple similar technical terms, not to restrict the order or to limit there to only two. For example, "first feature" and "second feature" refer to two categories / two features; the first feature can be one or more, and the second feature can also be one or more.

[0013] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0014] Before introducing the technical solutions of the embodiments of this disclosure, the technical terms that may be used in this disclosure will be further explained: Image rendering refers to the process of using a computer to generate a final visual image based on information such as models, materials, lighting, and textures in a scene.

[0015] When relying on rendering engines for real-time rendering, achieving more natural and realistic results often requires the rendering engine to invest more computational resources in representing object details, significantly increasing rendering time and making it difficult to meet real-time requirements. Conversely, to ensure real-time output, computational load is typically compressed and detail levels reduced, leading to distortion, insufficient texture, and poor aesthetics, resulting in a significant gap between the user's actual visual experience and expectations. Therefore, existing technologies struggle to effectively balance real-time performance and realism. Simply relying on rendering engines for detail enhancement also introduces high time complexity, making it difficult to simultaneously meet the application demands for high realism and high real-time performance.

[0016] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, this disclosure proposes an image enhancement method for rendering scenarios, which can effectively enhance rendering results, improve the realism and usability of distorted images, and reduce resource consumption and processing time, thereby achieving the ability to generate enhanced images in real time.

[0017] This disclosure provides an image enhancement method for rendering scenes. Figure 1This is a flowchart illustrating an image enhancement method in a rendering scene according to an embodiment of the present disclosure. This image enhancement method can be applied to an image enhancement device in a rendering scene. The image enhancement device is located in an electronic device. This electronic device includes, but is not limited to, fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers or ordinary servers. Mobile devices include, but are not limited to, image detail enhancement devices, which can be mobile phones, tablets, etc. In some possible implementations, the image enhancement method in the rendering scene can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the image enhancement methods in this rendering scene include: S101. Generate a mask corresponding to the object to be enhanced in the target image based on the structured layer information.

[0018] S102. Extract the target region image from the target image based on the mask.

[0019] S103. Input the target region image and preset prompt words for enhancing image details into a pre-trained image generation model to generate a region-enhanced image.

[0020] S104. Based on the mask, the region enhancement image and the target image are fused to obtain the image to be displayed.

[0021] Here, structured layer information refers to a data set that can characterize the hierarchical relationships, category attributes, spatial locations, and boundary ranges between different objects, regions, or visual elements in a target image. For example, it includes layer identifiers, contour information, transparency information, depth relationships, or segmentation results corresponding to various objects in a scene. The target image refers to the original image to be enhanced, which can be a scene image output by a rendering engine. The object to be enhanced refers to the target entity or region in the target image that needs enhancement, such as a person's face, animal fur, or food surface texture. A mask refers to region identification information used to mark the location and extent of the object to be enhanced in the target image, where the marked region corresponds to the object to be enhanced, and the unmarked region corresponds to the non-enhanced region.

[0022] In this embodiment, the category identifier, spatial coordinates, boundary contour, and occlusion relationship between the object to be enhanced and other layers can be determined by parsing the structured layer information corresponding to the target image. Based on the category identifier or object index, the region corresponding to the object to be enhanced can be extracted from the data of each layer. Subsequently, a mask that accurately corresponds to the object to be enhanced in the target image can be generated by combining the object segmentation results, layer channels, depth information, contour point sets, or semantic tags.

[0023] Here, the target region image refers to the local image content corresponding to the object to be enhanced, which is separated or extracted from the target image. It includes the pixel information of the object to be enhanced itself, and may also include a certain range of context region around the object to be enhanced.

[0024] In this embodiment, the mask and the target image can be aligned first, and the corresponding pixels in the target image can be filtered, cropped, or extracted based on the effective area identified by the mask, thereby obtaining a target area image containing only the object to be enhanced. For example, the image content within the mask can be preserved and irrelevant background outside the mask can be suppressed by multiplying pixels, or the target image can be locally cropped according to the bounding rectangle or minimum enclosing area of ​​the mask.

[0025] Here, pre-set prompts refer to textual descriptions or semantic control information used to guide the image generation model in enhancing the details of the target region image. An image generation model is a pre-trained model capable of generating or editing image content based on the input image and semantic prompts.

[0026] In this embodiment, the target region image and preset prompt word information can be input into a pre-trained image generation model. The image generation model, while maintaining the original object's main structure, pose, contour, and basic semantics, performs enhancement processing on the target region image, thereby outputting an enhanced region image. For example, the image generation model can first perform feature encoding on the target region image, and then combine it with prompt word information to form cross-modal conditional constraints, thereby generating local image content that better meets the expected style and realism requirements.

[0027] Here, the image to be displayed refers to the final output image obtained by fusing the region enhancement image with the original target image according to the mask. It not only retains the original content of the unenhanced area in the target image, but also shows the enhanced details in the area where the object to be enhanced is located.

[0028] In this embodiment, a mask can be used as a fusion weight or region constraint to map the region-enhanced image back to the position in the target image corresponding to the object to be enhanced. Within the mask-covered area, the region-enhanced image and the target image are synthesized using methods such as pixel replacement, weighted fusion, edge feathering, or multi-resolution fusion to obtain the image to be displayed. Specifically, the image to be displayed is generated within a preset time interval after the rendering engine outputs the target image; the preset time interval can be on the order of seconds.

[0029] The technical solution of this disclosure can effectively improve the texture details, realism, and overall aesthetics of various objects in the rendering scene without significantly increasing the real-time computing burden of the rendering engine. Simultaneously, by employing local extraction and local enhancement methods, it avoids high-complexity re-rendering of the entire image. This not only helps reduce processing time and computing resource consumption, improving real-time performance and practicality, but also reduces style drift or distortion in irrelevant areas caused by global processing. The resulting image is improved in visual quality, local realism, and overall harmony, while reducing image enhancement time to the second level, thereby enhancing the user's visual perception and interactive experience.

[0030] In some embodiments, generating a mask corresponding to the object to be enhanced in the target image based on structured layer information includes: determining identification information and a region mapping to be enhanced corresponding to the object to be enhanced from the structured layer information; generating a mask based on the identification information using the region mapping to be enhanced; the mask includes target region marker values ​​and non-target region marker values.

[0031] Here, identification information refers to attribute or index information used to characterize the object to be enhanced, distinguishing it from other objects, regions, or layers in the structured layer information. The region mapping to be enhanced refers to a data representation that reflects the spatial distribution, regional positional relationships, or pixel correspondences of the object to be enhanced in the target image, used to express the locations in the target image that belong to and do not belong to the object to be enhanced.

[0032] In this embodiment, the structured layer information can originate from scene description data, layered composition data, object instance segmentation results, or other data files that can reflect the hierarchical relationships and region affiliations of objects in an image, output by the rendering engine. After obtaining the structured layer information, the identification information corresponding to the object to be enhanced can be retrieved from the structured layer information based on the pre-determined object to be enhanced. For example, when the object to be enhanced is a person, the instance identifier (ID), semantic category label, or layer number corresponding to the person instance can be determined from the structured layer information; when the object to be enhanced is an animal, the object identifier or local attribute identifier corresponding to the animal body can be determined. Further, the region mapping to be enhanced associated with the identification information can be extracted from the structured layer information. For example, the instance segmentation mask corresponding to the identification information can be read directly, or the region mapping result of the object to be enhanced in the target image can be constructed based on the transparency channel, contour point set, projection area, depth relationship, and pixel affiliation relationship of the object layer.

[0033] Here, the target region marker value and the non-target region marker value refer to the marker values ​​used to distinguish between the regions to be enhanced and the regions not to be enhanced when generating the mask. Specifically, the target region marker value is used to indicate the regions in the mask that belong to the object to be enhanced, and the non-target region marker value is used to indicate the regions that do not belong to the object to be enhanced.

[0034] In this embodiment, a mask matrix with the same or corresponding size as the target image can be established first. The areas covered by the mapped region to be enhanced are assigned target region label values, while the uncovered areas are assigned non-target region label values, thereby generating a mask characterizing the spatial extent of the object to be enhanced. Specifically, if the region to be enhanced is mapped as a pixel-level region, values ​​can be directly assigned according to pixel positions; if the region to be enhanced is mapped as a contour, bounding box, or set of region coordinates, it can first be rasterized or projected onto the target image coordinate system before label value filling.

[0035] In this way, the truly important object regions needing enhancement can be located with high accuracy in complex rendering scenes, thus avoiding the mistaken inclusion of background areas or irrelevant objects in subsequent processing. Simultaneously, by fully utilizing the existing object identity information and region correspondences in the structured layer information, relatively accurate masking results can be obtained without relying entirely on additional image segmentation calculations, which helps reduce processing complexity and improve processing efficiency. Furthermore, by clearly distinguishing image regions using target region marker values ​​and non-target region marker values, stable and clear regional constraints can be provided for subsequent local extraction, detail enhancement, and fusion processes, thereby improving the targeting, accuracy, and visual consistency of local enhancements and the final image.

[0036] In some embodiments, the identification information includes at least one of color identification, semantic identification, and object identification.

[0037] Here, color coding refers to the method of distinguishing and marking different layers, regions, or objects in a target image using pre-defined color information.

[0038] In this embodiment of the disclosure, when the identification information is a color identifier, the target color code corresponding to the object to be enhanced can be determined first from the structured layer information, and pixel regions that are consistent with the target color code or meet the preset color similarity conditions can be retrieved from the layer data, index map, or annotation map corresponding to the target image. For example, if independent color labels have been assigned to people, animals, food, or other objects in the structured layer information, the color tag values ​​of each pixel in the image can be traversed, and the pixel positions whose color values ​​match the target color code can be determined as the regions to be enhanced. Specifically, a color tolerance range can also be set to include pixels whose colors are close to the target color code in the regions to be enhanced. After determining the corresponding pixel range, this range can be mapped to a mask matrix with the same size as the target image, and the pixel positions can be assigned target region tag values, while other positions can be assigned non-target region tag values, thereby generating a mask.

[0039] Here, semantic identifiers refer to identifier information used to represent the semantic attributes of object categories or regions in a target image.

[0040] In this embodiment of the disclosure, when the identification information is a semantic identifier, the semantic category corresponding to the object to be enhanced can be identified based on the semantic category labels, semantic segmentation results, or scene semantic description information in the structured layer information, and the region belonging to the semantic category can be extracted as the region to be enhanced. For example, if the object to be enhanced is a human face, animal fur, or food surface texture, the corresponding semantic category label can be determined first, such as "human," "face," "fur," "food," etc., and then all pixel positions assigned to the semantic category can be extracted from the semantic segmentation map, semantic layer, or region semantic mapping table. Subsequently, these pixel positions can be marked as target regions, and target region label values ​​can be assigned in the mask, while other regions that do not belong to the semantic category can be assigned non-target region label values, thereby obtaining the mask. In particular, to avoid the semantic category being too broad and including irrelevant regions, spatial constraints, depth information, region area thresholds, adjacency relationships, or local context rules can be further combined to filter and refine the semantic region to ensure that the final mask fits the true boundary of the object to be enhanced more closely.

[0041] Here, object identifier refers to the identification information used to uniquely mark a specific object instance in the target image, and is used to distinguish different objects under the same semantic category from each other.

[0042] In this embodiment of the disclosure, when the identification information is an object identifier, the specific instance region corresponding to the object to be enhanced can be directly retrieved based on the instance number, object number, or layer number in the structured layer information. For example, when multiple objects of the same type exist in the target image, such as multiple characters, multiple animals, or multiple food instances, the region mapping corresponding to the target object can be accurately found from the instance segmentation results, layer index information, or object management table using the object identifier. Subsequently, all pixel positions of the object instance in the target image can be extracted, and target region label values ​​can be assigned to these pixel positions in the mask matrix. Other object instances and background regions in the image can be assigned non-target region label values, thereby generating a mask only for the specified object.

[0043] Specifically, color identifiers, semantic identifiers, and object identifiers can be used individually or in combination. For example, semantic identifiers can be used to first determine the broad category of "people," then object identifiers can be used to select a specific person from multiple "people" instances, and finally, color identifiers can be used to further refine the boundaries of the target region. Through the synergistic effect of various identifiers, more accurate, stable, and adaptable masking results for complex scenes can be generated, providing a reliable foundation for subsequent target region extraction and local enhancement.

[0044] Thus, by further refining the identification information into color identifiers, semantic identifiers, and object identifiers, and employing corresponding mask generation methods for different types of identification information, it is possible to more flexibly adapt to layer data and scene description information from different sources and with different structural forms. Mask generation based on color identifiers facilitates the rapid location of the area to be enhanced using layer-coded colors, offering advantages such as simplicity and high processing efficiency. Mask generation based on semantic identifiers can accurately filter areas with specific semantic attributes at the object category level, ensuring consistency between the enhancement processing and the semantics of the target content. Mask generation based on object identifiers can accurately locate specific instances in complex scenes with multiple similar objects, avoiding the misselection of other similar objects.

[0045] In some embodiments, after generating a mask corresponding to the object to be enhanced in the target image based on the structured layer information, the method further includes: expanding the outer edge of the mask by a predetermined range according to preset expansion parameters to generate an extended mask; smoothing the transition region corresponding to the edge of the extended mask according to preset blur parameters to generate an edge transition mask, and using the edge transition mask as a mask.

[0046] Here, the preset expansion parameters refer to the parameter information used to control the outward expansion method and degree of the mask edge, which is used to determine the range, scale or shape when expanding the boundary of the target area on the basis of the original mask.

[0047] In this embodiment, an original mask can be obtained first, and the outer region of the mask edge can be expanded according to preset expansion parameters. Exemplarily, the expansion process can be implemented through morphological dilation, boundary expansion based on distance transform, convolution expansion based on structuring elements, and other region expansion methods, thereby expanding the boundary of the target region covered by the original mask outward by a predetermined range to generate an expanded mask. Specifically, the expansion range can be adaptively determined according to the target image resolution, the size of the object to be enhanced, or the complexity of the object boundary, to ensure that the expanded region can cover the detailed transition area near the edge of the target object without excessively intruding into a large amount of irrelevant background area.

[0048] Here, the preset blur parameters refer to the parameter information used to control the intensity and range of smoothing processing of the transition area at the edge of the extended mask, and are used to determine the change between the target area and the non-target area in the edge transition mask.

[0049] In this embodiment, the transition region corresponding to the edge of the extended mask can be smoothed according to preset blur parameters to generate an edge transition mask. For example, the transition zone region formed by extending outward from the original mask boundary in the extended mask can be identified first. Then, Gaussian blur, mean smoothing, feathering, bilateral filtering, or other edge softening processing can be applied to this transition zone region to make the mask value change continuously between the target region and the non-target region, rather than switching abruptly. In the resulting edge transition mask, the position closer to the target region can maintain a higher mask weight, while the position closer to the non-target region gradually decreases the mask weight, thus forming a smooth edge fusion band. Specifically, after the smoothing process is completed, the edge transition mask can be numerically normalized, clipped at the boundary, or corrected for local consistency to meet the weight usage requirements when fusing the enhanced region image with the target image. Finally, the edge transition mask can be used as the actual mask in subsequent steps to replace the initially generated hard boundary mask.

[0050] Thus, performing edge expansion and blur smoothing processing after generating the original mask effectively expands the adjustable area near the edge of the object to be enhanced. This allows the enhancement result to cover the boundary transition between the object and its surroundings, reducing problems such as stitching marks, edge breaks, brightness jumps, or color discontinuities caused by overly harsh mask boundaries. Furthermore, by using preset expansion and blur parameters to controllably adjust the mask, the edge transition range and smoothing intensity can be flexibly set according to different object types, image resolutions, and enhancement requirements, improving the applicability and adjustability of the solution.

[0051] In some embodiments, extracting a target region image from a target image based on a mask includes: determining the bounding box of the region corresponding to the object to be enhanced based on the mask; and extracting the target region image from the target image based on the bounding box.

[0052] In this embodiment, the target region within the mask can be identified first, and the pixel positions in the mask assigned with target region marker values ​​can be considered as the effective region corresponding to the object to be enhanced. Subsequently, the effective region can be scanned to statistically analyze the horizontal and vertical distribution range of all target region pixels in the image coordinate system, and determine their minimum and maximum horizontal, vertical, and minimum vertical coordinates, thereby generating a bounding box that can enclose the region corresponding to the object to be enhanced. Specifically, the bounding box can be a rectangular bounding box, or it can be constructed as a bounding box with a preset outward margin, to fully cover the object to be enhanced while appropriately preserving its surrounding context region. Furthermore, to avoid isolated noise points or local gaps in the mask affecting the bounding box calculation results, preprocessing operations such as connected component analysis, noise removal, hole filling, boundary smoothing, or minimum area filtering can be performed on the mask before the bounding box is determined.

[0053] In this embodiment of the disclosure, after determining the bounding box of the region corresponding to the object to be enhanced, the bounding box can be mapped onto the target image as a local cropping region. Based on the coordinate range defined by the bounding box, image content corresponding to that range is extracted from the target image to obtain the target region image. For example, corresponding pixel blocks can be extracted from the target image according to the coordinates of the upper left corner, width, and height of the bounding box, ensuring that the target region image at least contains the object to be enhanced. Specifically, to enable the subsequent image generation model to better understand the relationship between the object to be enhanced and its surrounding environment, a preset pixel range can be further extended outward from the bounding box, so that the extracted target region image simultaneously includes background information, lighting transition information, or texture reference information near the object's edges.

[0054] Thus, by first accurately determining the spatial extent of the object to be enhanced in the target image using a mask, then generating a corresponding bounding box based on this spatial extent, and further extracting the target region image from the target image based on the bounding box, the localized and precise extraction of the object to be enhanced can be effectively achieved. This avoids the waste of computational resources and interference from irrelevant regions caused by directly processing the entire image indiscriminately. Simultaneously, the bounding box facilitates the rapid determination of the cropping range of the target region, offering advantages such as simple implementation, high processing efficiency, and ease of engineering deployment. Furthermore, it can incorporate surrounding contextual information as needed while ensuring the complete preservation of the main body of the object to be enhanced, thereby improving the subsequent image generation model's understanding of local structure, edge transitions, and environmental relationships.

[0055] In some embodiments, after extracting the target region image, the method further includes scaling the target region image to make the resolution of the target region image meet a preset resolution range.

[0056] Here, scaling refers to image processing operations that adjust the spatial dimensions of the target region image to change its width, height, or overall resolution.

[0057] In this embodiment, the current size information of the target region image can be read, including pixel width, pixel height, aspect ratio, and corresponding resolution, and a scaling operation can be determined according to pre-set processing rules. When it is determined that the size of the target region image is not suitable for direct input into the subsequent image generation model, it can be scaled. For example, if the target region image size is too small, it can be enlarged by upsampling to increase the effective pixel coverage of the image in the model input space; if the target region image size is too large, it can be compressed by downsampling to reduce the computational burden and memory usage during subsequent model inference. During the scaling process, an appropriate interpolation algorithm can be selected according to the image content features and accuracy requirements. In particular, to avoid deformation of the target object caused by scaling, the original aspect ratio can be kept unchanged, and the scaled image can be adjusted to a uniform input size through edge padding, center alignment, or edge compensation.

[0058] Thus, by scaling the extracted target region image to meet a preset resolution range, the problem of inconsistent input caused by differences in size, cropping range, or original image resolution among different objects to be enhanced can be effectively solved. This allows the subsequent image generation model to perform region enhancement processing at a relatively stable input scale. Furthermore, controlling the target region image within a preset resolution range helps to achieve a balance between enhancement effect, computational complexity, and model adaptability, thereby improving the stability, versatility, and visual quality of the final output image of the entire image enhancement process.

[0059] In some embodiments, before inputting the target region image and preset prompt words for enhancing image details into a pre-trained image generation model, the method further includes: obtaining the object type of the object to be enhanced; determining the current task requirements based on the object type; determining a target computing resource pool from a set of computing resource pools based on the current task requirements; and calling the image generation model using the target computing resource pool.

[0060] Here, object type refers to the semantic category identifier of the object to be enhanced, which is used to characterize the content category to which the main enhanced object in the target area image belongs. It can include people, food, animals, clothing, buildings, landscapes, goods, etc.

[0061] In this embodiment, target detection, semantic segmentation, or instance segmentation can be performed on the target image to identify the main object in the image. Then, based on the identification results, the target region image is extracted, and the object to be enhanced is classified using a pre-trained image classification model, multimodal recognition model, or label-based content analysis module to obtain the object type. Alternatively, the object type can be directly obtained based on type markers in the target image.

[0062] Here, the current task requirements refer to the resource requirements description information corresponding to completing the current image enhancement task, which may include, but is not limited to, computing power requirements, video memory requirements, graphics processor model requirements, inference latency requirements, parallel processing requirements, model version adaptation requirements, and corresponding graphics processor pool selection information.

[0063] In this embodiment of the disclosure, after obtaining the object type, the current task requirements can be generated based on the object type. For example, a mapping relationship library between object types and task requirements can be pre-established, or a task requirement prediction model can be constructed based on historical training data. When the identification result indicates that the object to be enhanced belongs to a specific category, the resource requirement parameters corresponding to that category can be searched from the mapping relationship library, or the matching task requirements can be dynamically calculated through the prediction model. The task requirements may include information such as the computing power level required to perform the image enhancement task, the range of video memory capacity, the applicable graphics processor model, the supported model framework, the target processing latency, and the recommended graphics processor pool identifier. In particular, the current task requirements can be modified and refined by comprehensively considering the resolution of the target region image, the area of ​​the region to be enhanced, the complexity of the preset prompt words, the expected output quality level, and the current load.

[0064] Here, the computing resource pool set refers to a pre-built and maintained set of resources used to perform image enhancement tasks. Essentially, it can be a graphics processing unit (GPU) cluster or a heterogeneous computing cluster. This computing resource pool set includes GPU resources of various models, as well as multiple GPU pools composed of GPUs of the same or similar models. Each GPU pool can be divided according to dimensions such as model adaptability, computing performance, video memory capacity, task priority, and object type preference.

[0065] In this embodiment of the disclosure, after determining the current task requirements, a target computing resource pool matching the task requirements can be selected from a pre-built set of computing resource pools. For example, the parameters in the current task requirements can be compared with the resource description information of each graphics processor pool. Then, rule matching, priority ranking, scoring mechanisms, heuristic scheduling algorithms, or intelligent scheduling models are used to comprehensively evaluate multiple candidate graphics processor pools, and a target computing resource pool that meets the performance requirements and has superior resource utilization efficiency is selected. The resource description information may include graphics processor model, number of graphics processors, single-card computing power, available video memory, current load rate, queuing latency, model adaptation list, and priority configuration for object-oriented types. Specifically, if multiple graphics processor pools meet the current task requirements, the final target computing resource pool for executing the current enhanced task can be determined based on load balancing strategies, real-time resource usage, service level agreement requirements, or task response latency requirements.

[0066] In this embodiment, after selecting the target computing resource pool, a pre-trained image generation model can be loaded or invoked on the target computing resource pool. The target region image and preset prompts for enhancing image details are input into the image generation model to perform image detail enhancement processing. For example, task instructions can be issued to the corresponding computing nodes in the target computing resource pool to complete operations such as model instance creation, inference environment initialization, memory allocation, and input data loading. Then, the image generation model performs generative optimization on the texture, edges, colors, lighting, or structural details in the target region based on the original content of the target region image and the enhancement direction described by the preset prompts, and outputs the enhanced target region image.

[0067] In this way, it is possible to select more suitable computing resources and graphics processor pools based on the enhancement characteristics of different object types such as people, food, and animals. This improves the adaptability and output quality of image generation models in detail enhancement scenarios and avoids poor enhancement effects caused by improper resource allocation. Furthermore, it can dynamically allocate target computing resource pools according to current task requirements, improving the utilization efficiency of heterogeneous graphics processor clusters, reducing the indiscriminate occupation of high-performance resources, thereby reducing task queuing time and overall inference costs, and improving processing stability and scalability in multi-task concurrent scenarios.

[0068] In some embodiments, the image generation model is obtained by: determining low-rank training parameters corresponding to the object type based on the object type; performing quantization and acceleration processing on the base model to determine the target base model; and loading the low-rank training parameters into the pre-trained target base model to obtain the image generation model.

[0069] Here, low-rank training parameters refer to a set of lightweight incremental parameters obtained by training on the base model using a low-rank adaptation method for a specific object type. These parameters can perform low-dimensional decomposition and learning only for the weight changes of some layers while keeping the main parameters of the base model basically unchanged.

[0070] In this embodiment, the object type of the object to be enhanced can be obtained first based on the results of the preceding processing, and the corresponding low-rank training parameters can be selected from a pre-built parameter mapping relationship according to the object type. For example, a database associating object types with low-rank training parameters can be pre-established to store multiple sets of low-rank parameters trained separately for different object types. Once it is identified that the current object to be enhanced belongs to a category such as people, food, or animals, the low-rank training parameters corresponding to that category can be selected from the database through methods such as table lookup matching, tag indexing, rule retrieval, or similarity search. Specifically, if the image content involves multiple object types simultaneously, the main parameter set can be determined from multiple low-rank training parameters based on the main object weight, region proportion, task priority, or fusion strategy, or multiple low-rank training parameters can be combined and loaded.

[0071] In this embodiment, quantization and acceleration processing can be performed on a pre-trained general-purpose base model to obtain a target base model suitable for online invocation and efficient inference. For example, quantization processing may compress the floating-point weights in the base model from a higher-precision format to a lower-precision format to reduce memory usage and computational overhead. Acceleration processing may include model structure pruning, inference graph optimization, operator fusion, tensor compilation optimization, cache optimization, parallel scheduling optimization, and inference engine adaptation for specific hardware platforms, thereby improving the model's execution speed and throughput in a real-world deployment environment. Specifically, the quantization and acceleration strategies can be adaptively adjusted based on the graphics processor model, memory capacity, supported precision, and driver environment in the selected computing resource pool to ensure that the processed target base model meets both the quality requirements of the image enhancement task and the stable deployment needs under current computing resource conditions.

[0072] In this embodiment of the disclosure, low-rank training parameters can be loaded into a target base model that has undergone quantization and acceleration processing, thereby constructing an image generation model for the current object type. For example, according to a preset model layer mapping relationship, the low-rank training parameters can be injected into the attention layer, linear layer, or other adaptable layers corresponding to the target base model, so that the base model, while retaining its general image generation capabilities, further possesses detailed representation capabilities and semantic bias capabilities that match the current object type.

[0073] In some implementations, the base model can be a Flux model. The Flux model is an image generation model built on a transformer structure, possessing a large parameter scale and exhibiting strong generation capabilities and robust ecosystem adaptability among open-source image generation models. Specifically, the Flux model can include two core components: a Multi-Modal Diffusion Transformer (MM-DiT) and a Single-Diffusion Transformer (Single-DiT). MM-DiT receives multimodal inputs such as text and image information and performs joint fusion processing. Single-DiT further increases the model depth based on the fused data, enhancing the model's learning and generation capabilities for overall modal information. In this embodiment, the Flux model's strong image generation and detail reconstruction capabilities can be utilized to enhance the target region in the rendered result, making the enhanced image closer to a realistic and usable image in terms of detail, texture, and overall visual effect.

[0074] In some implementations, low-rank training parameters can be achieved using Low-Rank Adaptation (LoRA). This involves transforming the fine-tuning process for the entire model parameters into a training process for two low-rank matrices, thus avoiding direct full-scale fine-tuning of the Flux base model and significantly reducing training and deployment costs. Specifically, while the Flux model possesses strong general image generation capabilities, its realistic enhancement capabilities may still be insufficient in certain specific scenarios. Therefore, LoRA can be introduced to fine-tune the model on target category data, enhancing its realistic representation, detail restoration, and scene adaptability across different image categories, thereby achieving better output results in rendering image enhancement tasks.

[0075] In some implementations, quantization can employ the Singular Value Decomposition Quantization (Svdquant) method, a low-loss quantization approach for Diffusion Transformer (DiT) structures. Specifically, since unquantized Flux models have high computational resource and latency requirements during inference, making them unsuitable for real-time applications, quantization is needed to reduce inference costs. Svdquant can preserve the original model's generation effect as much as possible at a higher quantization compression rate, thereby reducing memory usage, computational burden, and inference time while minimizing the impact of quantization on the final image quality. This makes Flux models more suitable for real-time image enhancement scenarios.

[0076] In some implementations, inference acceleration strategies can be combined to further shorten the processing time of the Flux model. For example, acceleration methods based on caching mechanisms and methods based on reducing the number of inference steps can be employed. The caching mechanism can utilize a wavespeed scheme to reduce the overhead of repetitive computations in a single round of inference, thus reducing inference waiting time. The method for reducing the number of inference steps can employ a Hyper-Stable Diffusion (Hyper-SD) scheme to compress the number of iterations required for the overall inference process while ensuring that the quality of the generated results is generally acceptable. By applying these acceleration strategies to both the single-round inference process and the overall inference process, the response speed of the Flux model in rendering result enhancement tasks can be further improved, thereby better meeting the requirements for real-time performance and processing efficiency in practical applications.

[0077] In this way, low-rank training parameters only supplement the model's local capabilities in a targeted manner, without requiring repeated training and complete replacement of the entire base model. This significantly reduces model training costs, storage costs, and switching overhead, while improving reusability across different object types. Through quantization and acceleration processing, memory usage and computational latency during model inference can be effectively reduced, making the image generation model more suitable for deployment in real-world business environments. Furthermore, combining object types with dedicated low-rank training parameters enhances the model's detail generation quality and semantic consistency across different content scenarios such as people, food, and animals, thus balancing enhancement effects, inference efficiency, and resource utilization.

[0078] In some embodiments, inputting a target region image and preset prompt words for enhancing image details into a pre-trained image generation model to generate a region-enhanced image by the image generation model includes: inputting the target region image and preset prompt words into the image generation model to perform enhancement processing on the target region image through an image-to-image generation pipeline to generate a region-enhanced image; the enhancement processing includes at least one of realistic detail enhancement processing and aesthetic detail enhancement processing.

[0079] Here, the image-to-image generation pipeline refers to a model processing flow that uses an input image as a basis and incorporates prompts to generate or edit the input image in a targeted manner. It preserves the basic structure, contour layout, subject position, pose relationships, and main semantic content of the original target region image. Based on this, and combined with the enhancement direction expressed by the preset prompts, it performs detail reconstruction, texture supplementation, sharpness enhancement, texture optimization, color adjustment, or visual style improvement on the image, either locally or overall. Realistic detail enhancement processing refers to a type of enhancement processing aimed at improving the realism, naturalness, and objective detail representation of the target region image. This processing emphasizes the consistency between image content and the visual laws of the real world, typically focusing on restoring or strengthening the realistic texture, edge contours, light and dark transitions, local structure, material characteristics, and spatial hierarchy of the target object. Aesthetic detail enhancement processing refers to a type of enhancement processing aimed at improving the visual expressiveness, aesthetic quality, and visual appeal of the target region image.

[0080] In this embodiment, the target region image and preset prompt words for enhancing image details can be jointly input into a pre-trained image generation model. The image generation model then performs enhancement processing on the target region image through an image-to-image generation pipeline to generate a region-enhanced image. For example, the target region image can first be preprocessed, and the preset prompt words can be encoded so that image content information and semantic guidance information can be jointly input into the image generation model as conditions. Subsequently, the image generation model, while preserving the original main structure, contour features, and semantic content of the target region, performs realistic detail enhancement processing and / or aesthetic detail enhancement processing on the target region according to the enhancement direction defined by the preset prompt words. Realistic detail enhancement processing can improve the realistic texture, edge sharpness, material hierarchy, and natural lighting of the target region, while aesthetic detail enhancement processing can optimize the color, texture, local refinement, and overall visual appeal of the target region. The final output is a region-enhanced image that corresponds to the original target region content but with higher detail quality.

[0081] In this way, the image-to-image generation pipeline can avoid the content offset problem caused by complete regeneration, making the enhancement result more consistent and controllable with the original target area. At the same time, by distinguishing between realistic detail enhancement processing and aesthetic detail enhancement processing, flexible choices can be made between improving realism and optimizing aesthetics according to different business needs, thereby balancing the effect of visual detail restoration and the improvement of subjective visual perception.

[0082] In some embodiments, before fusing the region-enhanced image with the target image, the method further includes scaling the region-enhanced image so that the resolution of the region-enhanced image meets the original size of the target region image.

[0083] In this embodiment of the disclosure, before fusing the region-enhanced image with the target image, scaling processing can be performed on the region-enhanced image to ensure that the resolution of the region-enhanced image meets the original size of the target region image. For example, the size information of the target region image within the original target image can be obtained first. This size information includes the width, height, and corresponding pixel resolution range of the target region image. Then, the current resolution of the region-enhanced image is compared with the original size of the target region image. When there is a size inconsistency, proportional deviation, or resolution difference between the two, a preset image scaling algorithm is used to adjust the size of the region-enhanced image so that the adjusted region-enhanced image matches the original size of the target region image in terms of width, height, or overall resolution.

[0084] In this way, the accurate alignment of the enhanced area in terms of spatial dimensions can be effectively guaranteed, thereby avoiding problems such as fusion misalignment, inaccurate area coverage, edge misalignment or local deformation caused by size mismatch.

[0085] In some embodiments, before fusing the region enhancement image with the target image, the method further includes: performing smoothing and / or morphological processing on the edges of the mask, and using the processed mask as the mask; the morphological processing includes at least one of dilation, erosion, opening, and closing operations; adjusting the edge transition regions of the region enhancement image based on the mask, and using the adjusted region enhancement image as the region enhancement image.

[0086] In this embodiment of the disclosure, before fusing the region-enhanced image with the target image, edge optimization processing can be performed on the mask used to define the target region to improve the smoothness and accuracy of the subsequent fusion region boundaries. For example, an initial mask corresponding to the target region can be obtained first, and then smoothing and / or morphological processing can be performed on the edge regions of the mask. The smoothing process can employ Gaussian blur, mean filtering, median filtering, edge feathering, or other edge softening methods to reduce overly abrupt grayscale changes at the mask boundaries, transforming the mask edges from hard boundaries into transitional boundaries with gradual characteristics. The morphological processing can, according to actual fusion needs, perform at least one of the existing techniques of dilation, erosion, opening, and closing operations on the mask. Finally, after completing the smoothing and / or morphological processing, the optimized mask is determined as the mask actually used in the subsequent fusion process.

[0087] In this embodiment, the edge transition region of the region-enhanced image can be further adjusted based on a mask to improve the visual continuity between the region-enhanced image and the target image at the boundary connection. For example, the target subject region, edge transition region, and external background boundary range in the region-enhanced image can be determined based on the processed mask. The edge transition region refers to a pixel region within a certain width range near the mask boundary. Gradient blending, transparency adjustment, color compensation, brightness correction, texture smoothing, sharpness matching, or local feathering processing can be performed on the edge transition region to gradually transition the region-enhanced image towards the original region features of the target image near the boundary, rather than forming obvious stitching marks. Specifically, the pixels at the edges of the region-enhanced image can be adaptively corrected by combining the color distribution, lighting conditions, sharpness level, and background texture information of the corresponding region in the target image to ensure that the details inside the enhanced region are preserved while the boundary can be naturally connected with the surrounding original image. Finally, the processed region-enhanced image is used as the final region-enhanced image for fusion.

[0088] This effectively improves the boundary blending quality between the enhanced region and the original image, avoiding problems such as stitching marks, jagged edges, misaligned contours, color breaks, and discontinuous lighting caused by overly harsh mask edges, unreasonable coverage, or incomplete local structures. Simultaneously, smoothing enhances the gradation of the mask boundaries, while morphological processing improves the integrity and accuracy of the masked area. Combined with fine-tuning of the transition areas at the edges of the enhanced image, this allows the enhanced region to maintain its internal detail quality while creating a more natural, soft, and consistent visual transition with the surrounding areas of the target image.

[0089] In some embodiments, fusing a region enhancement image with a target image based on a mask to obtain an image to be displayed includes: determining the fusion position of the region enhancement image in the target image according to the relative positional relationship between the mask and the target image; and performing weighted fusion of the region enhancement image and the target image at the fusion position to obtain the image to be displayed.

[0090] In this embodiment of the disclosure, before fusing the region enhancement image with the target image based on a mask, the fusion position of the region enhancement image in the target image can be determined first based on the relative positional relationship between the mask and the target image. For example, mask information corresponding to the target region can be obtained first. The mask is used to characterize the spatial distribution range, boundary contour, and pixel coverage relationship of the target region in the target image. Subsequently, the positional parameters of the mask in the target image coordinate system can be analyzed, such as the bounding box coordinates, center position, contour range, width and height dimensions of the mask, and the mapping relationship between each pixel and the pixel grid of the target image, thereby determining the accurate region of the target region in the target image. Next, the size information, effective content range, and correspondence between the region enhancement image and the target region image can be combined to position the region enhancement image at the target location in the target image, ensuring that the main content in the region enhancement image and the target region to be replaced or enhanced in the target image are consistent in position, scale, and boundary. In particular, if there is a coordinate offset or angle difference between the region enhancement image and the target image, coordinate transformation, translation correction or geometric alignment can be further performed to ensure that the region enhancement image can accurately cover the corresponding region in the target image during subsequent fusion.

[0091] In this embodiment of the disclosure, after determining the fusion location, the region-enhanced image and the target image can be weighted and fused at the fusion location to obtain the image to be displayed. For example, a mask can be used as the basis for controlling the fusion weights, and pixel-by-pixel or region-by-region weighting calculations can be performed within the fusion location. The target region inside the mask can have a higher weight to ensure that the enhanced details in the region-enhanced image are fully preserved in the fusion result. Transition regions near the mask boundary can use gradient weights to achieve a smooth transition between the region-enhanced image and the target image in terms of color, brightness, texture, and edges. For regions outside the mask, the original pixel content of the target image is preserved to avoid unnecessary impact on non-target regions. Specific weighted fusion methods can include linear blending based on transparency coefficients, pixel-by-pixel fusion based on pixel weight maps, edge gradient fusion based on feathered masks, or an adaptive fusion method combining color correction and brightness compensation. This ensures that the fused image retains enhanced details within the target region while reducing abruptness and splicing artifacts at boundary connections. After completing the weighted fusion, the fused complete image is output and determined as the image to be displayed.

[0092] This effectively ensures accurate spatial alignment of the enhanced areas, avoiding problems such as coverage offset, boundary misalignment, or local content mismatch caused by inaccurate positioning. Furthermore, by using masking to control the fusion range and employing a weighted fusion method to progressively process edge transition areas, it not only preserves the enhanced image's internal detail improvement but also achieves a more natural connection between the enhanced areas and the surrounding content of the target image in terms of color, brightness, texture, and contour. This reduces abrupt stitching, edge breaks, and visual abrupt changes.

[0093] In some implementations, the renderer can first output its corresponding rendered image as the target image. Then, region analysis and localization are performed on the target image to determine the minimum bounding rectangle of the target region, and the image content corresponding to this minimum bounding rectangle is extracted. Next, the extracted target region image can be resized, scaling its long side to a resolution range of 512-1024 to make the adjusted image resolution more suitable for Flux model inference. The scaled target region image is then input into the Flux model, and detail enhancement processing is performed on the target region image through an image-to-image pipeline to obtain the enhanced result image. Finally, the enhanced result image is scaled back to the original size of the target region and pasted back onto the corresponding area in the original rendered image, thus obtaining the final image to be displayed with enhanced target region details.

[0094] Figure 2 The diagram illustrates another flowchart of the image enhancement method in a rendering scene according to an embodiment of this disclosure, as follows: Figure 2 As shown, it includes: S201. Obtain the rendered image as the target image.

[0095] S202. Extract the target area image and perform scaling processing, and then split the processed target area image.

[0096] S203a. Construct an animal image generation model based on the target base model and low-rank training parameters, and input the split animal target region image into the animal image generation model.

[0097] S203b. Construct a food image generation model based on the target base model and low-rank training parameters, and input the split food target region image into the food image generation model.

[0098] S203c: Construct a person image generation model based on the target base model and low-rank training parameters, and input the split person target region image into the person image generation model.

[0099] S204. Scale the region enhancement image output by the image generation model to the original size of the target image.

[0100] S205. Fuse the region enhancement image and the target image to obtain the image to be displayed.

[0101] It should be understood that Figure 2 The schematic diagrams shown are merely illustrative and not limiting, and are scalable; those skilled in the art can use them as a basis. Figure 2 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0102] This disclosure provides an image enhancement device for rendering scenes, such as... Figure 3 As shown, the device may include: a mask generation module 301, used to generate a mask corresponding to the object to be enhanced in the target image based on structured layer information; the structured layer information is generated by the rendering engine during the rendering of the target image; a region extraction module 302, used to extract the target region image from the target image based on the mask; a detail enhancement module 303, used to input the target region image and preset prompt words for enhancing image details into a pre-trained image generation model, so that the image generation model generates a region enhancement image; the region enhancement image is the image after detail enhancement of the object to be enhanced; and a fusion processing module 304, used to fuse the region enhancement image with the target image based on the mask to obtain the image to be displayed; wherein the image to be displayed is generated within a preset time interval after the rendering engine outputs the target image.

[0103] In some embodiments, the mask generation module 301 includes: an information extraction submodule, used to determine the identification information and the region mapping to be enhanced corresponding to the object to be enhanced from the structured layer information; and an information processing submodule, used to generate a mask based on the identification information according to the region mapping to be enhanced; the mask includes target region marker values ​​and non-target region marker values.

[0104] In some embodiments, the identification information includes at least one of color identification, semantic identification, and object identification.

[0105] In some embodiments, the image enhancement device in the rendering scene further includes: a mask extension module 305 ( Figure 3 (Not shown in the image), used to extend the outer edge of the mask by a predetermined range according to preset extension parameters, to generate an extended mask; Mask blurring module 306 ( Figure 3 (not shown in the image) is used to smooth the transition region corresponding to the edge of the extended mask according to the preset blur parameters, generate the edge transition mask, and use the edge transition mask as the mask.

[0106] In some embodiments, the region extraction module 302 includes: a boundary localization submodule, used to determine the bounding box of the region corresponding to the object to be enhanced based on the mask; and an image extraction submodule, used to extract the target region image from the target image based on the bounding box.

[0107] In some embodiments, the image enhancement device in the rendering scene further includes: a first scaling module 307 ( Figure 3 (Not shown in the image) is used to scale the target area image so that the resolution of the target area image meets a preset resolution range.

[0108] In some embodiments, the image enhancement device in the rendering scene further includes: a type acquisition module 308 ( Figure 3 (Not shown in the image), used to obtain the object type of the object to be enhanced; Requirements determination module 309 ( Figure 3 (Not shown in the image), used to determine the current task requirements based on the object type; Resource Selection Module 310 ( Figure 3 (Not shown in the image), used to determine the target computing resource pool from the set of computing resource pools based on the current task requirements; Model Invocation Module 311 ( Figure 3 (not shown in the image), used to invoke the image generation model using the target computing resource pool.

[0109] In some embodiments, the image generation model is obtained by: determining low-rank training parameters corresponding to the object type based on the object type; performing quantization and acceleration processing on the base model to determine the target base model; and loading the low-rank training parameters into the pre-trained target base model to obtain the image generation model.

[0110] In some embodiments, the detail enhancement module 303 includes: an enhancement processing submodule, configured to input a target region image and preset prompt word information into an image generation model, so that the image generation model performs enhancement processing on the target region image through an image-to-image generation pipeline to generate a region-enhanced image; the enhancement processing includes at least one of realistic detail enhancement processing and aesthetic detail enhancement processing.

[0111] In some embodiments, the image enhancement device in the rendering scene further includes: a second scaling module 312 ( Figure 3 (Not shown in the image) is used to scale the region-enhanced image so that the resolution of the region-enhanced image matches the original size of the target region image.

[0112] In some embodiments, the image enhancement device in the rendering scene further includes: a masking processing module 313 ( Figure 3(not shown), used to perform smoothing and / or morphological processing on the edges of the mask, and to use the processed mask as the mask; the morphological processing includes at least one of dilation, erosion, opening and closing operations; edge transition module 314 ( Figure 3 (not shown in the image) is used to adjust the edge transition regions of the region enhancement image based on the mask, and the adjusted region enhancement image is used as the region enhancement image.

[0113] In some embodiments, the fusion processing module 304 includes: a fusion positioning submodule, configured to determine the fusion position of the region enhancement image in the target image based on the relative positional relationship between the mask and the target image; and a weighted fusion submodule, configured to perform weighted fusion of the region enhancement image and the target image at the fusion position to obtain the image to be displayed.

[0114] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0115] The image enhancement device in the rendering scene of this disclosure can effectively improve the texture details, realism, and overall aesthetics of various objects to be enhanced in the rendering scene without significantly increasing the real-time computing burden of the rendering engine. Simultaneously, by employing local extraction and local enhancement methods, it avoids high-complexity re-rendering of the entire image. This not only helps reduce processing time and computing resource consumption, improving real-time performance and practicality, but also reduces style drift or distortion in irrelevant areas caused by global processing. The resulting image to be displayed is improved in terms of visual quality, local realism, and overall harmony, while reducing image enhancement time to the second level, thereby enhancing the user's visual perception and interactive experience.

[0116] This disclosure provides a scene illustration of an image enhancement method in a rendering scene, such as... Figure 4 As shown.

[0117] As previously described, the image enhancement method for rendering scenarios provided in this disclosure is applied to electronic devices. The electronic device is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Specifically, the electronic device may perform the following operations: A mask corresponding to the object to be enhanced in the target image is generated based on the structured layer information; the structured layer information is generated by the rendering engine during the rendering of the target image; the target region image is extracted from the target image based on the mask; the target region image and preset prompt words for enhancing image details are input into a pre-trained image generation model to generate a region enhancement image; the region enhancement image is the image after enhancing the details of the object to be enhanced; the region enhancement image is fused with the target image based on the mask to obtain the image to be displayed; the image to be displayed is generated within a preset time interval after the rendering engine outputs the target image.

[0118] It should be understood that Figure 4 The scene diagrams shown are merely illustrative and not restrictive; those skilled in the art can interpret them based on... Figure 4 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0119] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0120] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0121] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0122] like Figure 5As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0123] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0124] The computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as image enhancement methods for rendering scenes. For example, in some embodiments, the image enhancement methods for rendering scenes can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the image enhancement methods for rendering scenes described above can be performed. Alternatively, in other embodiments, computing unit 501 may be configured to perform image enhancement methods in a rendering scene by any other suitable means (e.g., by means of firmware).

[0125] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0128] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0129] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0130] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0131] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0132] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for image enhancement in a rendering scene, characterized in that, The method includes: A mask corresponding to the object to be enhanced in the target image is generated based on the structured layer information; the structured layer information is generated by the rendering engine during the rendering process of obtaining the target image; Extract the target region image from the target image based on the mask; The target region image and preset prompt words for enhancing image details are input into a pre-trained image generation model to generate a region-enhanced image; the region-enhanced image is the image of the object to be enhanced after detail enhancement. The enhanced image of the region is fused with the target image based on the mask to obtain the image to be displayed; wherein the image to be displayed is generated within a preset time interval after the rendering engine outputs the target image.

2. The method according to claim 1, characterized in that, The step of generating a mask corresponding to the object to be enhanced in the target image based on the structured layer information includes: The identification information and the mapping of the region to be enhanced corresponding to the object to be enhanced are determined from the structured layer information; Based on the mapping of the region to be enhanced, the mask is generated using the identification information; the mask includes target region label values ​​and non-target region label values.

3. The method according to claim 2, characterized in that, The identification information includes at least one of color identification, semantic identification, and object identification.

4. The method according to claim 1, characterized in that, After generating the mask corresponding to the object to be enhanced in the target image based on the structured layer information, the method further includes: The outer edge of the mask is extended by a predetermined range according to preset extension parameters to generate an extended mask. The transition region corresponding to the edge of the extended mask is smoothed according to the preset blur parameters to generate an edge transition mask, and the edge transition mask is used as the mask.

5. The method according to claim 1, characterized in that, Extracting the target region image from the target image based on the mask includes: The bounding box of the region corresponding to the object to be enhanced is determined based on the mask; Based on the bounding box, the target region image is extracted from the target image.

6. The method according to claim 1, characterized in that, After extracting the target region image, the process also includes: The target region image is scaled to ensure that the resolution of the target region image meets a preset resolution range.

7. The method according to claim 1, characterized in that, Before inputting the target region image and preset prompts for enhancing image details into the pre-trained image generation model, the method further includes: Obtain the object type of the object to be enhanced; Determine the current task requirements based on the object type; Based on the current task requirements, a target computing resource pool is determined from the set of computing resource pools; The image generation model is invoked using the target computing resource pool.

8. The method according to claim 7, characterized in that, The image generation model is obtained in the following way: Based on the object type, determine the low-rank training parameters corresponding to the object type; The basic model is quantized and accelerated to determine the target basic model; The low-rank training parameters are loaded into the pre-trained target base model to obtain the image generation model.

9. The method according to claim 1, characterized in that, The step of inputting the target region image and preset prompt words for enhancing image details into a pre-trained image generation model, so that the image generation model generates a region-enhanced image, includes: The target region image and the preset prompt information are input into the image generation model, so that the image generation model performs enhancement processing on the target region image through the image-to-image generation pipeline to generate the region enhancement image; the enhancement processing includes at least one of realistic detail enhancement processing and aesthetic detail enhancement processing.

10. The method according to claim 1 or 6, characterized in that, Before fusing the region-enhanced image with the target image, the method further includes: The enhanced region image is scaled so that its resolution matches the original size of the target region image.

11. The method according to claim 1, characterized in that, Before fusing the region-enhanced image with the target image, the method further includes: The edges of the mask are smoothed and / or morphologically processed, and the processed mask is used as the mask; the morphological processing includes at least one of dilation, erosion, opening, and closing operations. The edge transition region of the region enhancement image is adjusted based on the mask, and the adjusted region enhancement image is used as the region enhancement image.

12. The method according to claim 1, characterized in that, The step of fusing the enhanced region image with the target image based on the mask to obtain the image to be displayed includes: Based on the relative positional relationship between the mask and the target image, the fusion position of the region enhancement image in the target image is determined; At the fusion location, the enhanced region image and the target image are weighted and fused to obtain the image to be displayed.

13. An image enhancement device for a rendering scene, characterized in that, The device includes: The mask generation module is used to generate a mask corresponding to the object to be enhanced in the target image based on the structured layer information; the structured layer information is generated by the rendering engine during the rendering process of obtaining the target image; The region extraction module is used to extract a target region image from the target image based on the mask; The detail enhancement module is used to input the target region image and preset prompt words for enhancing image details into a pre-trained image generation model, so that the image generation model generates a region-enhanced image; the region-enhanced image is the image of the object to be enhanced after detail enhancement; The fusion processing module is used to fuse the enhanced region image with the target image based on the mask to obtain an image to be displayed; wherein the image to be displayed is generated within a preset time interval after the rendering engine outputs the target image.

14. An electronic device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

15. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-12.

Images