An image rendering method, apparatus, device and medium

Abstract

Embodiments of the present disclosure relate to an image rendering method, device, equipment and medium. In at least one embodiment of the present disclosure, for a first resolution rendering task of a target image, instead of directly executing the first resolution rendering task by a renderer, a second resolution rendering is executed by the renderer, the second resolution being smaller than the first resolution; then a plurality of layers generated in the second resolution rendering process of the renderer are obtained, and the plurality of layers are processed to obtain a first resolution image corresponding to the target image. Since the time for the renderer to execute the second resolution rendering is at least half shorter than that for the first resolution rendering, and the time for processing the plurality of layers is at least half shorter than that for the second resolution rendering, the embodiments of the present disclosure can shorten the time for rendering the first resolution image and improve user experience.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of image processing, and particularly relate to an image rendering method, device, equipment and medium. BACKGROUND

[0002] For the image rendering task of ultra-high definition resolution (such as 4K resolution, 10K resolution), the current renderer calculates a large number of light samplings for each pixel point based on the Monte Carlo ray tracing rendering mode, and the rendering time is relatively long, and the rendering time is basically proportional to the resolution, for example, the rendering time of 4K resolution is at least 2 times of that of 2K resolution. Therefore, it is urgent to provide an image rendering scheme to reduce the rendering time. SUMMARY

[0003] At least one embodiment of the present disclosure provides an image rendering method, device, equipment and medium.

[0004] In a first aspect, an embodiment of the present disclosure provides an image rendering method, comprising:

[0005] in response to a first resolution rendering task of a target image, performing second resolution rendering on the target image by a renderer, the second resolution being less than the first resolution;

[0006] obtaining a plurality of layers from the renderer, the plurality of layers being layers generated by the renderer in the process of the second resolution rendering;

[0007] processing the plurality of layers based on the first resolution to obtain a first resolution image corresponding to the target image.

[0008] In some embodiments, processing the plurality of layers based on the first resolution to obtain a first resolution image corresponding to the target image comprises:

[0009] performing super-resolution reconstruction on each first-type layer in the plurality of layers to obtain a first resolution layer corresponding to each first-type layer;

[0010] performing magnification processing on each second-type layer in the plurality of layers to obtain a first resolution layer corresponding to each second-type layer;

[0011] combining the first resolution layer corresponding to each first-type layer and the first resolution layer corresponding to each second-type layer to obtain the first resolution image corresponding to the target image;

[0012] wherein the first-type layer is a layer with a detail information amount greater than or equal to a preset threshold, and the second-type layer is a layer with a detail information amount less than the preset threshold.

[0013] In some embodiments, the first-type layer includes at least one of the following layers: a diffuse reflection layer, an illumination layer, a reflection layer, and a refraction layer.

[0014] The second type of layer includes at least one of the following: a reflection intensity layer, a refraction intensity layer, a highlight layer, a self-luminous layer, and a background layer.

[0015] In some embodiments, after the plurality of layers are processed based on the first resolution to obtain the first resolution image corresponding to the target image, the image rendering method further includes:

[0016] performing super-resolution reconstruction on the target image to obtain a first resolution RGB image;

[0017] based on the lighting layer and the first resolution RGB image, removing white edges from the first resolution image corresponding to the target image to obtain a first resolution image corresponding to the target image after the white edges are removed.

[0018] In some embodiments, based on the lighting layer and the first resolution RGB image, removing white edges from the first resolution image corresponding to the target image to obtain a first resolution image corresponding to the target image after the white edges are removed includes:

[0019] determining a first resolution white edge mask image corresponding to the lighting layer;

[0020] converting the first resolution RGB image from an RGB space to an LAB space to obtain an L channel high frequency image corresponding to the first resolution RGB image;

[0021] based on the first resolution white edge mask image and the L channel high frequency image corresponding to the first resolution RGB image, removing white edges from the first resolution image corresponding to the target image to obtain a first resolution image corresponding to the target image after the white edges are removed.

[0022] In some embodiments, determining a first resolution white edge mask image corresponding to the lighting layer includes:

[0023] determining white edges in an edge of the lighting layer;

[0024] setting values of each pixel point of the white edges to 1, and setting values of each pixel point of non-white edges in the edge of the lighting layer to 0 to obtain a second resolution white edge mask image corresponding to the lighting layer;

[0025] based on the first resolution, performing upscaling processing on the second resolution white edge mask image, and setting pixel points with values greater than 0 obtained after the upscaling processing to 1 to obtain the first resolution white edge mask image corresponding to the lighting layer.

[0026] In some embodiments, determining white edges in an edge of the lighting layer includes:

[0027] determining the edge in the lighting layer;

[0028] determine a luminance difference value of the pixels on both sides of the edge;

[0029] determine the edge as a white edge if the luminance difference value is greater than or equal to a preset luminance threshold.

[0030] In some embodiments, the first resolution image corresponding to the target image is removed of the white edge based on the first resolution white edge mask image and the L channel high frequency image corresponding to the first resolution RGB image, to obtain a first resolution image corresponding to the target image and removed of the white edge, including:

[0031] determine a first resolution non-white edge mask image corresponding to the light map layer based on the first resolution white edge mask image;

[0032] convert the first resolution image corresponding to the target image from the RGB space to the LAB space to obtain an L channel high frequency image corresponding to the first resolution image corresponding to the target image;

[0033] merge the L channel high frequency image corresponding to the first resolution RGB image and the first resolution white edge mask image to obtain a first resolution first synthesis image;

[0034] merge the L channel high frequency image corresponding to the first resolution image corresponding to the target image and the first resolution non-white edge mask image to obtain a first resolution second synthesis image;

[0035] merge the first resolution first synthesis image, the first resolution second synthesis image, and the L channel low frequency image corresponding to the first resolution RGB image to obtain a first resolution L channel image removed of the white edge;

[0036] convert the first resolution L channel image removed of the white edge, the A channel image and the B channel image corresponding to the first resolution RGB image to the RGB space to obtain a first resolution image corresponding to the target image and removed of the white edge.

[0037] In a second aspect, the embodiments of the present disclosure further provide an image rendering device, including:

[0038] a rendering unit configured to respond to a first resolution rendering task of a target image, and perform second resolution rendering on the target image by a renderer, the second resolution being smaller than the first resolution;

[0039] an acquisition unit configured to acquire a plurality of layers from the renderer, the plurality of layers being layers generated by the renderer in the process of the second resolution rendering;

[0040] a processing unit configured to process the plurality of layers based on the first resolution to obtain a first resolution image corresponding to the target image.

[0041] In a third aspect, the embodiments of the present disclosure further provide an electronic device, comprising: a processor and a memory; the processor is configured to execute the steps of the image rendering method according to the first aspect by invoking programs or instructions stored in the memory.

[0042] In a fourth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, wherein the computer-readable storage medium stores programs or instructions, and the programs or instructions cause a computer to execute the steps of the image rendering method according to the first aspect.

[0043] It can be seen that, in at least one embodiment of the present disclosure, for the first resolution rendering task of the target image, the first resolution rendering task is not directly executed by the renderer, but the second resolution rendering is executed by the renderer, and the second resolution is smaller than the first resolution; then, a plurality of layers generated in the second resolution rendering process of the renderer are acquired, and the plurality of layers are processed to obtain the first resolution image corresponding to the target image. Since the time of the second resolution rendering executed by the renderer is at least half shorter than that of the first resolution rendering, and the time of processing the plurality of layers is at least half shorter than that of the second resolution rendering, the time of rendering the first resolution image can be shortened and the user experience can be improved in the embodiments of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0045] Figure 1 An exemplary flowchart of an image rendering method provided by the embodiments of the present disclosure is shown in FIG. 1;

[0046] Figure 2 An exemplary flowchart of processing a plurality of layers provided by the embodiments of the present disclosure is shown in FIG. 2;

[0047] Figure 3 An exemplary flowchart of an image optimization method provided by the embodiments of the present disclosure is shown in FIG. 3;

[0048] Figure 4 An exemplary flowchart of removing white edges provided by the embodiments of the present disclosure is shown in FIG. 4;

[0049] Figure 5 An exemplary flowchart of determining a white edge mask provided by the embodiments of the present disclosure is shown in FIG. 5;

[0050] Figure 6 Another exemplary flowchart of removing white edges provided by the embodiments of the present disclosure is shown in FIG. 6;

[0051] Figure 7 An exemplary block diagram of an image rendering device provided by an embodiment of the present disclosure is shown in FIG. 1.

[0052] Figure 8 A structural schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown in FIG. 2.

[0053] Figure 9 A flowchart of a process for super-resolution reconstruction based on an ESRGAN model is shown in FIG. 3. DETAILED DESCRIPTION

[0054] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. The specific embodiments described herein are merely used to explain the present disclosure, but not to limit the present disclosure. Based on the described embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present disclosure.

[0055] It should be noted that, in this document, relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any such actual relationship or order between such entities or actions.

[0056] At present, for the image rendering task of super-high definition resolution, for example, 4K villa panorama, the renderer performs the 4K image rendering task to obtain the corresponding 4K rendering image, and the rendering time is about 500 seconds. In some schemes of the prior art, the high dynamic image is super-resolution reconstructed to a higher resolution image based on the generative adversarial network, and artifacts are easily generated at the edge of the image. In some schemes of the prior art, the super-high definition resolution reconstruction is performed based on the gradient information of the image, but the effect is not obvious in the small detail part, such as the hair part, which is still relatively blurred.

[0057] In order to reduce the rendering time, an image rendering scheme is provided by an embodiment of the present disclosure, which is for the 4K image rendering task, and instead of directly performing the 4K image rendering task by the renderer, the renderer performs the 2K image rendering, and then obtains a plurality of layers generated in the 2K image rendering process of the renderer, and processes the plurality of layers to obtain the 4K image. Since the time for the renderer to perform the 2K image rendering is about 120 seconds, and the time for processing the plurality of layers is about 30 seconds, the rendering time of the 4K image is about 150 seconds according to the embodiment of the present disclosure, and the rendering time of the 4K image is about 200 seconds in the rare case of full load of the cluster, so the rendering time can be shortened by at least half.

[0058] Figure 1is an exemplary flowchart of an image rendering method provided by an embodiment of the present disclosure, and an execution subject of the image rendering method is an electronic device of any type, such as a portable mobile device like a smartphone, a tablet computer, a notebook computer, and the like, or a fixed device like a desktop computer, a smart television, and the like. The image rendering method can include, but is not limited to, the following steps 101 to 103:

[0059] In step 101, in response to a first resolution rendering task of a target image, a second resolution rendering of the target image is performed by a renderer, and the second resolution is lower than the first resolution.

[0060] The target image can be understood as a to-be-rendered original image, and the original image is an RGB (red, green, and blue color mode) image.

[0061] The first resolution is an ultra-high definition resolution, such as a 4K resolution, an 8K resolution, a 10K resolution, a 12K resolution, and the like. The first resolution rendering task can be understood as a task of rendering the to-be-rendered original image into a first resolution image.

[0062] The second resolution is lower than the first resolution, and the second resolution can be understood as a low resolution compared to the first resolution, such as a 2K resolution.

[0063] In this embodiment, in response to the first resolution rendering task, the first resolution rendering task is not directly performed by the renderer, but the second resolution rendering is performed by the renderer to avoid the problem of long time consumption of directly rendering the first resolution.

[0064] For example, for a 4K rendering task of the original image, if 4K rendering is directly performed, it needs to consume about 500 seconds, and in this embodiment, in response to the 4K rendering task, the original image is not directly rendered by the renderer in 4K, but the original image is rendered by the renderer in 2K, and the time of 2K rendering is about 120 seconds, so the problem of long time consumption of directly rendering the 4K image can be avoided.

[0065] In step 102, a plurality of layers are obtained from the renderer, and the plurality of layers are layers generated by the renderer in the process of the second resolution rendering.

[0066] The renderer generates a plurality of layers in the rendering process, and the plurality of layers include at least one of the following layers: a diffuse reflection layer, an illumination layer, a reflection layer, a reflection intensity layer, a refraction layer, a refraction intensity layer, a highlight layer, a self-luminous layer, and a background layer. The image output by the renderer is an image obtained by the renderer by combining the above plurality of layers.

[0067] For example, VRay (a renderer) generates multiple layers during the rendering process: VRayDiffuseFilter (diffuse reflection layer), VRayRawTotalLighting (lighting layer), VRayRawReflection (reflection layer), VRayReflectionFilter (reflection intensity layer), VRayRawRefraction (refraction layer), VRayRefractionFilter (refraction intensity layer), VRaySpecular (highlight layer), VRaySelfIllumination (self-illumination layer), and VRayBackground (background layer).

[0068] The image output by VRay is combined by the following formula:

[0069] ComposeImage (image output by VRay) = VRayDiffuseFilter (diffuse reflection layer) x VRayRawTotalLighting (lighting layer) + VRayRawReflection (reflection layer) x VRayReflectionFilter (reflection intensity layer) + VRayRawRefraction (refraction layer) x VRayRefractionFilter (refraction intensity layer) + VRaySpecular (highlight layer) + VRaySelfIllumination (self-illumination layer) + VRayBackground (background layer).

[0070] The layers generated by the renderer during the second resolution rendering process and the image output by the renderer are both of the second resolution.

[0071] In step 103, the multiple layers are processed based on the first resolution to obtain a first resolution image corresponding to the target image.

[0072] Since the second resolution rendering is performed by the renderer in step 101, although the problem of long time consumption of directly rendering the first resolution is avoided, both the layers generated by the renderer during the rendering process and the image output by the renderer are of the second resolution, not the first resolution. Therefore, in the embodiment, the multiple layers are processed based on the first resolution to obtain a first resolution image corresponding to the target image.

[0073] In the embodiment, since the time of the second resolution rendering performed by the renderer is shortened by at least half compared with the first resolution rendering, and the time of processing the multiple layers is shortened by at least half compared with the second resolution rendering, the time of rendering the first resolution image can be shortened and the user experience can be improved in the embodiment of the disclosure.

[0074] For example, for a 4K rendering task of an original picture, in this embodiment, the electronic device responds to the 4K rendering task, performs 2K rendering of the original picture through the renderer, the time of 2K rendering is about 120 seconds, and the time of processing a plurality of layers is about 30 seconds, therefore, the 4K image is rendered for about 150 seconds in this embodiment, and the 4K image is rendered for about 200 seconds in the rare case of full load of the cluster, compared with the time of about 500 seconds required for direct 4K rendering, the rendering time is shortened by at least half.

[0075] Figure 2 An exemplary flowchart for processing a plurality of layers is provided in this embodiment of the present disclosure. The plurality of layers are layers generated by the renderer in the rendering process. In some embodiments, the plurality of layers are layers generated by the renderer in the process of rendering at a second resolution (for example, 2K resolution), and these layers are all 2K resolution. The electronic device processes the plurality of layers generated by the renderer in the process of rendering at a second resolution (for example, 2K resolution) based on a first resolution (for example, 4K resolution) to obtain a first resolution image.

[0076] In step 201, each first type layer in the plurality of layers is super-resolution reconstructed to obtain a first resolution layer corresponding to each first type layer.

[0077] The first type layer is a layer whose amount of detail information is greater than or equal to a preset threshold, and the detail information includes but is not limited to texture information and other information reflecting image details. The first type layer includes at least one of the following layers: a diffuse reflection layer, an illumination layer, a reflection layer, and a refraction layer. Taking VRay (a renderer) as an example, the first type layer includes VRayDiffuseFilter (diffuse reflection layer), VRayRawTotalLighting (illumination layer), VRayRawReflection (reflection layer), and VRayRawRefraction (refraction layer).

[0078] Since the layers generated by the renderer are HDR (High-Dynamic Range) layers, in order to reduce the data processing amount, before super-resolution reconstruction of each first type layer, the layer needs to be high-dynamic compressed. High-dynamic compression is a mature technology in the field of image processing, and will not be described again.

[0079] The first type of layer is compressed in high dynamic range, and then super-resolution reconstruction is performed by a super-resolution model. The super-resolution model can be selected from existing different super-resolution models, such as an ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks) model. In this embodiment, super-resolution reconstruction is performed by the super-resolution model, so that the super-resolution layer output by the super-resolution model contains more clear image detail information.

[0080] Since the first type of layer is compressed in high dynamic range and then super-resolution reconstruction is performed by the super-resolution model, high dynamic recovery needs to be performed on the super-resolution layer output by the super-resolution model to obtain the corresponding first resolution layer.

[0081] For example, the renderer generates a 2K resolution (1920x1080) HDR (High-Dynamic Range) light map layer, the electronic device inputs the light map layer into the ESRGAN model for 4 times super-resolution reconstruction after high dynamic compression, and the electronic device performs high dynamic recovery on the 4K resolution (3840x2160) LDR (Low-Dynamic Range) light map layer output by the ESRGAN model to obtain a 4K resolution HDR light map layer.

[0082] Figure 9 A flowchart of a super-resolution reconstruction process based on the ESRGAN model is shown in FIG. 6. In Figure 9 The ESRGAN model includes a plurality of convolution layers (Conv), a plurality of basic blocks, and an upsampling layer (Upsampling), and the connection relationship between the layers is as shown in Figure 9 The functions of the layers can be referred to the related literature of the ESRGAN model, and will not be described in detail.

[0083] Figure 9 In the renderer, the 2K resolution (1920x1080) HDR light map layer is first compressed in high dynamic range to obtain a low resolution (Low Resolution) LDR light map layer, that is, a 2K resolution LDR light map layer; then input into the ESRGAN model for 4 times super-resolution reconstruction, and the ESRGAN model outputs a super-resolution (Super-Resolution) LDR light map layer, that is, a 4K resolution LDR light map layer; and the 4K resolution LDR light map layer is further recovered in high dynamic range to obtain a 4K resolution HDR light map layer.

[0084] In step 202, the second type of layers in the plurality of layers are zoomed to obtain the first resolution layers corresponding to the second type of layers.

[0085] The second type of layers are layers whose amount of detail information is less than a preset threshold. The second type of layers include at least one of the following layers: a reflection intensity layer, a refraction intensity layer, a highlight layer, a self-illumination layer, and a background layer. Taking VRay (a kind of renderer) as an example, the second type of layers include VRayReflectionFilter (reflection intensity layer), VRayRefractionFilter (refraction intensity layer), VRaySpecular (highlight layer), VRaySelfIllumination (self-illumination layer), and VRayBackground (background layer).

[0086] In this embodiment, the second type of layers in the plurality of layers are zoomed (Resize) based on the first resolution to obtain the first resolution layers corresponding to the second type of layers. For example, for a second type of layer with a 2K resolution, it can be zoomed to 4K resolution through Resize.

[0087] In step 203, the first resolution layers corresponding to the first type of layers and the first resolution layers corresponding to the second type of layers are merged to obtain the first resolution image corresponding to the target image.

[0088] Taking VRay (a kind of renderer) as an example, the first resolution image corresponding to the target image is merged by the following formula:

[0089] ComposeImage (the first resolution image corresponding to the target image) = the first resolution layer corresponding to VRayDiffuseFilter (diffuse reflection layer) × the first resolution layer corresponding to VRayRawTotalLighting (lighting layer) + the first resolution layer corresponding to VRayRawReflection (reflection layer) × the first resolution layer corresponding to VRayReflectionFilter (reflection intensity layer) + the first resolution layer corresponding to VRayRawRefraction (refraction layer) × the first resolution layer corresponding to VRayRefractionFilter (refraction intensity layer) + the first resolution layer corresponding to VRaySpecular (highlight layer) + the first resolution layer corresponding to VRaySelfIllumination (self-illumination layer) + the first resolution layer corresponding to VRayBackground (background layer).

[0090] Figure 3An exemplary flowchart of an image optimization method provided by an embodiment of the present disclosure. The method can optimize a first resolution image corresponding to a target image in order to remove white edges in the first resolution image, where the white edges are caused by misalignment of edges of multiple first resolution images when the multiple first resolution images are merged.

[0091] In step 301, a first resolution RGB image is obtained by performing super-resolution reconstruction on the target image.

[0092] In the present embodiment, the way of performing super-resolution reconstruction on the target image is similar to the super-resolution reconstruction mentioned in step 201 in Figure 2 For brevity, the super-resolution reconstruction will not be described again.

[0093] In some embodiments, in order to reduce the amount of data processing, the target image can be compressed in high dynamic range first, and then super-resolution reconstruction is performed to obtain a first resolution RGB image as an LDR (Low-Dynamic Range) image.

[0094] It should be noted that there is no white edge in the hard edge (for example, the edge of a table) in the first resolution RGB image.

[0095] In step 302, the white edge in the first resolution image corresponding to the target image is removed based on the light map layer and the first resolution RGB image to obtain the first resolution image corresponding to the target image after the white edge is removed.

[0096] The inventors of the present scheme found that the white edge appears at the edge with a large luminance difference in the light map layer, and there is no white edge in the first resolution RGB image. Therefore, in the present embodiment, the white edge in the first resolution image corresponding to the target image is removed based on the light map layer and the first resolution RGB image.

[0097] Figure 4 An exemplary flowchart of removing the white edge provided by an embodiment of the present disclosure is applicable to step 302 in Figure 3 As shown in Figure 4 , removing the white edge in the first resolution image corresponding to the target image based on the light map layer and the first resolution RGB image to obtain the first resolution image corresponding to the target image after the white edge is removed includes steps 401 to 403:

[0098] In step 401, a first resolution white edge mask image corresponding to the light map layer is determined.

[0099] The light map layer is a layer generated during the second resolution rendering process of the renderer, and therefore, the resolution of the light map layer is the second resolution.

[0100] The resolution of the white edge mask is the first resolution, and the value of each pixel corresponding to the white edge in the first resolution white edge mask is 1, while the value of each pixel corresponding to the non-white edge is 0.

[0101] In step 402, the first resolution RGB image is converted from RGB space to LAB (a device-independent color model) space to obtain the L (luminance) channel high-frequency image corresponding to the first resolution RGB image.

[0102] The L-channel high-frequency image is a brightness image with large gradient changes, corresponding to the edges, textures, and other details in the first-resolution RGB image. The L-channel low-frequency image corresponds to the non-edge parts of the first-resolution RGB image.

[0103] The L-channel high-frequency image is determined as follows: If the first-resolution RGB image is an HDR (High-Dynamic Range) image, it is compressed to high dynamic range and then converted from RGB space to LAB space; if the first-resolution RGB image is an LDR (Low-Dynamic Range) image, it is directly converted from RGB space to LAB space; after conversion to LAB space, the L-channel image can be separated, and then mean filtering is performed on the L-channel image to obtain the L-channel low-frequency image, and then the L-channel high-frequency image is obtained by subtracting the L-channel low-frequency image from the L-channel image.

[0104] In step 403, based on the first resolution white edge mask and the L channel high-frequency image corresponding to the first resolution RGB image, the white edge is removed from the first resolution image corresponding to the target image to obtain the first resolution image corresponding to the target image after removing the white edge.

[0105] In this embodiment, by separating the brightness image (i.e., the L-channel high-frequency image) and using the first resolution white edge mask, the white edge of the first resolution image corresponding to the target image can be removed, thus obtaining the first resolution image corresponding to the target image after removing the white edge.

[0106] Figure 5 An exemplary process for determining a white-edge mask pattern provided in this disclosure is applicable to... Figure 4 Step 402 in the example. Figure 5 As shown, determining the first resolution white border mask corresponding to the lighting layer includes steps 501 to 503:

[0107] In step 501, the white edges in the lighting layer are determined.

[0108] In this embodiment, first, the edges in the illumination layer are determined; then the brightness difference of the pixels on both sides of the edges is determined; finally, the edges with a brightness difference greater than or equal to a preset brightness threshold are determined as white edges.

[0109] The preset brightness threshold is, for example, 0.6 candela per square meter. Those skilled in the art can set the preset brightness threshold according to actual needs, and the specific value of the preset brightness threshold is not limited in this embodiment.

[0110] In some embodiments, the edges in the illumination layer can be determined by a Laplace edge operator or other edge algorithm.

[0111] In step 502, the value of each pixel point of the white edge is set to 1, and the value of each pixel point of the non-white edge in the edge of the illumination layer is set to 0, to obtain a second-resolution white edge mask graph corresponding to the illumination layer.

[0112] Since the illumination layer is a layer generated during the second-resolution rendering process of the renderer, the resolution of the illumination layer is the second resolution, and thus the resolution of the white edge mask graph determined from the illumination layer is the second resolution.

[0113] In step 503, the second-resolution white edge mask graph is enlarged based on the first resolution, and the pixel points with a value greater than 0 after the enlargement are set to 1, to obtain a first-resolution white edge mask graph corresponding to the illumination layer.

[0114] In this embodiment, after the Resize processing of the second-resolution white edge mask graph based on the first resolution, there are multiple pixel points with a value greater than 0 and less than 1, which are located on both sides of the white edge. In order to improve the reliability of removing the white edge, the pixel points with a value greater than 0 after the enlargement are set to 1, to obtain a first-resolution white edge mask graph corresponding to the illumination layer.

[0115] Figure 6 Another exemplary flowchart for removing the white edge provided by the embodiments of the present disclosure is applicable to step 403 in Figure 4 As shown in Figure 6 , the first-resolution image corresponding to the target image is removed of the white edge based on the first-resolution white edge mask graph and the L-channel high-frequency image corresponding to the first-resolution RGB image, to obtain a first-resolution image after removing the white edge corresponding to the target image, including steps 601 to 606:

[0116] In step 601, based on the first-resolution white edge mask graph, a first-resolution non-white edge mask graph corresponding to the illumination layer is determined.

[0117] Set each pixel point with a value of 1 in the first resolution white border mask map to 0 and set each pixel point with a value of 0 to 1 to obtain a first resolution non-white border mask map corresponding to the light map layer.

[0118] In step 602, the first resolution image corresponding to the target image is converted from the RGB space to the LAB space to obtain an L channel high frequency image corresponding to the first resolution image corresponding to the target image.

[0119] The L channel high frequency image is a luminance image with large gradient changes, and corresponds to edges, textures and other details in the first resolution image. The L channel low frequency image corresponds to non-edge parts in the first resolution image.

[0120] The L channel high frequency image is determined in the following manner: if the first resolution image is an HDR (High-Dynamic Range) image, the first resolution image is compressed in high dynamic range and then converted from the RGB space to the LAB space; if the first resolution image is an LDR (Low-Dynamic Range) image, the first resolution image is directly converted from the RGB space to the LAB space; after conversion to the LAB space, the L channel image can be separated, and then the L channel image is subjected to mean filtering to obtain the L channel low frequency image, so that the L channel high frequency image is obtained by subtracting the L channel low frequency image from the L channel image.

[0121] In step 603, the L channel high frequency image corresponding to the first resolution RGB image and the first resolution white border mask map are merged to obtain a first resolution first synthesis image.

[0122] The first resolution first synthesis image = the L channel high frequency image corresponding to the first resolution RGB image x the first resolution white border mask map.

[0123] As can be seen, there is no white border in the first resolution first synthesis image, that is, the white border is removed using the L channel high frequency image corresponding to the first resolution RGB image.

[0124] In step 604, the L channel high frequency image corresponding to the first resolution image corresponding to the target image and the first resolution non-white border mask map are merged to obtain a first resolution second synthesis image.

[0125] The first resolution second synthesis image = the L channel high frequency image corresponding to the first resolution image corresponding to the target image x the first resolution non-white border mask map.

[0126] As can be seen, in this embodiment, the edges of the non-white border retain the detail information using the L channel high frequency image corresponding to the first resolution image corresponding to the target image.

[0127] In step 605, the first resolution first composite image, the first resolution second composite image and the L channel low frequency image corresponding to the first resolution RGB image are merged to obtain the first resolution L channel image after white border removal.

[0128] The first resolution L channel image after white border removal = the first resolution first composite image + the first resolution second composite image + the L channel low frequency image corresponding to the first resolution RGB image.

[0129] It can be seen that in the embodiment, the white border is removed by using the L channel high frequency image corresponding to the first resolution RGB image, the edge other than the white border is used to retain the detail information by using the L channel high frequency image corresponding to the first resolution image corresponding to the target image, and then the L channel low frequency image corresponding to the first resolution RGB image is added to obtain the first resolution L channel image after white border removal.

[0130] In step 606, the first resolution L channel image after white border removal, the A channel image and the B channel image corresponding to the first resolution RGB image are converted to the RGB space to obtain the first resolution image after white border removal corresponding to the target image.

[0131] If the first resolution image after white border removal corresponding to the target image is an LDR (Low-Dynamic Range) image, high dynamic recovery is performed.

[0132] In some embodiments, the time consumption of 4K rendering for different scenes by using the prior art common 4K rendering scheme (i.e., the renderer directly performs a 4K image rendering task) and the image rendering scheme disclosed in the above embodiments (denoted as super 4K rendering scheme) is shown in the following table:

[0133] Scene name Normal 4k rendering scheme time consumption (seconds) Super-resolution 4k rendering scheme time consumption (seconds) Nezha daytime 3.0 412 145 Luxury villa 3.0 394 191 Villa 3.0 383 200

[0134] It can be seen from the comparison that the time consumption of generating a 4K resolution image based on the image rendering scheme disclosed in the above embodiments is within 200 seconds, and the time consumption of generating a 4K resolution image in the rare case of full load of the cluster is 200 seconds, which shortens the rendering time by at least half.

[0135] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art can understand that the embodiments of the present disclosure are not limited by the action sequence described, because according to the embodiments of the present disclosure, certain steps can be performed in other sequences or simultaneously. In addition, those skilled in the art can understand that the embodiments described in the specification all belong to optional embodiments.

[0136] Figure 7An exemplary block diagram of an image rendering apparatus is provided for embodiments of the present disclosure. The image rendering apparatus can be applied to any type of electronic device, such as a portable mobile device like a smartphone, a tablet computer, a notebook computer, and the like, or a fixed device like a desktop computer, a smart television, and the like. As shown in Figure 7 The image rendering apparatus can include, but is not limited to, the following units: a rendering unit 71, an obtaining unit 72, and a processing unit 73.

[0137] The rendering unit 71 is configured to perform, in response to a first resolution rendering task of a target image, second resolution rendering on the target image by a renderer, the second resolution being smaller than the first resolution.

[0138] The obtaining unit 72 is configured to obtain, from the renderer, a plurality of layers, the plurality of layers being layers generated by the renderer in the process of the second resolution rendering.

[0139] The processing unit 73 is configured to process the plurality of layers based on the first resolution to obtain a first resolution image corresponding to the target image.

[0140] In some embodiments, the processing unit 73 is configured to: perform super-resolution reconstruction on each first-type layer in the plurality of layers to obtain a first resolution layer corresponding to each first-type layer; perform upscaling processing on each second-type layer in the plurality of layers to obtain a first resolution layer corresponding to each second-type layer; and combine the first resolution layers corresponding to the first-type layers and the first resolution layers corresponding to the second-type layers to obtain the first resolution image corresponding to the target image; wherein the first-type layer is a layer with a detail information amount greater than or equal to a preset threshold; and the second-type layer is a layer with a detail information amount less than the preset threshold.

[0141] In some embodiments, the first-type layer includes at least one of the following layers: a diffuse reflection layer, an illumination layer, a reflection layer, and a refraction layer; and the second-type layer includes at least one of the following layers: a reflection intensity layer, a refraction intensity layer, a highlight layer, a self-luminous layer, and a background layer.

[0142] In some embodiments, the image rendering apparatus further includes a reconstruction unit 74 and an optimization unit 75, which are not shown in Figure 7

[0143] The reconstruction unit 74 is configured to perform super-resolution reconstruction on the target image to obtain a first resolution RGB image.

[0144] The optimization unit 75 is configured to remove white edges from the first resolution image corresponding to the target image based on the illumination layer and the first resolution RGB image to obtain a first resolution image corresponding to the target image after the white edges are removed.

[0145] ​In some embodiments, the optimization unit 75 comprises a determining subunit, a converting subunit and a removing subunit:

[0146] The determining subunit is configured to determine a first-resolution white border mask corresponding to the illumination layer.

[0147] The converting subunit is configured to convert the first-resolution RGB image from an RGB space to an LAB space to obtain a first-resolution L-channel high-frequency image corresponding to the first-resolution RGB image.

[0148] The removing subunit is configured to remove a white border from a first-resolution image corresponding to the target image based on the first-resolution white border mask and the first-resolution L-channel high-frequency image corresponding to the first-resolution RGB image, to obtain a first-resolution image corresponding to the target image after the white border is removed.

[0149] In some embodiments, the determining subunit is configured to: determine a white border in an edge of the illumination layer; set a value of each pixel point of the white border to 1 and set a value of each pixel point of a non-white border in the edge of the illumination layer to 0 to obtain a second-resolution white border mask corresponding to the illumination layer; and perform an upscaling process on the second-resolution white border mask based on a first resolution, and set each pixel point with a value greater than 0 obtained after the upscaling process to 1 to obtain the first-resolution white border mask corresponding to the illumination layer.

[0150] In some embodiments, the determining subunit is configured to: determine an edge in the illumination layer; determine a luminance difference value of pixel points on two sides of the edge; and determine the edge with the luminance difference value greater than or equal to a preset luminance threshold as the white border.

[0151] In some embodiments, the removing subunit is configured to:

[0152] determine a first-resolution non-white border mask corresponding to the illumination layer based on the first-resolution white border mask;

[0153] convert the first-resolution image corresponding to the target image from the RGB space to the LAB space to obtain a first-resolution L-channel high-frequency image corresponding to the target image;

[0154] merge the first-resolution L-channel high-frequency image corresponding to the first-resolution RGB image and the first-resolution white border mask to obtain a first-resolution first synthesis image;

[0155] merge the first-resolution L-channel high-frequency image corresponding to the target image and the first-resolution non-white border mask to obtain a first-resolution second synthesis image;

[0156] merge the first-resolution first synthesis image, the first-resolution second synthesis image and a first-resolution L-channel low-frequency image corresponding to the first-resolution RGB image to obtain a first-resolution L-channel image after the white border is removed.

[0157] The first resolution L channel image after white border removal, the A channel image and the B channel image corresponding to the first resolution RGB image are converted to the RGB space to obtain the first resolution image after white border removal corresponding to the target image.

[0158] The details of the above disclosed embodiments of the image rendering device can refer to the details of the above disclosed embodiments of the image processing method, and will not be repeated here.

[0159] In some embodiments, the division of each unit in each of the above device embodiments is only a logical functional division, and another division mode can be used in actual implementation, for example, at least two units can be implemented as one unit; each unit can also be divided into multiple sub-units. It can be understood that each unit or sub-unit can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application.

[0160] Figure 8 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. The electronic device is, for example, a portable mobile device such as a smartphone, a tablet computer, a notebook computer, and the like, and is, for example, a fixed device such as a desktop computer, a smart television, and the like.

[0161] As shown in Figure 8 , the electronic device includes at least one processor 801, at least one memory 802, and at least one communication interface 803. Each component in the electronic device is coupled together through a bus system 804. The communication interface 803 is used for information transmission between the external device. Understandably, the bus system 804 is used to realize the connection communication between the components. In addition to including a data bus, the bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 804 in the Figure 8 .

[0162] It can be understood that the memory 802 in the present embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0163] In some embodiments, the memory 802 stores the following elements, executable units or data structures, or a subset of them, or an extended set of them: an operating system and an application program.

[0164] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic tasks and processing hardware-based tasks. The application programs include various application programs, such as a media player (Media Player), a browser (Browser), and the like, for implementing various application tasks. The program for implementing the image rendering method provided in the embodiments of the present disclosure can be included in the application programs.

[0165] In the embodiments of the present disclosure, the processor 801 executes the program or instruction stored in the memory 802, specifically, the program or instruction stored in the application program, to implement the steps of the embodiments of the image rendering method provided in the present disclosure.

[0166] The image rendering method provided in the embodiments of the present disclosure can be applied to the processor 801 or implemented by the processor 801. The processor 801 can be an integrated circuit chip having a processing capability of signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware or the instruction in the form of software in the processor 801. The processor 801 mentioned above can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general processor can be a microprocessor or the processor can also be any conventional processor.

[0167] The steps of the image rendering method provided in the embodiments of the present disclosure can be directly embodied as a hardware decoding processor for execution, or executed by a combination of hardware and software units in the decoding processor. The software unit can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register or other mature storage media in the art. The storage media is located in the memory 802, and the processor 801 reads the information in the memory 802 and completes the steps of the method in combination with the hardware thereof.

[0168] The embodiments of the present disclosure also provide a computer readable storage medium storing programs or instructions, which cause a computer to execute the steps of the embodiments of the image rendering method. To avoid repeated description, the description is not repeated here. In some embodiments, the computer readable storage medium is a non-transitory computer readable storage medium.

[0169] The embodiments of the present disclosure also provide a computer program product, which comprises a computer program stored in a non-transitory computer readable storage medium, and at least one processor of a computer reads and executes the computer program from the storage medium, so that the computer executes the steps of the image rendering method embodiments. To avoid repeated description, the details are not described here.

[0170] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or apparatus that includes a list of elements not only includes those elements, but also includes other elements not expressly listed, or inherent to such a process, method, article, or apparatus. Without more limitations, the elements defined by the statement "comprise" do not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0171] Those skilled in the art can understand that although some embodiments described herein include certain features included in other embodiments rather than other features, the combination of features of different embodiments means within the scope of the present disclosure and forms different embodiments.

[0172] Those skilled in the art can understand that the description of each embodiment is focused, and the parts not described in detail in a certain embodiment can refer to the relevant description of other embodiments.

[0173] Although the embodiments of the present disclosure are described in conjunction with the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present disclosure, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. An image rendering method, the method comprising: Step 1, in response to a first resolution rendering task of a target image, performing second resolution rendering on the target image by a renderer, the second resolution being smaller than the first resolution; Step 2, obtaining a plurality of layers from the renderer, the plurality of layers being layers generated by the renderer in the process of the second resolution rendering, the plurality of layers comprising the following layers: diffuse reflection layer, illumination layer, reflection layer, reflection intensity layer, refraction layer, refraction intensity layer, highlight layer, self-luminous layer and background layer, the image output by the renderer being an image obtained by the renderer combining the plurality of layers, the output image being combined by the following formula: output image = diffuse reflection layer x illumination layer + reflection layer x reflection intensity layer + refraction layer x refraction intensity layer + highlight layer + self-luminous layer + background layer, the layers generated by the renderer in the process of the second resolution rendering and the image output by the renderer all being the second resolution; Step 3, processing the plurality of layers based on the first resolution to obtain a first resolution image corresponding to the target image, comprising: performing super-resolution reconstruction on each first type layer in the plurality of layers to obtain a first resolution layer corresponding to each first type layer; performing magnification processing on each second type layer in the plurality of layers to obtain a first resolution layer corresponding to each second type layer; combining the first resolution layer corresponding to each first type layer and the first resolution layer corresponding to each second type layer to obtain the first resolution image corresponding to the target image; wherein the first type layer is a layer with a detail information amount greater than or equal to a preset threshold, the first type layer comprising the following layers: diffuse reflection layer, illumination layer, reflection layer and refraction layer; the second type layer is a layer with a detail information amount less than the preset threshold, the second type layer comprising the following layers: reflection intensity layer, refraction intensity layer, highlight layer, self-luminous layer and background layer; Step 4, performing super-resolution reconstruction on the target image to obtain a first resolution RGB image, wherein the first resolution RGB image is an LDR image; based on the illumination layer and the first resolution RGB image, removing white edges from the first resolution image corresponding to the target image to obtain the first resolution image corresponding to the target image after removing white edges. The removing white edges from the first resolution image corresponding to the target image based on the illumination layer and the first resolution RGB image to obtain the first resolution image corresponding to the target image after removing white edges comprises:

2. The method of claim 1, wherein, determining a first resolution white edge mask corresponding to the illumination layer; converting the first resolution RGB image from RGB space to LAB space to obtain an L channel high frequency image corresponding to the first resolution RGB image; ​ remove the white border from the first resolution image corresponding to the target image based on the first resolution white border mask image and the L channel high frequency image corresponding to the first resolution RGB image, to obtain the first resolution image after removing the white border corresponding to the target image.

3. The method of claim 2, wherein, The determining the first resolution white border mask image corresponding to the illumination layer includes: determining the white border in the edge of the illumination layer; setting the value of each pixel point of the white border to 1, and setting the value of each pixel point of the non-white border in the edge of the illumination layer to 0, to obtain the second resolution white border mask image corresponding to the illumination layer; based on the first resolution, the second resolution white border mask image is enlarged, and the pixel point with a value greater than 0 obtained after the enlargement is set to 1, to obtain the first resolution white border mask image corresponding to the illumination layer.

4. The method of claim 3, wherein, The determining the white border in the edge of the illumination layer includes: determining the edge in the illumination layer; determining the luminance difference of the pixel points on both sides of the edge; determining the edge with the luminance difference greater than or equal to the preset luminance threshold as the white border.

5. The method of claim 2, wherein, The removing the white border from the first resolution image corresponding to the target image based on the first resolution white border mask image and the L channel high frequency image corresponding to the first resolution RGB image includes: determining the first resolution non-white border mask image corresponding to the illumination layer based on the first resolution white border mask image; converting the first resolution image corresponding to the target image from the RGB space to the LAB space, to obtain the L channel high frequency image corresponding to the first resolution image corresponding to the target image; merging the L channel high frequency image corresponding to the first resolution RGB image and the first resolution white border mask image, to obtain the first resolution first synthesis image; merging the L channel high frequency image corresponding to the first resolution image corresponding to the target image and the first resolution non-white border mask image, to obtain the first resolution second synthesis image; merging the first resolution first synthesis image, the first resolution second synthesis image, and the L channel low frequency image corresponding to the first resolution RGB image, to obtain the first resolution L channel image after removing the white border; converting the first resolution L channel image after removing the white border, the A channel image and the B channel image corresponding to the first resolution RGB image to the RGB space, to obtain the first resolution image after removing the white border corresponding to the target image.

6. An image rendering device, the device comprising: a rendering unit configured to, in response to a first resolution rendering task of a target image, render the target image at a second resolution by a renderer, the second resolution being smaller than the first resolution; The acquisition unit is configured to acquire a plurality of layers from the renderer, the plurality of layers being layers generated by the renderer in the process of rendering at the second resolution, the plurality of layers including the following layers: diffuse reflection layer, illumination layer, reflection layer, reflection intensity layer, refraction layer, refraction intensity layer, highlight layer, self-luminous layer, and background layer, the image output by the renderer being an image obtained by the renderer by combining the plurality of layers, and the image output by the renderer being combined by the following formula: Output image = diffuse reflection layer x illumination layer + reflection layer x reflection intensity layer + refraction layer x refraction intensity layer + highlight layer + self-luminous layer + background layer, the layers generated by the renderer in the process of rendering at the second resolution and the image output by the renderer both being at the second resolution. The processing unit is configured to process the plurality of layers based on the first resolution to obtain a first resolution image corresponding to the target image, including: performing super-resolution reconstruction on each first-type layer in the plurality of layers to obtain a first resolution layer corresponding to each first-type layer; performing magnification processing on each second-type layer in the plurality of layers to obtain a first resolution layer corresponding to each second-type layer; combining the first resolution layer corresponding to each first-type layer and the first resolution layer corresponding to each second-type layer to obtain the first resolution image corresponding to the target image; wherein the first-type layer is a layer with a detail information amount greater than or equal to a preset threshold, and the first-type layer includes the following layers: diffuse reflection layer, illumination layer, reflection layer, and refraction layer; and the second-type layer is a layer with a detail information amount less than the preset threshold, and the second-type layer includes the following layers: reflection intensity layer, refraction intensity layer, highlight layer, self-luminous layer, and background layer. The white border removal unit is configured to perform super-resolution reconstruction on the target image to obtain a first resolution RGB image, wherein the first resolution RGB image is an LDR image; and remove white borders from the first resolution image corresponding to the target image based on the illumination layer and the first resolution RGB image to obtain the first resolution image corresponding to the target image after the white borders are removed.

7. An electronic device, comprising: comprising: a processor and a memory; the processor is configured to execute the steps of the image rendering method according to any one of claims 1 to 5 by invoking programs or instructions stored in the memory.

8. A computer readable storage medium, wherein, The computer readable storage medium stores programs or instructions, which cause the computer to execute the steps of the image rendering method according to any one of claims 1 to 5.

Images