Efficient rendering method based on scene-aware mapping for complex scenes

Abstract

This disclosure presents an efficient rendering method based on scene-aware mapping for complex scenes. One specific implementation of this method includes acquiring a scene model, camera parameters, and user gaze point information corresponding to the target scene; performing spatial mapping processing on the scene model based on the camera parameters to obtain a two-dimensional geometric scene graph; generating a consistent visual importance graph based on the user gaze point information and the two-dimensional geometric scene graph; generating a pixel size control graph based on the two-dimensional geometric scene graph and the consistent visual importance graph; and rendering the two-dimensional geometric scene graph based on the pixel size control graph to obtain a rendered scene image. This implementation can reduce the computational resources required for rendering and improve rendering speed while ensuring the quality of human visual perception.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the fields of computer graphics and virtual reality, and specifically to a method for efficient rendering of complex scenes based on scene-aware mapping. Background Technology

[0002] Virtual reality technology is widely used in entertainment games, cultural tourism, manufacturing and other fields, which places higher demands on the realism and efficiency of virtual reality scene rendering. Currently, the common methods for video image rendering are: for each video image, a foveated rendering method based on rasterization and ray tracing is used, which uses more ray samples for each pixel in the foveated region and fewer ray samples for each pixel in the surrounding region; or a foveated rendering method based on mapping is used, which constructs mapping and inverse mapping functions according to the characteristics of the human visual system and renders the image through a deferred rendering pipeline.

[0003] However, in practice, it has been found that the following technical problems often arise when using the above method for image rendering:

[0004] First, because the rendering process only considers the characteristics of the human visual system and ignores the characteristics of the scene itself, the rendering quality of the salient area outside the gaze point is poor, resulting in poor overall image rendering quality.

[0005] Second, if there are a large number of pixels to be rendered in the video image, a lot of time needs to be spent on lighting processing for each pixel during the rendering process, which results in a long rendering time and low rendering efficiency.

[0006] The information disclosed in this background section is only intended to enhance the understanding of the background of the present disclosure concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0008] Some embodiments of this disclosure propose an efficient rendering method based on scene-aware mapping for complex scenes, in order to solve one or more of the technical problems mentioned in the background section above.

[0009] Some embodiments of this disclosure provide an efficient rendering method based on scene-aware mapping for complex scenes. The method includes: acquiring a scene model, camera parameters, and user gaze point information corresponding to the target scene; performing spatial mapping processing on the scene model based on the camera parameters to obtain a two-dimensional geometric scene map; generating a consistent visual importance map based on the user gaze point information and the two-dimensional geometric scene map; generating a pixel size control map based on the two-dimensional geometric scene map and the consistent visual importance map; and rendering the two-dimensional geometric scene map based on the pixel size control map to obtain a scene rendering image.

[0010] The embodiments of this disclosure have the following beneficial effects: The efficient rendering method based on scene-aware mapping for complex scenes, as described in some embodiments of this disclosure, can improve the quality of image rendering and human visual perception. Specifically, the reason for the overall poor quality of image rendering results is that, during the rendering process, only the characteristics of the human visual system are considered, ignoring the characteristics of the scene itself, resulting in poor rendering quality of the periphery of the gaze point area. Based on this, the efficient rendering method based on scene-aware mapping for complex scenes, as described in some embodiments of this disclosure, firstly, acquires the scene model, camera parameters, and user gaze point information corresponding to the target scene. This facilitates subsequent scene image rendering of the target scene based on the user's visual characteristics. Secondly, based on the aforementioned camera parameters, the aforementioned scene model undergoes spatial mapping processing to obtain a two-dimensional geometric scene map. This allows the perception features of the target scene to be obtained. Then, based on the aforementioned user gaze point information and the aforementioned two-dimensional geometric scene map, a consistent visual importance map is generated. This allows the visual importance of each pixel in the scene in the current state to be determined. Finally, based on the aforementioned two-dimensional geometric scene map and the aforementioned consistent visual importance map, a pixel size control map is generated. Therefore, a mapping relationship between high-resolution and low-resolution images can be constructed based on scene-aware features and user visual features. Finally, based on the aforementioned pixel size control map, the aforementioned two-dimensional geometric scene map is rendered to obtain a scene-rendered image. Thus, during image rendering, each pixel in the image can be mapped and rendered according to the size change of each pixel in the pixel size control map. Therefore, the efficient rendering method based on scene-aware mapping for complex scenes in some embodiments of this disclosure, by combining two-dimensional geometric scene information and user gaze point information when generating a consistent visual importance map, can make the visual importance of each pixel in the scene more accurate, especially improving the visual importance of salient regions surrounding the gaze point region. Furthermore, by constructing a pixel size control map based on visual importance, it is convenient to render the image according to the visual importance corresponding to each pixel in the target scene. Therefore, the rendering quality can be improved for salient regions surrounding the gaze point region. This improves the overall quality of the image rendering result and the user's visual perception quality. Furthermore, because appropriate computing resources can be allocated to each pixel for rendering according to the pixel size control map, the waste of computing resources can be reduced. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0012] Figure 1This is a flowchart of some embodiments of the efficient rendering method based on scene-aware mapping for complex scenes according to the present disclosure;

[0013] Figure 2 This is a schematic diagram illustrating the pixel coordinate changes during the stretching of logarithmic polar coordinate mapping into rectangular mapping, according to the efficient rendering method based on scene-aware mapping for complex scenes disclosed herein. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1 A flow 100 of some embodiments of an efficient rendering method based on scene-aware mapping for complex scenes according to the present disclosure is shown. This efficient rendering method based on scene-aware mapping for complex scenes includes the following steps:

[0021] Step 101: Obtain the scene model, camera parameters, and user gaze point information corresponding to the target scene.

[0022] In some embodiments, the execution entity (e.g., a virtual reality device) of an efficient rendering method based on scene-aware mapping for complex scenes can acquire scene models, camera parameters, and user gaze point information corresponding to the target scene via wired or wireless connections. The complex scene can be a scene containing numerous objects with complex features. These complex features may include, but are not limited to, complex textures or complex lighting. The target scene can be a complex scene requiring scene image rendering. The scene image can be a two-dimensional image used to display the corresponding scene. The scene model can be a three-dimensional mesh model with lighting constructed based on the target scene using a preset modeling tool. The camera parameters can be the parameters of the camera set when capturing a single frame image of the target scene. Camera parameters may include, but are not limited to, field of view and focal length. The user gaze point information can be the position coordinates of the user's gaze point on the unrendered captured image, obtained through an eye-tracking device.

[0023] As an example, the aforementioned execution entity can obtain a scene model from a file database, camera parameters from the video camera, and user gaze information from an eye-tracking device before rendering images of the target scene captured by the video camera. The eye-tracking device can be a device used to measure and record eye position and movement. For example, the eye-tracking device can be an eye tracker.

[0024] It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other currently known or future wireless connection methods.

[0025] Step 102: Based on the camera parameters, perform spatial mapping processing on the scene model to obtain a two-dimensional geometric scene map.

[0026] In some embodiments, the execution entity can perform spatial mapping processing on the scene model based on the camera parameters to obtain a two-dimensional geometric scene image. The two-dimensional geometric scene image can be a two-dimensional image including scene depth information, normal vector information, and reflectivity information of the target scene. Each pixel in the two-dimensional geometric scene image can include three channels. The three channels can correspond to the scene depth information, normal vector information, and reflectivity information, respectively. The scene depth information can represent the distance from an object or point in the scene to the observation point (camera lens). The normal vector information can represent the orientation of an object at the corresponding pixel. The reflectivity information can represent the reflectivity of an object at the corresponding pixel. The two-dimensional geometric scene image can be obtained by performing spatial mapping processing on the scene model based on the camera parameters through the following steps:

[0027] The first step is to perform an MVP transformation on the aforementioned scene model to obtain the scene model to be mapped. The MVP (Model-View-Projection) transformation can include model transformation, view transformation, and projection transformation. The scene model to be mapped can be a scene model in clip space that needs to be mapped to a 2D screen.

[0028] The second step is to perform viewport transformation on the above-mentioned scene model to be mapped to obtain a two-dimensional geometric scene graph.

[0029] Step 103: Generate a consistent visual importance map based on user gaze point information and two-dimensional geometric scene graph.

[0030] In some embodiments, the aforementioned execution entity can generate a consistent visual importance map in various ways based on the aforementioned user gaze point information and the aforementioned two-dimensional geometric scene graph. The consistent visual importance map can be a visual importance map with pixel values ​​between 0 and 1. The visual importance map can have the same resolution as the aforementioned two-dimensional geometric scene graph. Each pixel value in the visual importance map can represent the intensity of visual importance. The visual importance map can represent the intensity distribution of visual importance corresponding to each pixel in the two-dimensional geometric scene graph. Visual importance can be the importance of visual features to human visual perception. The aforementioned visual features can include, but are not limited to, image contrast, saliency, edges, contours, and highlights. It should be noted that since computational resources are fixed, the sum of all pixel values ​​in the aforementioned visual importance map is also a fixed value.

[0031] In some optional implementations of certain embodiments, the aforementioned execution entity may generate a consistent visual importance map based on the aforementioned user gaze point information and the aforementioned two-dimensional geometric scene graph through the following steps:

[0032] The first step is to extract visual features from the aforementioned two-dimensional geometric scene image to obtain a visual feature map. The visual feature map has the same resolution as the aforementioned two-dimensional geometric scene image. Each pixel value in the visual feature map represents the intensity of the visual feature. Each pixel value can be a value between 0 and 1. The visual feature map represents the intensity distribution of the visual features corresponding to each pixel in the two-dimensional geometric scene image. Specifically, the following steps can be performed:

[0033] The first sub-step involves performing saliency detection on the aforementioned two-dimensional geometric scene graph using a pre-defined visual saliency detection method to obtain a first feature map. This first feature map characterizes the distribution of saliency levels among the pixels in the two-dimensional geometric scene graph. The saliency level of a pixel can be considered as the degree to which it attracts visual attention. For example, the visual saliency detection method can be, but is not limited to, one of the following: GR (Graph Regularized) algorithm or CA (Context-Aware) algorithm.

[0034] The second sub-step involves performing contour detection on the aforementioned two-dimensional geometric scene image using a preset contour detection method to obtain a second feature map. This second feature map can be a binary image displaying the contours of various geometric objects in the two-dimensional geometric scene image. For example, the contour detection method can be, but is not limited to, one of the following: Hough transform, or edge-tracking-based contour detection.

[0035] The third sub-step involves performing specular detection on the aforementioned two-dimensional geometric scene image using a preset specular detection method to obtain a third feature map. This third feature map characterizes the distribution of specular regions in the two-dimensional geometric scene image. For example, the specular detection method can be, but is not limited to, one of the following: thresholding or histogram methods.

[0036] The fourth sub-step involves fusing the first, second, and third feature maps using a perceptual filtering method to obtain a visual feature map. Specifically, the perceptual filtering method can be used to select the maximum value from the corresponding pixel positions in the first, second, and third feature maps for each pixel position in the initial visual feature map, and to determine the updated initial visual feature map as the final visual feature map. The initial visual feature map can be initially set as a feature map where all pixel values ​​are 0.

[0037] The second step is to generate a visual acuity map corresponding to the aforementioned two-dimensional geometric scene map based on the user gaze point information. This visual acuity map characterizes the distribution of visual acuity for each pixel in the two-dimensional geometric scene map. Visual acuity represents the importance of a pixel in an image. Visual acuity can be represented by pixel values ​​between 0 and 1, measured in terms of minimum resolvable angle. The minimum resolvable angle is the smallest angle a person can distinguish under certain conditions (e.g., distance, brightness). The visual acuity in the visual acuity map can be generated using the following formula:

[0038] a(p) = w0 + m × e(p, g).

[0039] Where a(·) represents a function that takes a pixel as input and outputs the corresponding visual acuity. p represents a pixel in the 2D geometric scene graph. g represents the user's gaze point. a(p) represents the visual acuity corresponding to pixel p in the visual acuity graph. w0 represents the minimum resolvable angle of the visual acuity value. m represents the visual acuity slope. e(·) represents the eccentricity function. e(p, g) represents the eccentricity of pixel p relative to the user's gaze point g. Eccentricity can characterize the angular distance from a pixel to the user's gaze point.

[0040] The third step is to generate a visual importance map based on the aforementioned visual feature map and visual acuity map. Visual acuity in the visual acuity map can represent the human ability to accurately identify minute details. Measured by the minimum resolvable angle, the higher the visual acuity, the larger the smallest resolvable detail, and the smaller the visual importance of details of the same size to the human visual system. Visual features can represent the ability or intensity of a scene to attract people's attention. The higher the value of a visual feature, the more attention the corresponding pixel attracts, and the greater the visual importance of these pixels to the human visual system. Therefore, visual importance is positively correlated with visual features. Thus, the pixel values ​​in the visual importance map can be generated using the following formula, based on the aforementioned visual feature map and visual acuity map:

[0041]

[0042] Here, vi(·) represents a function that takes a pixel as input and outputs the pixel value representing the visual importance of that pixel. vi(p) represents the pixel value corresponding to the visual importance of pixel p. v represents the motion velocity of the user's gaze point, i.e., the positional difference between two user gaze points corresponding to two frames. G(·) represents a linear function that takes the motion velocity of the user's gaze point as input and outputs a correction coefficient for visual importance. f(·) represents a function that takes a pixel as input and outputs the pixel value representing the visual feature of that pixel. f(p) represents the value of the visual feature corresponding to pixel p. k represents a predefined weighting coefficient. The correspondence data between f(p) and vi(p) can be collected, and k can be obtained through data fitting.

[0043] The fourth step is to normalize the visual importance of each pixel in the above visual importance map to obtain a consistent visual importance map. The pixel values ​​in the consistent visual importance map can be generated using the following formula:

[0044] NorVi(p)=h×w×vi(p) / ∑vi.

[0045] Where NorVi(·) represents the normalization function that normalizes the pixel values ​​in the visual importance map. NorVi(p) represents the result of normalizing the pixel value corresponding to pixel point p in the visual importance map. h represents the height of the down-resolution image. The down-resolution image can be an image obtained by downsampling the above two-dimensional geometric scene map. w represents the width of the down-resolution image. ∑vi represents the sum of all pixel values ​​in the visual importance map.

[0046] Optionally, each pixel in the aforementioned two-dimensional geometric scene graph can correspond one-to-one with a mapped pixel in the mapped pixel set. The mapped pixels in the aforementioned mapped pixel set can be the result of mapping pixels in the two-dimensional geometric scene graph from one Cartesian coordinate system to another. In practice, since the logarithmic polar coordinate mapping method usually suffers from low space utilization efficiency after mapping, this disclosure employs a method of stretching the logarithmic polar coordinate mapping into a rectangular mapping to determine the mapped pixel corresponding to each pixel in the two-dimensional geometric scene graph.

[0047] As an example, Figure 2 This diagram illustrates the pixel coordinate changes during the stretching of logarithmic polar coordinate mapping into rectangular mapping in the efficient scene-aware mapping method for complex scenes disclosed in this disclosure. Figure 2 The diagram includes six sub-graphs: a, b, c, d, e, and f. Sub-graph a shows a 7x7 resolution 2D geometric scene, where the top-left corner (0, 0) represents the user's gaze point, and dark blocks represent visual features. Sub-graph b shows a consistent visual importance map corresponding to the aforementioned 2D geometric scene, where lighter pixel colors indicate higher visual importance. Pixels closer to the gaze point and visual features have higher visual importance. Sub-graph c shows a 2D geometric scene using different colors to represent each pixel for clearer observation. For logarithmic polar coordinate mapping, the mapping function typically transforms a square region into a near-circular region in polar coordinates. This disclosure approximates logarithmic polar coordinate mapping as a transformation from rectangular coordinates to rectangular coordinates to ensure that the transformed regions have the same area. Sub-graph d shows the area of ​​each mapped pixel calculated using the logarithmic polar coordinate transformation. Black areas in sub-graph d represent regions to which no pixels are mapped. Next, this disclosure stretches the circular regions in sub-graph d into square regions as in sub-graph e to fully utilize the buffer. Since the mapping function between the c-subgraph and the e-subgraph needs to be injective and surjective, it must satisfy the following two constraints:

[0048] Constraint 1: For a square region, its mapped region remains square (surjective). Specifically, for constraint 1, the following formula can be used to map the subgraph of length c to a region with side length s. i The square region is mapped to the subgraph e with side length s′ iSquare area:

[0049]

[0050] In the above formula, 'i' represents the number of pixels corresponding to one side of the square region in subimage 'c'. i This represents the side length of the square region in subgraph c. s i represents the side length of the square region in the sub-image e. a(e) represents the visual sensitivity of any pixel. e in a(e) represents the eccentricity. sqrt(·) represents the square root function. dx represents the derivative with respect to the independent variable x. dy represents the derivative with respect to the independent variable y. The independent variables x and y are the x-coordinate and y-coordinate of the lower right corner of a pixel in the two-dimensional geometric scene image, respectively.

[0051] Constraint 2: The adjacency relationship between pixels within a square region and on the edges of the square remains unchanged (injective). Specifically, for constraint 2, all pixels are adjacent when pixels on the edges of a square are adjacent. In practice, pixel adjacency can be ensured by calculating the lengths of the four sides of the mapped pixel. Specifically, starting with a square with a side length of 1, and continuing up to the maximum side length, if each square maintains adjacency on its edges, then all squares will maintain adjacency. Therefore, a pixel may not be a rectangle, but rather resemble a parallelogram or trapezoid.

[0052] In practice, taking a dark pixel in subimage c as an example, the top-left corner is determined by its top-left pixel, the top-right corner by its top pixel, and the bottom-left corner by its left pixel. Therefore, it is only necessary to calculate the mapped coordinates of the bottom-right corner of this pixel. The mapped coordinates can be the coordinates of the mapped point after the bottom-right corner of the pixel is mapped to the transformed region. The ordinate in the mapped coordinates is equal to the side length of the square region in subimage e. The abscissa in the mapped coordinates can be determined by the ratio of the sum of the visual acuity of all pixels to the left of this point (i.e., the bottom-right corner of the dark pixel) to the sum of the visual acuity of all pixels at the edge of the square region. The abscissa in the mapped coordinates can be generated using the following formula:

[0053]

[0054] Furthermore, the equation for the pixels on the right edge of the square can be derived in a similar way. For simplicity, the gaze point is set to (0, 0), and only the mathematical derivation process for the pixels located to the lower right of the gaze point is listed. It should be noted that when the gaze point coordinates are not (0, 0), the equations for the remaining pixels can also be derived in a similar way. Therefore, for each pixel to the lower right of the gaze point in the above two-dimensional geometric scene diagram, the corresponding mapped pixel can be generated using the following formula:

[0055]

[0056] Where u(x, y) and v(x, y) represent the horizontal and vertical coordinates of the mapped pixel (u, v) corresponding to the pixel (x, y) in the two-dimensional geometric scene graph, respectively.

[0057] Step 104: Generate a pixel size control map based on the two-dimensional geometric scene map and the consistent visual importance map.

[0058] In some embodiments, the execution entity described above can generate a pixel size control map based on the two-dimensional geometric scene graph and the consistent visual importance graph in various ways. The pixel size control map can be a two-dimensional graph of the same size as the consistent visual importance graph. Each pixel in the pixel size control map can include two components, representing the pixel's vertical and horizontal dimensions, respectively. The pixel size control map can be used to allocate the required computational resources according to the size of each pixel during image rendering.

[0059] In some optional implementations of certain embodiments, the aforementioned execution entity may generate a pixel size control map based on the aforementioned two-dimensional geometric scene graph and the aforementioned consistent visual importance graph through the following steps:

[0060] The first step is to perform the following steps for each mapped pixel in the mapped pixel set corresponding to the above two-dimensional geometric scene graph:

[0061] The first sub-step involves determining the initial pixel scale information corresponding to the mapped pixels. This initial pixel scale information may include initial pixel width and initial pixel height values. The initial pixel width can be the horizontal width of the mapped pixel before optimization. The initial pixel height can be the vertical height of the mapped pixel before optimization. It should be noted that the initial pixel scale information is the same for each mapped pixel. The initial pixel scale information corresponding to the mapped pixels can be determined using the following formula:

[0062]

[0063] Where Δu(x, y) represents the horizontal change between the two mapped coordinates corresponding to pixels (x, y) and (x-1, y), i.e., the initial pixel width value of the mapped pixel. Δv(x, y) represents the vertical change between the two mapped coordinates corresponding to pixels (x, y) and (x, y-1), i.e., the initial pixel height value of the mapped pixel. u(x-1, y) represents the horizontal coordinate of the mapped pixel corresponding to pixel (x-1, y) in the 2D geometric scene graph. v(x, y-1) represents the vertical coordinate of the mapped pixel corresponding to pixel (x, y-1) in the 2D geometric scene graph. Let Δx represent the partial derivative of u with respect to x. Δx represents the change in the pixel's x-coordinate. Let Δy represent the partial derivative of v with respect to y. Δy represents the change in the pixel's ordinate.

[0064] Optionally, the execution entity can represent the pixels (x, y) in the above two-dimensional geometric scene diagram according to the corresponding mapped pixels (u, v) using the following formula:

[0065]

[0066] Where exp(·) represents the natural exponential function. x(u,v) and y(u,v) represent the x and y coordinates of a pixel (x, y) in the two-dimensional geometric scene graph, respectively.

[0067] The second sub-step involves determining the visual importance metric value corresponding to the mapped pixels based on the aforementioned consistent visual importance map. This visual importance metric value can be a pixel value in the consistent visual importance map. First, the pixels in the aforementioned two-dimensional geometric scene graph corresponding to the mapped pixels are identified as target pixels. Then, pixels that match the target pixels are selected from the pixels included in the consistent visual importance map. Matching the target pixel can mean that the pixel in the consistent visual importance map has the same position coordinates as the target pixel. Finally, the pixel value corresponding to the selected pixel is determined as the visual importance metric value.

[0068] The third sub-step involves performing scale optimization processing on the initial pixel scale information based on the aforementioned visual importance metric, resulting in optimized pixel scale information. This optimized pixel scale information includes optimized pixel width and optimized pixel height values. The optimized pixel width can be the optimized initial pixel width, and the optimized pixel height can be the optimized initial pixel height. The optimized pixel width and optimized pixel height values ​​can be generated using the following formula:

[0069]

[0070] Where, Δ(u / v) vi In this context, vi represents the visual importance metric corresponding to the mapped pixel (u, v). To simplify the formula, Δu is... vi or Δv vi The formulas are combined and expressed. Δ(u / v) represents Δu or Δv. When the above formula is Δuvi, In fact When the above expression is Δuvi In fact Δu is the shorthand form of Δu(x, y). Δv is the shorthand form of Δv(x, y). area represents the area of ​​the mapped pixel (u, v).

[0071] In practice, the size of a mapped pixel is related not only to its own visual importance but also to the size of its neighboring pixels. Considering that mapped pixels must satisfy the adjacency requirement of constraint 2 above, changing the size of only one pixel would lead to overlapping of adjacent pixels or gaps between pixels. Therefore, to change the size of a pixel and its neighboring pixels, it is necessary to traverse all pixels and determine the size of the mapped pixel and the horizontal and vertical variations of its neighboring pixels. Furthermore, since the "focus of attention" is typically assumed to decay monotonically around its center, this disclosure decides to use a logarithmic function to control the size variation of neighboring pixels, rather than uniformly scaling the size of neighboring pixels. Specifically, based on Δ(u / v)... vi Proportional to ln(1+dis), where dis is the horizontal and vertical distance between adjacent pixels and the mapped pixel at the center, the pixel size change of any adjacent pixel can be determined by the following formula:

[0072]

[0073] Where Δu / Δv represents Δu or Δv. To simplify the formula, cΔu / Δv(i,j) represents c Δu (i, j) or c Δv (i, j). c Δu (i, j) represents the horizontal variation between adjacent pixels. Δv (i, j) represents the vertical variation between adjacent pixels. When the above formula represents c Δu When (i, j), Δ(u / v) - Δ(u / v) vi The actual value is Δu-Δu vi When the above expression represents c Δv When (i, j), Δ(u / v) - Δ(u / v) vi The actual value is Δv - Δv vi i / j represents either i or j. In i / j, i represents the horizontal index of the adjacent region. j represents the vertical index of the adjacent region. l represents a value controlling the size of the adjacent region. The size of the adjacent region can be represented as (2l+1). For example, when l is 1, the adjacent region can be a 3x3 pixel block. The mapped pixel is located at the center of the adjacent region.

[0074] As an example, Figure 2 The f-subgraph in the image shows how a pixel changes size based on the visual importance of its neighboring pixels, compared to the e-subgraph. The black dashed lines in the f-subgraph can represent the edges of adjacent pixels.

[0075] To expedite the determination of pixel size changes between adjacent pixels and avoid repeatedly calculating changes in the size of individual pixels, this disclosure adopts the following scheme.

[0076] The second step is to construct the feature convolution kernel. The feature convolution kernel can be constructed using the following formula:

[0077]

[0078] Where K(i,j,u / v) represents the two-channel convolution kernel, and the two channels correspond to u and v respectively.

[0079] The third step involves constructing a pixel size change map based on the optimized pixel scale information and the corresponding initial pixel scale information for each mapped pixel in the aforementioned mapped pixel set. Each pixel value in the pixel size change map represents the size change of the corresponding mapped pixel. This pixel size change map characterizes the distribution of size changes for each mapped pixel. Each pixel in the pixel size change map can include two channels. The pixel values ​​in the pixel size change map can be generated using the following formula:

[0080] chv(x, y, u / v) = Δ(u / v) vi -Δ(u / v).

[0081] Here, chv(x, y, u / v) represents the size change of the mapped pixel (u, v), and also represents the pixel value corresponding to the mapped pixel (u, v) in the pixel size change map. chv(x, y, u / v) can be decomposed into chv(x, y, u) and chv(x, y, v). chv(x, y, u) represents the horizontal size change of the mapped pixel (u, v). chv(x, y, v) represents the vertical size change of the mapped pixel (u, v).

[0082] Fourth, based on the aforementioned feature convolution kernels and pixel size variation map, a pixel size control map is generated. The pixel values ​​in the pixel size control map can be generated using the following formula:

[0083] pscm(x,y,u / v)=Conv(K,chv(x,y,u / v))+Δ(u / v).

[0084] In this context, pscm(x, y, u / v) represents the pixel value corresponding to the mapped pixel (u, v) in the pixel size control graph. pscm(x, y, u / v) can be broken down into pscm(x, y, u) and pscm(x, y, v). pscm(x, y, u) and pscm(x, y, v) represent the values ​​of the two channels included in the pixel in the pixel size control graph. pscm(x, y, u) represents the channel value used to control the horizontal size of the corresponding mapped pixel. pscm(x, y, v) represents the channel value used to control the vertical size of the corresponding mapped pixel. Conv(·) represents the convolution operation.

[0085] Step 105: Based on the pixel size control map, render the two-dimensional geometric scene map to obtain the scene rendering image.

[0086] In some embodiments, the execution entity may, based on the pixel size control map, render the two-dimensional geometric scene map in various ways to obtain a scene rendering image. The scene rendering image may be a two-dimensional image displaying the target scene.

[0087] In some optional implementations of certain embodiments, the aforementioned execution entity may perform rendering processing on the aforementioned two-dimensional geometric scene graph based on the aforementioned pixel size control graph through the following steps to obtain a scene rendering image:

[0088] The first step is to map the aforementioned 2D geometric scene image to a preset buffer based on the pixel size control map, thereby obtaining the target 2D geometric scene image. This target 2D geometric scene image can be a low-resolution 2D geometric scene image. The preset buffer can be a cache space pre-allocated by the GPU (Graphics Processing Unit). For each pixel (x, y) in the aforementioned 2D geometric scene image, the information in the pixel can be stored in the pixel at coordinates (u, v) in the preset buffer using the following formula:

[0089]

[0090] Where W represents the width of the two-dimensional geometric scene graph, and H represents the height of the two-dimensional geometric scene graph.

[0091] The second step is to render the aforementioned target 2D geometric scene image to obtain the low-resolution scene image to be restored. This low-resolution scene image can be the target 2D geometric scene image containing lighting information. This can be achieved by using pixel shaders to render the lighting on each pixel of the target 2D geometric scene image, thus obtaining the low-resolution scene image to be restored.

[0092] The third step involves performing inverse mapping on the aforementioned pixel size control map to obtain the scene rendering image. Specifically, for each pixel (u, v) in the aforementioned low-resolution scene image to be restored, the pixel's color can be stored in the pixel at coordinates (x, y) in the scene rendering image using the following formula:

[0093]

[0094] Optionally, the scene rendering image described above can be a scene image with the same resolution as the two-dimensional geometric scene diagram described above.

[0095] The above-described scene rendering image generation steps and related content, as an inventive point of this disclosure, solve the second technical problem mentioned in the background art: "The video image rendering process is time-consuming and has low rendering efficiency." The reasons for the long rendering time and low rendering efficiency of video images are often as follows: If the number of pixels to be rendered in the video image is large, a significant amount of time needs to be spent on lighting processing for each pixel during the rendering process, resulting in a long rendering time and low rendering efficiency. Solving this problem can achieve the effect of shorter video image rendering time and improved rendering efficiency. To achieve this effect, firstly, based on the pixel size control map, the two-dimensional geometric scene map is mapped to a downscaled variable rate buffer to obtain the target two-dimensional geometric scene map. This allows for the acquisition of low-resolution scene perception features, facilitating subsequent rendering of a small number of pixels. Then, during the rendering process, the pixel shader can determine the direct and indirect lighting conditions of each pixel based on the information in the downscaled variable rate buffer, and obtain a low-resolution scene image rendering result. Therefore, less time is spent on lighting processing for a small number of pixels to obtain a low-resolution image. Finally, the low-resolution result is inversely mapped to the high-resolution result to obtain a scene rendering image with the same resolution as the aforementioned two-dimensional geometric scene graph. Therefore, when rendering video images, the rendering time can be shortened, and rendering efficiency improved.

[0096] Optionally, the aforementioned executing entity can control an associated display device to display the rendered image of the scene. The display device may be a head-mounted display device with a screen.

[0097] The embodiments of this disclosure have the following beneficial effects: The efficient rendering method based on scene-aware mapping for complex scenes, as described in some embodiments of this disclosure, can improve the quality of image rendering and human visual perception. Specifically, the reason for the overall poor quality of image rendering results is that, during the rendering process, only the characteristics of the human visual system are considered, ignoring the characteristics of the scene itself, resulting in poor rendering quality of the periphery of the gaze point area. Based on this, the efficient rendering method based on scene-aware mapping for complex scenes, as described in some embodiments of this disclosure, firstly, acquires the scene model, camera parameters, and user gaze point information corresponding to the target scene. This facilitates subsequent scene image rendering of the target scene based on the user's visual characteristics. Secondly, based on the aforementioned camera parameters, the aforementioned scene model undergoes spatial mapping processing to obtain a two-dimensional geometric scene map. This allows the perception features of the target scene to be obtained. Then, based on the aforementioned user gaze point information and the aforementioned two-dimensional geometric scene map, a consistent visual importance map is generated. This allows the visual importance of each pixel in the scene in the current state to be determined. Finally, based on the aforementioned two-dimensional geometric scene map and the aforementioned consistent visual importance map, a pixel size control map is generated. Therefore, a mapping relationship between high-resolution and low-resolution images can be constructed based on scene-aware features and user visual features. Finally, based on the aforementioned pixel size control map, the aforementioned two-dimensional geometric scene map is rendered to obtain a scene-rendered image. Thus, during image rendering, each pixel in the image can be mapped and rendered according to the size change of each pixel in the pixel size control map. Therefore, the efficient rendering method based on scene-aware mapping for complex scenes in some embodiments of this disclosure, by combining two-dimensional geometric scene information and user gaze point information when generating a consistent visual importance map, can make the visual importance of each pixel in the scene more accurate, especially improving the visual importance of salient regions surrounding the gaze point region. Furthermore, by constructing a pixel size control map based on visual importance, it is convenient to render the image according to the visual importance corresponding to each pixel in the target scene. Therefore, the rendering quality can be improved for salient regions surrounding the gaze point region. This improves the overall quality of the image rendering result and the user's visual perception quality. Furthermore, because appropriate computing resources can be allocated to each pixel for rendering according to the pixel size control map, the waste of computing resources can be reduced.

[0098] The technical contents not described in detail in this invention are common knowledge to those skilled in the art.

[0099] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for efficient rendering based on scene-aware mapping for complex scenes, comprising: obtaining a scene model, camera parameters and user gaze point information corresponding to a target scene; performing spatial mapping processing on the scene model based on the camera parameters to obtain a two-dimensional geometric scene graph; generating a consistent visual importance map based on the user gaze point information and the two-dimensional geometric scene graph; generating a pixel size control map based on the two-dimensional geometric scene graph and the consistent visual importance map; performing rendering processing on the two-dimensional geometric scene graph based on the pixel size control map to obtain a scene rendering image; wherein generating a pixel size control map based on the two-dimensional geometric scene graph and the consistent visual importance map comprises: for each mapping pixel in the mapping pixel set corresponding to the two-dimensional geometric scene graph, performing the following steps: determining initial pixel scale information corresponding to the mapping pixel, wherein the initial pixel scale information includes an initial pixel width value and an initial pixel height value; determining a visual importance measure value corresponding to the mapping pixel based on the consistent visual importance map; performing scale optimization processing on the initial pixel scale information according to the visual importance measure value to obtain optimized pixel scale information, wherein the optimized pixel scale information includes an optimized pixel width value and an optimized pixel height value; constructing a feature convolution kernel; constructing a pixel size change map based on the optimized pixel scale information and the corresponding initial pixel scale information of each mapping pixel in the mapping pixel set; generating a pixel size control map based on the feature convolution kernel and the pixel size change map.

2. The method of claim 1, wherein, The method further comprises: performing visual feature extraction on the two-dimensional geometric scene graph to obtain a visual feature map; generating an acuity map corresponding to the two-dimensional geometric scene graph according to the user gaze point information; generating a visual importance map based on the visual feature map and the acuity map; performing normalization processing on the visual importance of each pixel in the visual importance map to obtain a consistent visual importance map.

3. The method of claim 2, wherein, Each pixel in the two-dimensional geometric scene graph corresponds to a mapping pixel in the mapping pixel set, and the mapping pixel in the mapping pixel set is the result of mapping a pixel in the two-dimensional geometric scene graph from one Cartesian coordinate system to another Cartesian coordinate system.

4. The method according to one of claims 1 to 3, wherein performing rendering processing on the two-dimensional geometric scene graph based on the pixel size control map to obtain a scene rendering image, comprising: mapping the two-dimensional geometric scene graph to a predetermined buffer based on the pixel size control map to obtain a target two-dimensional geometric scene graph, wherein the target two-dimensional geometric scene graph is a low-resolution two-dimensional geometric scene graph; performing rendering processing on the target two-dimensional geometric scene graph to obtain a low-resolution scene image to be restored; performing inverse mapping processing on the low-resolution scene image to be restored based on the pixel size control map to obtain a scene rendering image.

5. The method of claim 4, wherein, The scene rendering image is a scene image with the same resolution as the two-dimensional geometric scene graph.

6. The method of claim 1, wherein, The method further comprises: controlling a display device associated with the scene to display the scene rendering image.

Images