Efficient rendering method oriented to complex scene and based on scene perception mapping

By generating a two-dimensional geometric scene graph and a consistent visual importance graph, and constructing a pixel size control graph, the problem of poor rendering quality and long rendering time around the gaze point area in virtual reality scene rendering is solved, achieving efficient and high-quality rendering effects.

CN118887296AActive Publication Date: 2024-11-01BEIHANG UNIV
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202410889045.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-03
Publication Date
2024-11-01
Estimated Expiration
2044-07-03

AI Technical Summary

Technical Problem

In existing virtual reality scene rendering methods, ignoring scene features leads to poor rendering quality around the gaze point area, and the rendering process is time-consuming and inefficient.

Method used

By acquiring scene models, camera parameters, and user gaze point information, a two-dimensional geometric scene graph and a consistent visual importance graph are generated, a pixel size control graph is constructed, and efficient rendering is performed by combining scene perception and visual features.

Benefits of technology

It improves image rendering quality and visual perception quality, reduces wasted computing resources, shortens rendering time, and improves rendering efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887296A_ABST
    Figure CN118887296A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses an efficient rendering method oriented to a complex scene and based on scene perception mapping. A specific embodiment of the method comprises the following steps: acquiring a scene model corresponding to a target scene, camera parameters and user fixation point information; based on the camera parameters, performing space mapping processing on the scene model to obtain a two-dimensional geometric scene graph; generating a consistent visual importance map based on the user fixation point information and the two-dimensional geometric scene graph; generating a pixel size control chart based on the two-dimensional geometric scene graph and the consistent visual importance graph; and performing rendering processing on the two-dimensional geometric scene graph based on the pixel size control graph to obtain a scene rendering image. According to the embodiment, the computing resources required by rendering can be reduced and the rendering speed can be improved while the visual perception quality of people is guaranteed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the fields of computer graphics and virtual reality, and in particular to a method for efficiently rendering complex scenes based on scene perception mapping. Background Art

[0002] Virtual reality technology is widely used in entertainment games, cultural tourism, manufacturing and other fields, which puts higher requirements on the realism and efficiency of virtual reality scene rendering. At present, when rendering video images, the commonly used methods are: for each video image, a foveated rendering method based on rasterization and ray tracing is used, using more light samples for each pixel in the foveated area and fewer light samples for each pixel in the surrounding area, or a mapping-based foveated rendering method is used, and mapping and inverse mapping functions are constructed according to the characteristics of the human visual system, and image rendering is performed through a deferred rendering pipeline.

[0003] However, in practice, it is found that when the above method is used for image rendering, the following technical problems often occur:

[0004] First, since only the characteristics of the human visual system are considered during the rendering process, the characteristics of the scene itself are ignored, resulting in poor rendering quality of the salient areas outside the fixation area, which leads to poor overall quality of the image rendering result.

[0005] Second, if there are a large number of pixels to be rendered in a video image, a lot of time will be needed to perform illumination processing on each pixel during the rendering process, which will result in a long video image rendering process and low rendering efficiency.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0007] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0008] Some embodiments of the present disclosure propose an efficient rendering method based on scene-aware mapping for complex scenes to solve one or more of the technical problems mentioned in the above background technology section.

[0009] Some embodiments of the present disclosure provide an efficient rendering method based on scene perception mapping for complex scenes, the method comprising: acquiring a scene model, camera parameters and user gaze point information corresponding to the target scene; based on the above camera parameters, performing spatial mapping processing on the above scene model to obtain a two-dimensional geometric scene graph; based on the above user gaze point information and the above two-dimensional geometric scene graph, generating a consistent visual importance map; based on the above two-dimensional geometric scene graph and the above consistent visual importance map, generating a pixel size control map; based on the above pixel size control map, performing rendering processing on the above two-dimensional geometric scene graph to obtain a scene rendered image.

[0010] The embodiments of the present disclosure have the following beneficial effects: through the efficient rendering method based on scene perception mapping for complex scenes of some embodiments of the present disclosure, the quality of image rendering and the quality of human visual perception can be improved. Specifically, the reason for the poor overall quality of the image rendering result is that only the characteristics of the human visual system are considered in the rendering process, and the characteristics of the scene itself are ignored, so that the peripheral rendering quality of the gaze point area is poor. Based on this, the efficient rendering method based on scene perception mapping for complex scenes of some embodiments of the present disclosure first obtains the scene model, camera parameters and user gaze point information corresponding to the target scene. Thereby, it is convenient to perform scene image rendering on the target scene according to the user's visual characteristics in the subsequent process. Secondly, based on the above-mentioned camera parameters, the above-mentioned scene model is spatially mapped to obtain a two-dimensional geometric scene graph. Thereby, the perceptual characteristics of the target scene can be obtained. Then, based on the above-mentioned user gaze point information and the above-mentioned two-dimensional geometric scene graph, a consistent visual importance map is generated. Thereby, the visual importance of each pixel in the scene in the current state can be determined. Afterwards, based on the above-mentioned two-dimensional geometric scene graph and the above-mentioned consistent visual importance map, a pixel size control map is generated. Thus, a mapping relationship between a high-resolution image and a low-resolution image can be constructed according to the scene perception features and the user visual features. Finally, based on the above pixel size control map, the above two-dimensional geometric scene graph is rendered to obtain a scene rendering image. Thus, in the image rendering process, each pixel in the image can be mapped and rendered according to the size change of each pixel in the pixel size control map. Therefore, in some embodiments of the present disclosure, the efficient rendering method based on scene perception mapping for complex scenes can make the visual importance of each pixel in the scene more accurate by combining the two-dimensional geometric scene information and the user's gaze point information when generating a consistent visual importance map, and can especially improve the visual importance of the significant area outside the gaze point area. Further, the pixel size control map is constructed by visual importance, so as to facilitate image rendering according to the visual importance corresponding to each pixel in the target scene. Thus, for the significant area outside the gaze point area, the rendering quality can be improved. Thus, the overall quality of the image rendering result and the visual perception quality of the user can be improved. And because appropriate computing resources can be allocated to each pixel for rendering according to the pixel size control map, the waste of computing resources can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0012] Figure 1is a flow chart of some embodiments of an efficient rendering method based on scene-aware mapping for complex scenes according to the present disclosure;

[0013] Figure 2 It is a schematic diagram of the change of pixel coordinates in the process of stretching the logarithmic polar coordinate mapping into a rectangular mapping according to the efficient rendering method based on scene-aware mapping for complex scenes disclosed in the present invention. DETAILED DESCRIPTION

[0014] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0015] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0016] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0017] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0018] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0019] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0020] Figure 1 The process 100 of some embodiments of the efficient rendering method based on scene-aware mapping for complex scenes according to the present disclosure is shown. The efficient rendering method based on scene-aware mapping for complex scenes comprises the following steps:

[0021] Step 101, obtaining a scene model, camera parameters and user gaze point information corresponding to a target scene.

[0022] In some embodiments, the execution subject (e.g., virtual reality device) of the efficient rendering method based on scene perception mapping for complex scenes can obtain the scene model, camera parameters and user gaze point information corresponding to the target scene through a wired connection or a wireless connection. Among them, the above-mentioned complex scene can be a scene including a large number of objects with complex features. The above-mentioned complex features can include but are not limited to complex textures or complex lighting. The above-mentioned target scene can be a complex scene that requires scene image rendering. The scene image can be a two-dimensional image for displaying the corresponding scene. The above-mentioned scene model can be a three-dimensional grid model with lighting constructed according to the above-mentioned target scene through a preset modeling tool. The above-mentioned camera parameters can be the parameters of the camera that need to be set when shooting a single frame image for the above-mentioned target scene. The camera parameters can include but are not limited to the field of view angle and focal length. The above-mentioned user gaze point information can be the information of the position coordinates of the user's gaze point on the unrendered captured image obtained by an eye tracking device.

[0023] As an example, before rendering the image of the target scene captured by the video camera, the execution subject may obtain the scene model from the file database, obtain the camera parameters from the video camera, and obtain the user's gaze point information through the eye tracking device. The eye tracking device may be a device for measuring and recording the position and movement of the eyes. For example, the eye tracking device may be an eye tracker.

[0024] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0025] Step 102: Based on the camera parameters, the scene model is spatially mapped to obtain a two-dimensional geometric scene graph.

[0026] In some embodiments, the execution subject may perform spatial mapping processing on the scene model based on the camera parameters to obtain a two-dimensional geometric scene graph. The two-dimensional geometric scene graph may be a two-dimensional image including scene depth information, normal vector information and reflectivity information of the target scene. Each pixel in the two-dimensional geometric scene graph may include three channels. The three channels may correspond to the scene depth information, normal vector information and reflectivity information, respectively. The scene depth information may characterize the distance from an object or point in the scene to an observation point (camera lens). The normal vector information may characterize the orientation of an object at a corresponding pixel point. The reflectivity information may characterize the reflectivity of an object at a corresponding pixel point. The scene model may be spatially mapped based on the camera parameters to obtain a two-dimensional geometric scene graph by the following steps:

[0027] The first step is to perform MVP transformation on the scene model to obtain a scene model to be mapped. The MVP (Model-View-Projection) transformation may include model transformation, view transformation and projection transformation. The scene model to be mapped may be a scene model in the clipping space to be mapped to a two-dimensional screen.

[0028] The second step is to perform viewport transformation on the above-mentioned scene model to be mapped to obtain a two-dimensional geometric scene graph.

[0029] Step 103: Generate a consistent visual importance map based on the user gaze point information and the two-dimensional geometric scene graph.

[0030] In some embodiments, the execution entity may generate a consistent visual importance map in various ways based on the user gaze point information and the two-dimensional geometric scene graph. The consistent visual importance map may be a visual importance map with pixel values ​​between 0 and 1. The visual importance map may have the same resolution as the two-dimensional geometric scene graph. Each pixel value in the visual importance map may characterize the intensity of visual importance. The visual importance map may characterize the intensity distribution of visual importance corresponding to each pixel in the two-dimensional geometric scene graph. Visual importance may be the importance of visual features to the visual perception of the human eye. The visual features may include but are not limited to image contrast, saliency, edges, contours, and highlights. It should be noted that since the computing resources are fixed, the sum of all pixel values ​​of the visual importance map is also a fixed value.

[0031] In some optional implementations of some embodiments, the execution subject may generate a consistent visual importance map based on the user gaze point information and the two-dimensional geometric scene graph through the following steps:

[0032] The first step is to extract visual features from the above-mentioned two-dimensional geometric scene graph to obtain a visual feature graph. The above-mentioned visual feature graph has the same resolution as the above-mentioned two-dimensional geometric scene graph. Each pixel value in the above-mentioned visual feature graph can represent the intensity of the visual feature. Each pixel value can be a value between 0 and 1. The above-mentioned visual feature graph can represent the intensity distribution of the visual features corresponding to each pixel point in the two-dimensional geometric scene graph. Specifically, the following steps can be performed:

[0033] In the first sub-step, a saliency detection is performed on the above-mentioned two-dimensional geometric scene graph by a preset visual saliency detection method to obtain a first feature map. The above-mentioned first feature map can characterize the distribution of the saliency degree of each pixel in the above-mentioned two-dimensional geometric scene graph. The saliency degree of a pixel point can be the degree to which the pixel point attracts visual attention. For example, the above-mentioned visual saliency detection method can be but is not limited to one of the following: GR (Graph Regularized) algorithm, CA (Context-Aware) algorithm.

[0034] In the second sub-step, contour detection is performed on the two-dimensional geometric scene graph by a preset contour detection method to obtain a second feature map. The second feature map may be a binary image showing the contours of each geometric object in the two-dimensional geometric scene graph. For example, the contour detection method may be, but is not limited to, one of the following: Hough transform, contour detection based on edge tracking.

[0035] In the third sub-step, a highlight detection method is used to perform highlight detection on the two-dimensional geometric scene graph to obtain a third feature map. The third feature map can characterize the distribution of highlight areas in the two-dimensional geometric scene graph. For example, the highlight detection method can be, but is not limited to, one of the following: a threshold method, a histogram method.

[0036] The fourth sub-step is to fuse the first feature map, the second feature map and the third feature map through a perceptual filtering method to obtain a visual feature map. Specifically, the perceptual filtering method may be to select the maximum value from the corresponding pixel positions of the first feature map, the second feature map and the third feature map as the pixel value of each pixel position in the initial visual feature map, and determine the updated initial visual feature map as the visual feature map. The initial visual feature map may be initially a feature map whose pixel values ​​are all 0.

[0037] The second step is to generate a visual acuity map corresponding to the above-mentioned two-dimensional geometric scene graph based on the above-mentioned user gaze point information. Among them, the above-mentioned visual acuity map can characterize the distribution of visual acuity corresponding to each pixel point in the two-dimensional geometric scene graph. Visual acuity can be the importance of a pixel in an image. Visual acuity can be represented by a pixel value between 0 and 1, measured by a minimum resolution angle. The minimum resolution angle can be the minimum angle that a person can distinguish under certain conditions (for example, distance, brightness). The visual acuity in the visual acuity map can be generated by the following formula:

[0038] a(p)=w0+m×e(p,g).

[0039] Wherein, a(·) represents a function that takes a pixel as input and outputs the visual acuity corresponding to the pixel. p represents a pixel in a two-dimensional geometric scene graph. g represents the user's gaze point. a(p) represents the visual acuity corresponding to the pixel p in the visual acuity graph. w0 represents the minimum resolution angle of the visual acuity value. m represents the visual acuity slope. e(·) represents the eccentricity function. e(p, g) represents the eccentricity of the pixel p relative to the user's gaze point g. The eccentricity can characterize the angular distance from the pixel to the user's gaze point.

[0040] The third step is to generate a visual importance map based on the above-mentioned visual feature map and the above-mentioned visual acuity map. Among them, the visual acuity in the visual acuity map can also represent the ability of humans to accurately identify tiny details. If measured by the minimum resolution angle, the greater the visual acuity, the larger the minimum detail that can be distinguished, and the smaller the visual importance of details of the same size to the human visual system. Visual features can represent the ability or intensity of a scene to attract people's attention. The larger the value of the visual feature, the more the corresponding pixel point can attract the attention of the human eye, and the greater the visual importance of these pixels to the human visual system. Therefore, visual importance is positively correlated with visual features. Therefore, the pixel values ​​in the visual importance map can be generated based on the above-mentioned visual feature map and the above-mentioned visual acuity map by the following formula:

[0041]

[0042] Among them, vi(·) represents a function that takes a pixel point as input and takes the pixel value representing the visual importance corresponding to the pixel point as output. vi(p) represents the pixel value corresponding to the visual importance of pixel point p. v represents the movement speed of the user's gaze point, that is, the position difference between the two user gaze points corresponding to the two frames of images. G(·) represents a linear function that takes the running speed of the user's gaze point as input and takes the correction coefficient of visual importance as output. f(·) represents a function that takes a pixel point as input and takes the pixel value representing the visual feature corresponding to the pixel point as output. f(p) represents the value of the visual feature corresponding to pixel point p. k represents a predefined weighting coefficient. The correspondence data between f(p) and vi(p) can be collected, and k can be obtained by data fitting.

[0043] The fourth step is to normalize the visual importance corresponding to each pixel in the above visual importance map to obtain a consistent visual importance map. The pixel value in the consistent visual importance map can be generated by the following formula:

[0044] NorVi(p)=h×w×vi(p) / ∑vi.

[0045] Wherein, NorVi(·) represents a normalization function for normalizing the pixel values ​​in the visual importance map. NorVi(p) represents the result of normalizing the pixel values ​​corresponding to the pixel point p in the visual importance map. h represents the height of the down-resolution image. The down-resolution image can be an image obtained by downsampling the above-mentioned two-dimensional geometric scene graph. w represents the width of the down-resolution image. ∑vi represents the sum of all pixel values ​​in the visual importance map.

[0046] Optionally, each pixel in the above-mentioned two-dimensional geometric scene graph can correspond one-to-one to a mapping pixel in the mapping pixel set. The mapping pixels in the above-mentioned mapping pixel set can be the result of mapping the pixels in the two-dimensional geometric scene graph from one rectangular coordinate system to another rectangular coordinate system. In practice, since the logarithmic polar coordinate mapping method usually has the problem of low efficiency in space utilization after mapping, the present disclosure adopts a method of stretching the logarithmic polar coordinate mapping into a rectangular mapping to determine the mapping pixel corresponding to each pixel in the two-dimensional geometric scene graph.

[0047] As an example, Figure 2 The schematic diagram shows the change of pixel coordinates in the process of stretching the logarithmic polar coordinate mapping into a rectangular mapping in the efficient rendering method based on scene-aware mapping for complex scenes disclosed in the present invention. Figure 2 It includes 6 sub-graphs, namely a, b, c, d, e and f. Sub-graph a shows a two-dimensional geometric scene graph with a resolution of 7*7. The point with the coordinates of the upper left corner (0, 0) represents the user's gaze point, and the dark color block represents the visual feature. Sub-graph b shows a consistent visual importance map corresponding to the above-mentioned two-dimensional geometric scene graph. The lighter the pixel color, the higher the visual importance. Among them, the visual importance close to the gaze point and the visual feature position is higher. The two-dimensional geometric scene graph shown in sub-graph c represents each pixel with a different color for clear observation. For logarithmic polar coordinate mapping, the mapping function usually converts a square area into a polar coordinate near-circular area. The present disclosure approximates the logarithmic polar coordinate mapping as a transformation from rectangular coordinates to rectangular coordinates to ensure that the transformed area has the same area. Sub-graph d shows the area of ​​each mapped pixel calculated using logarithmic polar coordinate changes. Among them, the black area in sub-graph d indicates that no pixel is mapped to the area. Then, the present disclosure stretches the circular area in sub-graph d into a square area in sub-graph e to fully utilize the buffer. Since the mapping function between subgraph c and subgraph e needs to be injective and surjective, the mapping function needs to comply with the following two constraints:

[0048] Constraint 1: For a square area, its mapping area remains square (surjective). For constraint 1, the following formula can be used to convert the side length of the c subgraph to s i The square area is mapped to the e subgraph with a side length of s′ iThe square area of ​​:

[0049]

[0050] Wherein, i in the above formula represents the number of pixels corresponding to one side of the square area in the c sub-image. i Represents the side length of the square area in subgraph c. s i represents the side length of the square area in the e subgraph. a(e) represents the visual acuity of any pixel. The e in a(e) represents the eccentricity. sqrt(·) represents the square root function. dx represents the differential with respect to the independent variable x. dy represents the differential with respect to the independent variable y. The independent variables x and y are the horizontal and vertical coordinates of the lower right corner of the pixel in the two-dimensional geometric scene graph, respectively.

[0051] Constraint 2: The adjacency relationship between pixels in the square area and on the edge of the square remains unchanged (injective). Among them, for constraint 2, when the pixels on the edge of the square are adjacent, all pixels are adjacent. In practice, the adjacency of pixels can be ensured by calculating the length of the four sides of the mapped pixels. Specifically, starting from the square with a side length of 1, until the side length is the maximum value, if the adjacent relationship can be maintained on the side of each square, then all squares can maintain adjacent relationships. Therefore, the pixel may not be a rectangle, but something like a parallelogram or a trapezoid.

[0052] In practice, taking the dark pixel in sub-image c as an example, the upper left corner is determined by its upper left pixel, the upper right corner is determined by its upper pixel, and the lower left corner is determined by its left pixel. Therefore, it is only necessary to calculate the mapping coordinates of the lower right corner of the pixel. Among them, the mapping coordinates can be the coordinates of the mapping point after the lower right corner of the pixel is mapped to the transformation area. The ordinate in the mapping coordinates is equal to the side length of the square area in sub-image e. The abscissa in the mapping coordinates can be determined by the ratio of the sum of the visual acuity of each pixel to the left of the point (i.e., the lower right corner of the dark pixel) to the sum of the visual acuity of all pixels on the edge of the square area. The abscissa in the mapping coordinates can be generated by the following formula:

[0053]

[0054] In addition, the equations for the right edge pixels of the square can also be derived in a similar way. For simplicity, the gaze point is set to (0, 0), and only the mathematical derivation process of the pixels located to the lower right of the gaze point is listed. It should be noted that when the gaze point coordinates are not (0, 0), the equations for the remaining pixels can also be derived in a similar way. Therefore, for each pixel to the lower right of the gaze point in the above two-dimensional geometric scene graph, the mapping pixel corresponding to the pixel can be generated by the following formula:

[0055]

[0056] Among them, u(x, y) and v(x, y) respectively represent the horizontal coordinate and vertical coordinate of the mapped pixel (u, v) corresponding to the pixel (x, y) in the two-dimensional geometric scene graph.

[0057] Step 104: Generate a pixel size control map based on the two-dimensional geometric scene graph and the consistent visual importance map.

[0058] In some embodiments, the execution subject may generate a pixel size control map based on the two-dimensional geometric scene graph and the consistent visual importance map in various ways. The pixel size control map may be a two-dimensional map of the same size as the consistent visual importance map. Each pixel in the pixel size control map may include two components, representing the vertical size and horizontal size of the pixel, respectively. The pixel size control map may be used to allocate the required computing resources according to the size of each pixel when rendering an image.

[0059] In some optional implementations of some embodiments, the execution subject may generate a pixel size control map based on the two-dimensional geometric scene graph and the consistent visual importance map through the following steps:

[0060] In the first step, for each mapped pixel in the mapped pixel set corresponding to the above two-dimensional geometric scene graph, the following steps are performed:

[0061] The first sub-step is to determine the initial pixel scale information corresponding to the above-mentioned mapped pixel, wherein the above-mentioned initial pixel scale information may include an initial pixel width value and an initial pixel height value. The above-mentioned initial pixel width value may be the horizontal width of the mapped pixel before optimization. The above-mentioned initial pixel height value may be the vertical height of the mapped pixel before optimization. It should be noted that the initial pixel scale information corresponding to each mapped pixel is the same. The initial pixel scale information corresponding to the above-mentioned mapped pixel may be determined by the following formula:

[0062]

[0063] Among them, Δu(x, y) represents the horizontal change between the two mapping coordinates corresponding to the pixel (x, y) and the pixel (x-1, y), that is, the initial pixel width value of the above-mentioned mapping pixel. Δv(x, y) represents the vertical change between the two mapping coordinates corresponding to the pixel (x, y) and the pixel (x, y-1), that is, the initial pixel height value of the above-mentioned mapping pixel. u(x-1, y) represents the horizontal coordinate of the mapping pixel corresponding to the pixel (x-1, y) in the two-dimensional geometric scene graph. v(x, y-1) represents the vertical coordinate of the mapping pixel corresponding to the pixel (x, y-1) in the two-dimensional geometric scene graph. represents the partial derivative of u with respect to x. Δx represents the change in the horizontal coordinate of the pixel. Represents the partial derivative of v with respect to y. Δy represents the change in the vertical coordinate of the pixel.

[0064] Optionally, the execution subject may express the pixel (x, y) in the two-dimensional geometric scene graph by the following formula according to the corresponding mapping pixel (u, v):

[0065]

[0066] Where exp(·) represents the natural exponential function. x(u, v) and y(u, v) represent the horizontal and vertical coordinates of the pixel (x, y) in the two-dimensional geometric scene graph, respectively.

[0067] The second sub-step is to determine the visual importance metric value corresponding to the above-mentioned mapped pixel based on the above-mentioned consistent visual importance map. The above-mentioned visual importance metric value may be a pixel value in the consistent visual importance map. First, the pixel corresponding to the above-mentioned mapped pixel in the above-mentioned two-dimensional geometric scene map is determined as the target pixel. Then, the pixel matching the above-mentioned target pixel is selected from the pixels included in the above-mentioned consistent visual importance map. Matching the above-mentioned target pixel may be that the pixel in the above-mentioned consistent visual importance map has the same position coordinates as the above-mentioned target pixel. Finally, the pixel value corresponding to the selected pixel is determined as the visual importance metric value.

[0068] In the third sub-step, according to the visual importance metric, the initial pixel scale information is scale-optimized to obtain the optimized pixel scale information. The optimized pixel scale information includes an optimized pixel width value and an optimized pixel height value. The optimized pixel width value may be the optimized initial pixel width value. The optimized pixel height value may be the optimized initial pixel height value. The optimized pixel width value and the optimized pixel height value included in the optimized pixel scale information may be generated according to the following formula:

[0069]

[0070] Among them, Δ(u / v) vi In the formula, vi represents the visual importance metric corresponding to the mapped pixel (u, v). To simplify the formula, Δu vi or Δv vi The formula is combined to express. Δ(u / v) represents Δu or Δv. When the above formula is Δuvi, The actual When the above formula is Δuvi, The actual Δu is the abbreviation of Δu(x, y). Δv is the abbreviation of Δv(x, y). area represents the area of ​​the mapped pixel (u, v).

[0071] In practice, the size of the mapped pixel is not only related to its own visual importance, but also to the size of the adjacent pixels. Considering that the mapped pixels must satisfy the above constraint 2, if only one pixel is changed in size, it will cause overlapping of adjacent pixels or gaps between pixels. Therefore, if the size of a pixel and its adjacent pixels is to be changed, it is necessary to traverse all pixels and determine the horizontal and vertical changes of the mapped pixel size and adjacent pixels. Because the "focus" is usually assumed to decay monotonically around its center, the present disclosure decides to use a logarithmic function to control the size change of adjacent pixels instead of uniformly scaling the size of adjacent pixels. Specifically, according to Δ(u / v) vi Proportional to ln(1+dis), where dis is the horizontal and vertical distance between the adjacent pixel and the mapped pixel at the center, the pixel size change of any adjacent pixel can be determined by the following formula:

[0072]

[0073] Where Δu / Δv represents Δu or Δv. To simplify the formula, cΔu / Δv(i, j) represents c Δu (i, j) or c Δv (i, j). Δu (i, j) represents the horizontal change of adjacent pixels. Δv (i, j) represents the vertical change of adjacent pixels. Δu (i, j), Δ(u / v)-Δ(u / v) vi Actually it is Δu-Δu vi When the above formula represents c Δv (i, j), Δ(u / v)-Δ(u / v) vi Actually it is Δv-Δv vi . i / j represents i or j. i in i / j represents the horizontal index of the adjacent region. j in i / j represents the vertical index of the adjacent region. l represents a value that controls the size of the adjacent region. The size of the adjacent region can be represented by (2l+1). For example, when l is 1, the adjacent region can be a 3*3 pixel block. Among them, the mapped pixel is located at the center of the adjacent region.

[0074] As an example, Figure 2 The f sub-image in Fig. 1 shows how a pixel changes size according to the visual importance of its neighboring pixels compared to the e sub-image. The black dashed lines in the f sub-image can be the edges of neighboring pixels.

[0075] In order to speed up the determination of pixel size changes of adjacent pixels and avoid repeated calculation of changes in single pixel size, the present disclosure decides to adopt the following solution.

[0076] The second step is to construct the feature convolution kernel. The feature convolution kernel can be constructed by the following formula:

[0077]

[0078] Among them, K(i, j, u / v) represents the convolution kernel of the two channels, and the two channels correspond to u and v respectively.

[0079] The third step is to construct a pixel size change map based on the optimized pixel scale information and the corresponding initial pixel scale information corresponding to each mapped pixel in the above-mentioned mapped pixel set. Each pixel value in the above-mentioned pixel size change map can be the size change amount of the corresponding mapped pixel. The above-mentioned pixel size change map can characterize the distribution of the size change amounts of each mapped pixel. Each pixel in the above-mentioned pixel size change map can include two channels. The pixel values ​​in the pixel size change map can be generated by the following formula:

[0080] chv(x, y, u / v) = Δ(u / v) vi -Δ(u / v).

[0081] Among them, chv(x, y, u / v) represents the size change of the mapped pixel (u, v), and also represents the pixel value corresponding to the mapped pixel (u, v) in the pixel size change graph. chv(x, y, u / v) can be split into chv(x, y, u) and chv(x, y, v). chv(x, y, u) represents the horizontal size change of the mapped pixel (u, v). chv(x, y, v) represents the vertical size change of the mapped pixel (u, v).

[0082] The fourth step is to generate a pixel size control map based on the above feature convolution kernel and the above pixel size change map. The pixel values ​​in the pixel size control map can be generated by the following formula:

[0083] pscm(x,y,u / v)=Conv(K,chv(x,y,u / v))+Δ(u / v).

[0084] Among them, pscm(x, y, u / v) represents the pixel value corresponding to the mapped pixel (u, v) in the pixel size control map. Pscm(x, y, u / v) can be split into pscm(x, y, u) and pscm(x, y, v). Pscm(x, y, u) and pscm(x, y, v) represent the values ​​of the two channels included in the pixel in the pixel size control map. Pscm(x, y, u) represents the channel value used to control the horizontal size of the corresponding mapped pixel. Pscm(x, y, v) represents the channel value used to control the vertical size of the corresponding mapped pixel. Conv(·) represents the convolution operation.

[0085] Step 105 , based on the pixel size control graph, the two-dimensional geometric scene graph is rendered to obtain a scene rendering image.

[0086] In some embodiments, the execution subject may render the two-dimensional geometric scene graph in various ways based on the pixel size control graph to obtain a scene rendering image, wherein the scene rendering image may be a two-dimensional image showing the target scene.

[0087] In some optional implementations of some embodiments, the execution subject may render the two-dimensional geometric scene graph based on the pixel size control graph to obtain a scene rendering image through the following steps:

[0088] The first step is to map the above two-dimensional geometric scene graph to a preset buffer based on the above pixel size control graph to obtain a target two-dimensional geometric scene graph. The above target two-dimensional geometric scene graph can be a low-resolution two-dimensional geometric scene graph. The above preset buffer can be a cache space pre-allocated by a GPU (Graphics Processing Unit). For each pixel (x, y) in the above two-dimensional geometric scene graph, the information in the pixel can be stored in the pixel with coordinates (u, v) in the preset buffer by the following formula:

[0089]

[0090] Wherein, W represents the width of the two-dimensional geometric scene graph, and H represents the height of the two-dimensional geometric scene graph.

[0091] The second step is to render the target two-dimensional geometric scene graph to obtain a low-resolution scene image to be restored. The low-resolution scene image to be restored may be a target two-dimensional geometric scene graph with lighting information. The pixel shader may be used to perform lighting rendering on each pixel in the target two-dimensional geometric scene graph to obtain the low-resolution scene image to be restored.

[0092] The third step is to perform inverse mapping on the low-resolution scene image to be restored based on the pixel size control map to obtain a scene rendering image. For each pixel (u, v) in the low-resolution scene image to be restored, the color of the pixel can be stored in the pixel with coordinates (x, y) in the scene rendering image by the following formula:

[0093]

[0094] Optionally, the scene rendering image may be a scene image having the same resolution as the two-dimensional geometric scene graph.

[0095] The above-mentioned scene rendering image generation step and its related contents, as an inventive point of an embodiment of the present disclosure, solve the technical problem 2 mentioned in the background technology, "the video image rendering process takes a long time and the rendering efficiency is low". The reasons for the long time and low rendering efficiency of the video image rendering process are often as follows: if the number of pixels to be rendered in the video image is large, a lot of time is needed to perform illumination processing on each pixel during the rendering process, thereby causing the video image rendering process to take a long time and the rendering efficiency to be low. If the above-mentioned problems are solved, the video image rendering process takes a short time and the rendering efficiency is improved. In order to achieve this effect, first, according to the pixel size control map, the two-dimensional geometric scene graph is mapped to the buffer of the reduced resolution rate to obtain the target two-dimensional geometric scene graph. Thus, low-resolution scene perception features can be obtained, which is convenient for subsequent rendering of a small number of pixels. Then, during the rendering process, the pixel shader can determine the direct and indirect illumination conditions of each pixel based on the information in the reduced resolution rate buffer, and obtain a low-resolution scene image rendering result. Thus, it takes less time to perform illumination processing on a small number of pixels to obtain a low-resolution image. Finally, the low-resolution result is inversely mapped to the high-resolution result to obtain a scene rendering image with the same resolution as the above-mentioned two-dimensional geometric scene graph. Therefore, when performing video image rendering, the image rendering time can be shortened and the rendering efficiency can be improved.

[0096] Optionally, the execution subject may control an associated display device to display the scene rendering image, wherein the display device may be a head-mounted display device with a display screen.

[0097] The embodiments of the present disclosure have the following beneficial effects: through the efficient rendering method based on scene perception mapping for complex scenes of some embodiments of the present disclosure, the quality of image rendering and the quality of human visual perception can be improved. Specifically, the reason for the poor overall quality of the image rendering result is that only the characteristics of the human visual system are considered in the rendering process, and the characteristics of the scene itself are ignored, so that the peripheral rendering quality of the gaze point area is poor. Based on this, the efficient rendering method based on scene perception mapping for complex scenes of some embodiments of the present disclosure first obtains the scene model, camera parameters and user gaze point information corresponding to the target scene. Thereby, it is convenient to perform scene image rendering on the target scene according to the user's visual characteristics in the subsequent process. Secondly, based on the above-mentioned camera parameters, the above-mentioned scene model is spatially mapped to obtain a two-dimensional geometric scene graph. Thereby, the perceptual characteristics of the target scene can be obtained. Then, based on the above-mentioned user gaze point information and the above-mentioned two-dimensional geometric scene graph, a consistent visual importance map is generated. Thereby, the visual importance of each pixel in the scene in the current state can be determined. Afterwards, based on the above-mentioned two-dimensional geometric scene graph and the above-mentioned consistent visual importance map, a pixel size control map is generated. Thus, a mapping relationship between a high-resolution image and a low-resolution image can be constructed according to the scene perception features and the user visual features. Finally, based on the above pixel size control map, the above two-dimensional geometric scene graph is rendered to obtain a scene rendering image. Thus, in the image rendering process, each pixel in the image can be mapped and rendered according to the size change of each pixel in the pixel size control map. Therefore, in some embodiments of the present disclosure, the efficient rendering method based on scene perception mapping for complex scenes can make the visual importance of each pixel in the scene more accurate by combining the two-dimensional geometric scene information and the user's gaze point information when generating a consistent visual importance map, and can especially improve the visual importance of the significant area outside the gaze point area. Further, the pixel size control map is constructed by visual importance, so as to facilitate image rendering according to the visual importance corresponding to each pixel in the target scene. Thus, for the significant area outside the gaze point area, the rendering quality can be improved. Thus, the overall quality of the image rendering result and the visual perception quality of the user can be improved. And because appropriate computing resources can be allocated to each pixel for rendering according to the pixel size control map, the waste of computing resources can be reduced.

[0098] The technical contents not elaborated in detail in the present invention belong to the common knowledge of those skilled in the art.

[0099] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. An efficient rendering method based on scene-aware mapping for complex scenes, comprising: Obtaining the scene model, camera parameters and user gaze point information corresponding to the target scene; Based on the camera parameters, spatial mapping is performed on the scene model to obtain a two-dimensional geometric scene graph; Generate a consistent visual importance map based on the user gaze point information and the two-dimensional geometric scene graph; generating a pixel size control map based on the two-dimensional geometric scene graph and the consistent visual importance map; Based on the pixel size control graph, the two-dimensional geometric scene graph is rendered to obtain a scene rendering image.

2. The method according to claim 1, wherein: The generating a consistent visual importance graph based on the user gaze point information and the two-dimensional geometric scene graph includes: Extracting visual features from the two-dimensional geometric scene graph to obtain a visual feature graph; Generating a visual acuity map corresponding to the two-dimensional geometric scene graph according to the user gaze point information; generating a visual importance map based on the visual feature map and the visual acuity map; The visual importance corresponding to each pixel in the visual importance map is normalized to obtain a consistent visual importance map.

3. The method according to claim 2, wherein: Each pixel in the two-dimensional geometric scene graph corresponds to a mapping pixel in a mapping pixel set, and the mapping pixels in the mapping pixel set are the results of mapping the pixels in the two-dimensional geometric scene graph from one rectangular coordinate system to another rectangular coordinate system.

4. The method according to claim 3, wherein: The generating a pixel size control map based on the two-dimensional geometric scene graph and the consistent visual importance map comprises: For each mapped pixel in the mapped pixel set corresponding to the two-dimensional geometric scene graph, the following steps are performed: Determining initial pixel scale information corresponding to the mapped pixel, wherein the initial pixel scale information includes an initial pixel width value and an initial pixel height value; Determining a visual importance metric value corresponding to the mapped pixel based on the consistent visual importance map; According to the visual importance metric value, the initial pixel scale information is subjected to scale optimization processing to obtain optimized pixel scale information, wherein the optimized pixel scale information includes an optimized pixel width value and an optimized pixel height value; Construct feature convolution kernel; constructing a pixel size change map based on optimized pixel scale information and corresponding initial pixel scale information corresponding to each mapped pixel in the mapped pixel set; A pixel size control map is generated based on the feature convolution kernel and the pixel size change map.

5. The method according to any one of claims 1 to 4, wherein: The step of rendering the two-dimensional geometric scene graph based on the pixel size control graph to obtain a scene rendering image includes: Based on the pixel size control graph, mapping the two-dimensional geometric scene graph to a preset buffer to obtain a target two-dimensional geometric scene graph, wherein the target two-dimensional geometric scene graph is a low-resolution two-dimensional geometric scene graph; Rendering the target two-dimensional geometric scene graph to obtain a low-resolution scene image to be restored; Based on the pixel size control map, an inverse mapping process is performed on the low-resolution scene image to be restored to obtain a scene rendering image.

6. The method according to claim 5, wherein: The scene rendering image is a scene image having the same resolution as the two-dimensional geometric scene graph.

7. The method according to claim 1, wherein: The method further comprises: Controlling an associated display device to display the scene rendering image.

Citation Information

Patent Citations

  • Temporal supersampling for foveated rendering systems

    CN110869884A

  • Gaze point rendering method and system, computer and readable storage medium

    CN116597288A

  • Real-time rendering method for dynamic environment illumination

    CN117011446A

  • Scene-adaptive fixation point neural radiation field rendering method and system

    CN117058293A

  • Method, device and equipment for generating data set through virtual reality engine

    CN117333600A