Gaze point rendering method and system based on global illumination decomposition

Abstract

The invention discloses a fixation point rendering method and system based on global illumination decomposition, and belongs to the technical field of computer graphics and virtual reality. Comprising the steps of obtaining scene data and performing illumination decomposition to generate a direct illumination image and an indirect illumination image; obtaining a fixation point position and expected image quality, and combining the fixation point position and the expected image quality with the direct illumination image and the indirect illumination image to determine direct illumination rendering parameters and indirect illumination rendering parameters; respectively constructing a direct illumination perception model and an indirect illumination perception model by utilizing the direct illumination rendering parameters and the indirect illumination rendering parameters; and taking the direct illumination perception model and the indirect illumination perception model as guidance, and utilizing the lightweight image reconstruction network to obtain a fixation point rendering image. The rendering efficiency can be greatly improved, and meanwhile the perception quality is kept; the problem that in a traditional fixation point rendering method, the influence of different illumination conditions on visual perception is not fully considered, and consequently rendering efficiency improvement is limited is solved.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics and virtual reality technology, and particularly to a fixation rendering method and system based on global illumination decomposition. Background Art

[0002] The statements in this section merely mention the background art related to the present invention and do not necessarily constitute prior art.

[0003] In the fields of computer graphics and virtual reality, the fixation rendering technology is a method that uses the visual characteristics of the human eye to improve the rendering efficiency. This technology dynamically adjusts the rendering quality of the image according to the difference in visual sensitivity of different regions of the human eye. Specifically, the fixation rendering technology provides high-quality rendering in the high-resolution region (i.e., the foveal region) in the center of the human eye, while reducing the rendering quality in the peripheral region, thereby reducing the synthesis of high-frequency details that are imperceptible to the observer. This method is considered very promising for real-time rendering and virtual reality due to its efficiency.

[0004] Although there have been various fixation rendering technologies based on visual perception models proposed, these models only consider the sensitivity of the human eye to aspects such as sharpness, contrast, and color sensitivity, but they usually do not consider the impact of different lighting effects on fixation rendering. In the rendering process, direct lighting and indirect lighting are two main parts, and among them, the computational cost of indirect lighting is usually much higher than that of direct lighting.

[0005] In addition, there are significant differences between direct lighting and indirect lighting visually, and these differences affect human visual perception. In scenes with many scattering objects, the frequency of indirect lighting is lower and can tolerate a higher level of fixation rendering. Summary of the Invention

[0006] To solve the deficiencies of the prior art, the present invention provides a fixation rendering method, system, electronic device, computer-readable storage medium, and computer program product based on global illumination decomposition. By taking advantage of the different degrees of human eye perception under different lighting conditions, the global illumination is decomposed, and different fixation rendering strategies are adopted for direct lighting and indirect lighting respectively, thereby further improving the rendering efficiency while maintaining the visual quality.

[0007] In the first aspect, the present invention provides a fixation rendering method based on global illumination decomposition;

[0008] A fixation rendering method based on global illumination decomposition includes:

[0009] Obtain scene data and perform illumination decomposition to generate a direct illumination image and an indirect illumination image;

[0010] Obtain the fixation point position and the expected image quality, and combine them with the direct illumination image and the indirect illumination image to determine the direct illumination rendering parameters and the indirect illumination rendering parameters;

[0011] Use the direct illumination rendering parameters and the indirect illumination rendering parameters to construct a direct illumination perception model and an indirect illumination perception model respectively; guided by the direct illumination perception model and the indirect illumination perception model, use a lightweight image reconstruction network to obtain the fixation point rendering image.

[0012] In some embodiments, combining the fixation point position and the expected image quality with the direct illumination image and the indirect illumination image to determine the direct illumination rendering parameters and the indirect illumination rendering parameters specifically includes:

[0013] Process the fixation point position, the expected image quality, the direct illumination image, and the indirect illumination image through a trained rendering parameter prediction network to obtain the direct illumination rendering parameters and the indirect illumination rendering parameters.

[0014] In some embodiments, processing the fixation point position, the expected image quality, the direct illumination image, and the indirect illumination image through a trained rendering parameter prediction network includes:

[0015] Stitch the direct illumination image and the indirect illumination image, and perform dimensionality reduction and feature extraction on the stitching result to obtain a light feature vector;

[0016] Stitch the light feature vector with the fixation point position and the bias information, and sequentially process the stitching result through a normalization layer, an activation function, and a fully connected layer, and generate the direct illumination rendering parameters and the indirect illumination rendering parameters with the expected image quality as a reference constraint.

[0017] In some embodiments, the using the direct illumination rendering parameters and the indirect illumination rendering parameters to construct a direct illumination perception model and an indirect illumination perception model respectively includes:

[0018] Use the comparison result of the radius and eccentricity of the central region of the image, and construct a direct illumination perception model according to the direct illumination rendering parameters, the truncation value of the outer region of the image, and the sampling factor of the central region;

[0019] Use the comparison result of the radius and eccentricity of the central region of the image, and construct an indirect illumination perception model according to the indirect illumination rendering parameters, the truncation value of the outer region of the image, and the sampling factor of the central region.

[0020] In some embodiments, the using the direct illumination perception model and the indirect illumination perception model as a guide and using a lightweight image reconstruction network to obtain the fixation point rendering image includes:

[0021] Calculate the sampling factor at each pixel position in the direct illumination image using the direct illumination perception model, and generate a direct illumination binary mask for direct illumination sparse rendering to obtain a direct illumination sparse rendering image; calculate the sampling factor at each pixel position in the indirect illumination image using the indirect illumination perception model, and generate an indirect illumination binary mask for indirect illumination sparse rendering to obtain an indirect illumination sparse rendering image;

[0022] Process the direct illumination sparse rendering image and the indirect illumination sparse rendering image respectively through a lightweight image reconstruction network to obtain a direct illumination fixation rendering image and an indirect illumination fixation rendering image;

[0023] Couple the direct illumination fixation rendering image and the indirect illumination fixation rendering image to generate a fixation rendering image.

[0024] In some embodiments, the lightweight image reconstruction network includes an image reconstruction sub-network and an image refinement sub-network. The image reconstruction sub-network includes a plurality of first convolutional units, a plurality of second convolutional units, and a third convolutional unit connected in sequence. The image refinement sub-network includes a plurality of fourth convolutional units, a plurality of fifth convolutional units, and a convolutional block connected in sequence.

[0025] In a second aspect, the present invention provides a fixation rendering system based on global illumination decomposition;

[0026] A fixation rendering system based on global illumination decomposition, comprising:

[0027] An illumination decomposition module, configured to: obtain scene data and perform illumination decomposition to generate a direct illumination image and an indirect illumination image;

[0028] A fixation rendering module, configured to: obtain a fixation position and an expected image quality, and combine them with the direct illumination image and the indirect illumination image to determine direct illumination rendering parameters and indirect illumination rendering parameters;

[0029] Use the direct illumination rendering parameters and the indirect illumination rendering parameters to respectively construct a direct illumination perception model and an indirect illumination perception model; guided by the direct illumination perception model and the indirect illumination perception model, use a lightweight image reconstruction network to obtain a fixation rendering image.

[0030] In a third aspect, the present invention provides an electronic device;

[0031] An electronic device, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of the above-mentioned fixation rendering method based on global illumination decomposition.

[0032] Fourthly, the present invention provides a computer-readable storage medium;

[0033] A computer-readable storage medium, on which computer programs / instructions are stored, and when the computer programs / instructions are executed by a processor, the steps of the above-mentioned fixation point rendering method based on global illumination decomposition are implemented.

[0034] Fifthly, the present invention provides a computer program product;

[0035] A computer program product, including computer programs / instructions, and when the computer programs / instructions are executed by a processor, the steps of the above-mentioned fixation point rendering method based on global illumination decomposition are implemented.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] 1. The technical solution provided by the present invention is based on global illumination decomposition, and realizes fixation point rendering for direct illumination and indirect illumination respectively, which can significantly improve the rendering speed. Especially in the rendering of indirect illumination, compared with the traditional fixation point rendering method, the rendering efficiency is greatly improved, and the perceptual quality is maintained at the same time.

[0038] 2. The technical solution provided by the present invention uses a deep learning network to predict the blur rate in different scenarios, automatically adjusts the rendering parameters of direct illumination and indirect illumination, and realizes adaptive rendering based on the scene content and lighting conditions; finds the most suitable fixation point rendering strategies for direct illumination and indirect illumination, realizes independent control under different lighting conditions, and improves the flexibility of fixation point rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0040] Figure 1 It is a schematic flow chart of the fixation point rendering method based on global illumination decomposition provided by the embodiment of the present invention;

[0041] Figure 2 It is a schematic network structure diagram of the rendering parameter prediction network provided by the embodiment of the present invention;

[0042] Figure 3 It is an example diagram of the scene used by the rendering parameter prediction network provided by the embodiment of the present invention;

[0043] Figure 4 It is an example diagram of the perception model provided by the embodiment of the present invention, wherein (a) is an example diagram of the direct illumination perception model, and (b) is an example diagram of the indirect illumination perception model;

[0044] Figure 5 An example diagram of binary masking provided by an embodiment of the present invention;

[0045] Figure 6 A schematic diagram of the network architecture of a lightweight image reconstruction network provided by an embodiment of the present invention;

[0046] Figure 7 An example diagram of time gain under the same quality provided by an embodiment of the present invention. Detailed implementation manners

[0047] It should be noted that the following detailed description is exemplary and is intended to provide further illustration of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0048] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or devices.

[0049] In the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0050] Embodiment 1

[0051] In the existing fixation rendering method, the influence of different lighting conditions on visual perception is not fully considered, resulting in limited improvement in rendering efficiency. Therefore, the present invention provides a fixation rendering method based on global illumination decomposition. By taking advantage of the different degrees of human eye perception under different lighting conditions, the global illumination is decomposed, and different fixation rendering strategies are adopted for direct illumination and indirect illumination respectively, so as to further improve the rendering efficiency while maintaining visual quality.

[0052] Next, in combination with Figures 1 - 7 , a fixation rendering method based on global illumination decomposition disclosed in this embodiment will be described in detail. The fixation rendering method based on global illumination decomposition includes:

[0053] S1. Obtain scene data and perform illumination decomposition to generate a direct illumination image and an indirect illumination image.

[0054] To improve the rendering efficiency, for different scene contents, render and perform light decomposition at a low resolution; specifically, use a renderer to render the scene data at a resolution of 240×135 to obtain a low-resolution direct illumination image and indirect illumination image, and complete the light decomposition.

[0055] S2. Obtain the fixation point position and the expected image quality, and combine them with the direct illumination image and the indirect illumination image to determine the direct illumination rendering parameters and the indirect illumination rendering parameters.

[0056] Here, the fixation point position of the user is obtained through an eye tracker, and the expected image quality is input through user interaction, in units of JOD.

[0057] The quality of the rendered image finally output by the existing fixation point rendering is unpredictable, and there is a risk of not meeting the user's needs; therefore, in this embodiment, a trained rendering parameter prediction network processes the fixation point position, the expected image quality, the direct illumination image, and the indirect illumination image to obtain the direct illumination rendering parameters and the indirect illumination rendering parameters, so as to further ensure the consistency and correctness of the final rendering result.

[0058] Furthermore, the rendering parameter prediction network includes a 3×3 two-dimensional convolutional layer Conv2D, a two-dimensional normalization layer BatchNorm2D, a two-dimensional max pooling layer MaxPool2D, a ReLU activation function, a concatenation layer, a one-dimensional normalization layer BatchNorm1D, a ReLU activation function, and a fully connected layer connected in sequence.

[0059] Specifically, the process of using a trained rendering parameter prediction network to process the fixation point position, the expected image quality, the direct illumination image, and the indirect illumination image to obtain the direct illumination rendering parameters and the indirect illumination rendering parameters includes:

[0060] First, concatenate the 3*H*W direct illumination image and the indirect illumination image (3*H*W); use the 3×3 two-dimensional convolutional layer Conv2D, the two-dimensional normalization layer BatchNorm2D, and the two-dimensional max pooling layer MaxPool2D to reduce the dimension of the concatenated direct illumination image and indirect illumination image (6*H*W) and extract features; then, activate the extracted features through the ReLU activation function; next, flatten the activation results into light feature vectors respectively, and concatenate the light feature vectors with the fixation point position and the bias information; after that, perform sequential processing on the concatenated results through the one-dimensional normalization layer BatchNorm1D, the ReLU activation function, and the fully connected layer, and output the direct illumination rendering parameter BR direct and the indirect illumination rendering parameter BR direct .

[0061] During the training process of the rendering parameter prediction network, the expected image quality input by the user is used as a label and input into the network together with the input image for training; the goal of the network is to predict the rendering parameters related to the user's expected image quality by extracting and processing the features of the low-resolution input image. Taking the expected image quality as the target value, the parameters of the network are adjusted through backpropagation during the training process, enabling the network to learn the relationship between the input image and the expected image quality, so as to accurately predict the rendering parameters corresponding to the image quality.

[0062] Therefore, in the data processing process of the above-mentioned trained rendering parameter prediction network, the rendering parameter prediction network converts the expected image quality into a "reference constraint" for rendering parameter prediction, and combines the expected image quality with the image content through the method of deep feature fusion, so as to predict the optimal rendering parameters. In this embodiment, the direct illumination rendering parameter BR direct refers to the direct illumination blur and the indirect illumination rendering parameter BR direct refers to the indirect illumination blur.

[0063] As an implementation method, the specific process of training the rendering parameter prediction network is as follows:

[0064] Step 1: Obtain network data including low-resolution images of multiple scenes, fixation point positions, image quality scores (JOD), and corresponding blur (BR direct and BR indirect ), and construct a training set.

[0065] Combined Figure 3 , here, the training scenes include Bar, Square, Pica, and PinkRoom, and these scenes represent various indoor and outdoor environments with different numbers of scattering objects. To ensure the accuracy of the prediction network, all training data comes from the actual final fixation point image, which is obtained after image reconstruction, to achieve end-to-end training, and this method ensures the consistency and correctness of the results; the image quality score represents the expected image quality.

[0066] Step 2: Use the training set to train the rendering parameter prediction network.

[0067] During the training, four scenes were used, with a total of 720,000 frames, covering different direct and indirect illuminations. The Adam optimizer was used, with 15 epochs of training, a learning rate of 1.0e-4, a batch size of 360, and the training time was about 4 hours.

[0068] Step 3: Use the new data of 4 scenes and the new scene BlueRoom to test the trained rendering parameter prediction network and evaluate the generalization ability of the network.

[0069] Finally, the deviation of the network prediction result is less than 0.5 JOD, far lower than the human eye perceivable threshold (JOD = 1). The difference in the blurring rate in the same video sequence is very small. To further reduce the calculation cost and improve the efficiency, this embodiment supports calling the prediction network once every 10 - 15 frames.

[0070] S3. Use the direct illumination rendering parameters and the indirect illumination rendering parameters to construct a direct illumination perception model and an indirect illumination perception model respectively.

[0071] Specifically, use the comparison result of the radius and eccentricity of the central region of the image, and construct a direct illumination perception model according to the direct illumination rendering parameters, the truncation value of the outer region of the image, and the sampling factor of the central region; the direct illumination perception model is expressed as:

[0072]

[0073] In the formula, S0 represents the sampling factor of the central region of the image, S max represents the truncation value of the outer region of the image, r fovea represents the radius size of the central region of the image, θ represents the eccentricity, BR direct represents the blurring rate of the direct illumination, which can be understood as the degree of blurring that increases from the central region to the outer region with the increase of the eccentricity.

[0074] Use the comparison result of the radius and eccentricity of the central region of the image, and construct an indirect illumination perception model according to the indirect illumination rendering parameters, the truncation value of the outer region of the image, and the sampling factor of the central region; the indirect illumination perception model is expressed as:

[0075]

[0076] In the formula, BR indirect represents the blurring rate of the indirect illumination. Here, the sampling factor of the central region of the image, the truncation value of the outer region of the image, the radius size of the central region of the image, and the eccentricity are default parameters in the construction of the perception model.

[0077] Combined with Figure 4 , for a scene containing more diffuse reflection objects, the indirect illumination blurring rate BR indirect is often higher than the direct illumination blurring rate BR direct .

[0078] S4. Guided by the direct illumination perception model and the indirect illumination perception model, use a lightweight image reconstruction network to obtain a fixation point rendering image. Specifically, it includes:

[0079] S401. Calculate the sampling factor at each pixel position in the direct lighting image using the direct lighting perception model, and generate a direct lighting binary mask to perform direct lighting sparse rendering and obtain a direct lighting sparse rendering image; Calculate the sampling factor at each pixel position in the indirect lighting image using the indirect lighting perception model, and generate an indirect lighting binary mask to perform indirect lighting sparse rendering and obtain an indirect lighting sparse rendering image.

[0080] The generation process of direct lighting sparse rendering images and indirect lighting sparse rendering images is the same, only the processing objects are different. Next, taking direct lighting as an example, the specific process is explained in detail.

[0081] First, the sampling factor s at each pixel p in the direct illumination image is calculated by the direct illumination perception model p , based on the sampling factor s p Calculate the sampling threshold t for each pixel p , expressed as:

[0082] t p =b / s p ;

[0083] Where b represents the weight of the sampling factor and threshold, which is set to 1 based on experience.

[0084] Threshold t p Determines whether each pixel is sampled and rendered.

[0085] Then, for each pixel p, generate a random noise value N between 0 and 1 p , the noise value N p With threshold t p Compare to determine the mask value M p Specifically, if N p <t p , then the mask value M p Set to 1, otherwise 0, and the above process forms Figure 5 The binary sampling mask shown.

[0086] Finally, the directly illuminated image is sparsely rendered based on the binary sampling mask. Specifically, only M p = 1 are used for rendering, which ensures that rendering resources are focused on the most perceptually important areas of the image.

[0087] S402, reconstructing the direct lighting sparse rendering image and the indirect lighting sparse rendering image through a lightweight image reconstruction network, completing the missing information, and obtaining the direct lighting foveated rendering image and the indirect lighting foveated rendering image.

[0088] CombinationFigure 6 , specifically, the lightweight image reconstruction network utilizes a two-stage hybrid architecture based on the W-Net framework, including an image reconstruction sub-network D for coarse image reconstruction and an image refinement sub-network K for image refinement. The image reconstruction sub-network D includes three consecutive first convolutional units (encoders), three second convolutional units, and one third convolutional unit (the first second convolutional unit is the central layer, and the rest are decoders). Each first convolutional unit consists of two convolutional blocks of equal depth and an average pooling layer connected in sequence. Each second convolutional block consists of two convolutional blocks of equal depth and one upsampling module connected in sequence. The third convolutional unit consists of two convolutional blocks of equal depth; the image refinement sub-network K includes three consecutive fourth convolutional units (encoders), three fifth convolutional units, and one convolutional block (the first fifth convolutional unit is the central layer, and the rest are decoders). The fourth convolutional unit consists of one convolutional block and an average pooling layer. The fifth convolutional unit consists of one convolutional block and an upsampling layer; each convolutional block includes one convolutional layer and a ReLU activation function.

[0089] The output H of the first second convolutional unit d1 serves as the input to the third fourth convolutional unit and the second fifth convolutional unit. The output H of the second second convolutional unit d2 serves as the input to the second fourth convolutional unit and the third fifth convolutional unit. The output H of the third second convolutional unit d3 serves as the input to the first fourth convolutional unit and the output of the last convolutional block in the image refinement sub-network K; meanwhile, the final output O of the image reconstruction sub-network D d serves as the initial input to the image refinement sub-network K.

[0090] The image refinement sub-network K predicts multi-scale convolutional kernels in the encoder and decoder stages and applies them to the output of the image reconstruction sub-network. Each of its blocks has only one convolutional layer for predicting the kernel, and the hidden state H of the image reconstruction sub-network in the decoder stage is utilized when predicting the kernel. d ; Recurrent connections are also introduced in the network to transfer the output hidden state between the decoder blocks of the image reconstruction sub-network and to transfer the output of the image reconstruction sub-network back to the input layer. The current and recurrent states are combined through concatenation operations in the channel dimension and appropriate upsampling to help reconstruct temporally stable image sequences.

[0091] Furthermore, the lightweight image reconstruction network performs lightweight design on the output channels of the encoder, decoder, and central layer in the image reconstruction sub-network to achieve a more compact feature representation while reducing the computational overhead. Similarly, the convolutional layers and decoders in the image refinement sub-network are also designed in a similar way to ensure more efficient resource utilization, improved inference speed, reduced memory consumption, and without affecting the ability to capture key image features.

[0092] In this embodiment, the number of channels is maintained at 16 - 20, and an inference speed of 3 ms can be achieved.

[0093] Furthermore, during the training process of the lightweight image reconstruction network, recursive connections are introduced in multiple parts of the lightweight image reconstruction network to accumulate state information. This accumulated information helps to reconstruct a temporally stable image sequence. The decoder module in the image reconstruction sub - network passes the output hidden state of the current block back to the input of the next training step through recursive connections; at a broader level, the output O of the image reconstruction sub - network d will be passed back to its input layer as part of the data for the next run. The current state and the recursive state are combined through a concatenation operation along the channel dimension, and appropriate upsampling is applied to make all inputs compatible with subsequent operations.

[0094] S403. Couple the direct - illumination fixation - point rendering image and the indirect - illumination fixation - point rendering image to generate a fixation - point rendering image.

[0095] Specifically, directly add the obtained direct - illumination fixation - point rendering image and the indirect - illumination fixation - point rendering image to obtain the final fixation - point rendering image. As shown in the example Figure 7 shown, it has no difference in perceptual quality from the full - resolution image and the traditional fixation - point rendering image, while reducing the computational cost.

[0096] Embodiment Two

[0097] This embodiment discloses a fixation - point rendering system based on global illumination decomposition, including:

[0098] An illumination decomposition module, configured to: obtain scene data and perform illumination decomposition to generate a direct - illumination image and an indirect - illumination image;

[0099] A fixation - point rendering module, configured to: obtain a fixation - point position and an expected image quality, and combine them with the direct - illumination image and the indirect - illumination image to determine direct - illumination rendering parameters and indirect - illumination rendering parameters;

[0100] Use the direct - illumination rendering parameters and the indirect - illumination rendering parameters to respectively construct a direct - illumination perception model and an indirect - illumination perception model; guided by the direct - illumination perception model and the indirect - illumination perception model, use a lightweight image reconstruction network to obtain a fixation - point rendering image.

[0101] It should be noted here that the above-mentioned global illumination decomposition module and fixation point rendering module correspond to the steps in the first embodiment. The examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in the first embodiment. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0102] Embodiment 3

[0103] Embodiment 3 of the present invention provides an electronic device, including a memory, a processor, and computer instructions stored on the memory and running on the processor. When the computer instructions are run by the processor, the steps of the above-mentioned fixation point rendering method based on global illumination decomposition are completed.

[0104] Embodiment 4

[0105] Embodiment 4 of the present invention provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned fixation point rendering method based on global illumination decomposition are completed.

[0106] Embodiment 5

[0107] Embodiment 5 of the present invention provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the steps of the above-mentioned fixation point rendering method based on global illumination decomposition are implemented.

[0108] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate for implementing in the process Figure 1 a process or multiple processes and / or blocks Figure 1 a block or multiple blocks the device with the functions specified.

[0109] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements in the process Figure 1 a process or multiple processes and / or blocks Figure 1 a block or multiple blocks the functions specified.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing device, where they perform a series of operational steps to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or steps for implementing the functions specified in multiple blocks.

[0111] In the above embodiments, the descriptions of the various embodiments have their own emphases. For parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0112] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A foveated rendering method based on global illumination decomposition, characterized in that: include: Acquire scene data and perform lighting decomposition to generate direct lighting images and indirect lighting images; Obtaining a gaze point position and an expected image quality, and combining them with the direct illumination image and the indirect illumination image to determine a direct illumination rendering parameter and an indirect illumination rendering parameter; Direct lighting rendering parameters and indirect lighting rendering parameters are used to construct direct lighting perception models and indirect lighting perception models respectively. Guided by the direct lighting perception model and the indirect lighting perception model, a lightweight image reconstruction network is used to obtain the foveated rendering image.

2. The foveated rendering method based on global illumination decomposition according to claim 1, characterized in that: The gaze point position and the expected image quality are combined with the direct illumination image and the indirect illumination image to determine the direct illumination rendering parameters and the indirect illumination rendering parameters as follows: The gaze point position, the expected image quality, the direct illumination image and the indirect illumination image are processed through a trained rendering parameter prediction network to obtain direct illumination rendering parameters and indirect illumination rendering parameters.

3. The foveated rendering method based on global illumination decomposition according to claim 2, characterized in that: Processing the gaze point position, the expected image quality, the direct illumination image, and the indirect illumination image through a trained rendering parameter prediction network includes: splicing the direct illumination image and the indirect illumination image, and performing dimensionality reduction and feature extraction on the splicing result to obtain an illumination feature vector; The illumination feature vector is concatenated with the gaze point position and bias information, and the concatenated results are processed sequentially through the normalization layer, activation function and fully connected layer. The direct illumination rendering parameters and indirect illumination rendering parameters are generated with the expected image quality as the reference constraint.

4. The foveated rendering method based on global illumination decomposition according to claim 1, characterized in that: The using of direct lighting rendering parameters and indirect lighting rendering parameters to respectively construct a direct lighting perception model and an indirect lighting perception model comprises: By using the comparison results of the central area radius and eccentricity of the image, a direct lighting perception model is constructed according to the direct lighting rendering parameters, the cutoff value of the peripheral area of ​​the image, and the sampling factor of the central area; By comparing the radius and eccentricity of the central area of ​​the image, an indirect lighting perception model is constructed according to the indirect lighting rendering parameters, the cutoff value of the peripheral area of ​​the image, and the sampling factor of the central area.

5. The foveated rendering method based on global illumination decomposition according to claim 1, characterized in that: The method of obtaining a foveated rendering image by using a lightweight image reconstruction network under the guidance of a direct illumination perception model and an indirect illumination perception model includes: The sampling factor at each pixel position in the direct lighting image is calculated using the direct lighting perception model, and a direct lighting binary mask is generated to perform direct lighting sparse rendering to obtain a direct lighting sparse rendering image; the sampling factor at each pixel position in the indirect lighting image is calculated using the indirect lighting perception model, and an indirect lighting binary mask is generated to perform indirect lighting sparse rendering to obtain an indirect lighting sparse rendering image; The direct lighting sparse rendering image and the indirect lighting sparse rendering image are processed respectively by a lightweight image reconstruction network to obtain a direct lighting foveated rendering image and an indirect lighting foveated rendering image; The direct lighting foveated rendering image and the indirect lighting foveated rendering image are coupled to generate a foveated rendering image.

6. The foveated rendering method based on global illumination decomposition according to claim 5, characterized in that: The lightweight image reconstruction network includes an image reconstruction subnetwork and an image refinement subnetwork, the image reconstruction subnetwork includes a plurality of first convolution units, a plurality of second convolution units, and a third convolution unit connected in sequence, and the image refinement subnetwork includes a plurality of fourth convolution units, a plurality of fifth convolution units, and a convolution block connected in sequence.

7. A foveated rendering system based on global illumination decomposition, characterized in that: include: The illumination decomposition module is configured to: obtain scene data and perform illumination decomposition to generate a direct illumination image and an indirect illumination image; A foveated rendering module is configured to: obtain a foveated point position and an expected image quality, and combine them with the direct illumination image and the indirect illumination image to determine a direct illumination rendering parameter and an indirect illumination rendering parameter; Direct lighting rendering parameters and indirect lighting rendering parameters are used to construct direct lighting perception models and indirect lighting perception models respectively. Guided by the direct lighting perception model and the indirect lighting perception model, a lightweight image reconstruction network is used to obtain the foveated rendering image.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the foveated rendering method based on global illumination decomposition as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the foveated rendering method based on global illumination decomposition as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the foveated rendering method based on global illumination decomposition as described in any one of claims 1 to 6 are implemented.

Images