Method, system, medium and electronic device for vr rendering optimization based on eye tracking

Abstract

The application discloses a VR rendering optimization method and system based on eyeball tracking, a medium and an electronic device, and the method comprises the following steps: preprocessing VR data to obtain target data; predicting a gaze center point at the next moment according to the target data and based on a trained bidirectional LSTM time sequence neural network; setting a dynamic resolution mapping of VR rendering according to the gaze center point; setting a plurality of rendering areas of the VR rendering according to the gaze center point; dynamically adjusting target dynamic quantization parameters corresponding to the plurality of rendering areas respectively based on the current network condition; performing VR dynamic rendering at the next moment based on the dynamic resolution mapping and the target dynamic quantization parameters corresponding to each rendering area respectively; therefore, the application allocates rendering tasks in advance according to the predicted gaze center point, so that the user can avoid dizziness; meanwhile, the rendering adopts dynamic resolution mapping, so that the rendering calculation amount can be reduced, and the fragmentation caused by traditional partitioning can be avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of virtual reality rendering, in particular to a VR rendering optimization method and system based on eyeball tracking, a medium and an electronic device. BACKGROUND

[0002] The current virtual reality technology has been widely applied in the fields of building visualization and industrial simulation, but the existing technical solutions have many technical defects, especially in the scene of high resolution and high refresh rate (8K@60fps or above). For example, the local rendering scheme relies on the terminal GPU to complete the full scene calculation, which requires high local device and high device cost, and cannot support 8K@60fps stable operation. In the scene requiring global illumination and dynamic shadow, simplification is often forced; although the cloud rendering can alleviate the terminal computing power pressure, the extreme demand for network bandwidth of high-precision picture transmission becomes a new obstacle. The 8K@60fps video stream encoded by H.266 standard consumes 8Gbps bandwidth, while the actual available bandwidth of most users is less than 2Gbps, forcing the picture to be downgraded to 4K resolution. More seriously, the end-to-end delay of the existing transmission scheme generally exceeds 50ms, and when the user quickly rotates the viewing angle to observe the simulation model details, the picture lag will cause many users to feel dizzy, affecting the use experience.

[0003] Some rendering schemes use the gaze point technology to reduce the computing power demand by resolution grading, but the scheme is not mature. The fixed partition multi-resolution strategy will cause visual fragmentation when used, and the gaze point prediction algorithm is inaccurate or slow, which is difficult to match the viewing angle change rate of the user, resulting in a large deviation between the user's gaze point and the actual picture center, reducing the use experience. SUMMARY

[0004] The present application provides a VR rendering optimization method and system based on eyeball tracking, a medium and an electronic device, which pre-allocates rendering tasks according to the predicted gaze center point, greatly reduces the delay experienced by the user, and avoids dizziness; at the same time, the rendering uses dynamic resolution mapping, which can reduce the rendering calculation amount while avoiding the fragmentation caused by traditional partitioning.

[0005] In a first aspect, a VR rendering optimization method based on eyeball tracking is provided, comprising the following steps:

[0006] Collecting VR data at the current time and pre-processing the VR data to obtain target data;

[0007] According to the target data, and based on the trained bidirectional LSTM time series neural network, predicting the gaze center point at the next time;

[0008] According to the gaze center point, setting the dynamic resolution mapping of VR rendering;

[0009] Setting multiple rendering areas for VR rendering according to the gaze center point;

[0010] Dynamically adjust the target dynamic quantization parameters corresponding to multiple rendering areas based on the current network conditions;

[0011] VR dynamic rendering at the next moment is performed based on the dynamic resolution mapping and the target dynamic quantization parameters corresponding to each rendering area.

[0012] According to the first aspect, in a first possible implementation manner of the first aspect, collecting VR data at the current moment and preprocessing the VR data to obtain target data includes:

[0013] Collect VR images at the current moment and the previous moment, use the pyramid Lucas-Kanade algorithm to calculate adjacent VR images, and obtain the optical flow matrix at the current moment;

[0014] Perform Kalman filtering on the angular velocity vector and acceleration vector of the VR device at the current moment;

[0015] Based on the original gaze vector of the VR device, the unit vector at the current moment is obtained.

[0016] According to the first possible implementation manner of the first aspect, in the second possible implementation manner of the first aspect, predicting the gaze center point at the next moment according to the target data and based on a trained bidirectional LSTM temporal neural network includes:

[0017] According to the current optical flow matrix F t , unit vector V t , angular velocity vector after Kalman filter processing and acceleration vector , get the input vector X t As shown in the following formula:

[0018] ;

[0019] Where, To transform the optical flow matrix F t Flattened to a vector; d is the dimension.

[0020] The trained bidirectional LSTM temporal neural network is used to predict the gaze center point at the next moment according to the input vector.

[0021] According to the first aspect, in a third possible implementation manner of the first aspect, the method for setting a dynamic resolution mapping for VR rendering according to the gaze center point is shown in the following formula:

[0022] ;

[0023] wherein, ;

[0024] In the formula, R(r) is a dynamic resolution map; R max is the highest resolution of the display; r is the pixel radius of the distance from the fixation center point p; is the radian value of the human foveal visual angle; PPC is the screen pixel density; d is the distance from the human eye to the screen; and λ is the attenuation coefficient.

[0025] According to a first aspect, in a fourth possible implementation manner of the first aspect, the step of dynamically adjusting the target dynamic quantization parameters corresponding to the plurality of rendering areas based on the current network condition comprises:

[0026] dynamically setting the target code rate of the rendered picture based on the current real-time available network bandwidth as shown in the following formula:

[0027] ;

[0028] dynamically adjusting the basic dynamic quantization parameters corresponding to the plurality of rendering areas based on the target code rate as shown in the following formula:

[0029] ;

[0030] calibrating the basic dynamic quantization parameters corresponding to each rendering area to obtain the target dynamic quantization parameters corresponding to each rendering area as shown in the following formula:

[0031] ;

[0032] In the formula, is the highest code rate required by the current rendered picture; K and C are both encoder constants; α is the code rate weight value corresponding to different rendering areas; β is an empirical scaling coefficient; and T is the texture complexity corresponding to different rendering areas.

[0033] According to a first aspect, in a fifth possible implementation manner of the first aspect, the step of setting the plurality of rendering areas of the VR rendering according to the fixation center point comprises:

[0034] setting the visual angle range corresponding to the focus area of the VR rendering as less than or equal to a first preset angle according to the fixation center point;

[0035] ​​​The visual angle range corresponding to the transition area of the VR rendering is greater than a first preset angle and less than a second preset angle.

[0036] The visual angle range corresponding to the edge area of the VR rendering is greater than or equal to a third preset angle.

[0037] According to a first aspect, in a sixth possible implementation manner of the first aspect, the trained bidirectional LSTM time sequence neural network comprises:

[0038] VR data under a previous sequence at a current moment is collected, and the VR data under each previous sequence moment is preprocessed;

[0039] An input vector is constructed for each previous sequence moment of the preprocessed VR data.

[0040] The bidirectional LSTM time sequence neural network is trained by using the input vectors corresponding to the current moment and the previous sequence moment, to obtain the trained bidirectional LSTM time sequence neural network.

[0041] According to a second aspect, a VR rendering optimization system based on eye tracking is provided, comprising:

[0042] A preprocessing module is configured to collect VR data at a current moment, and preprocess the VR data to obtain target data.

[0043] A prediction module is in communication connection with the preprocessing module, and is configured to predict a gaze center point at a next moment according to the target data and based on a trained bidirectional LSTM time sequence neural network.

[0044] A mapping module is in communication connection with the prediction module, and is configured to set a dynamic resolution mapping of VR rendering according to the gaze center point.

[0045] A rendering area setting module is in communication connection with the prediction module, and is configured to set a plurality of rendering areas of VR rendering according to the gaze center point.

[0046] A quantization parameter module is in communication connection with the rendering area setting module, and is configured to dynamically adjust target dynamic quantization parameters corresponding to the plurality of rendering areas based on a current network condition.

[0047] A dynamic rendering module is in communication connection with the mapping module and the quantization parameter module, and is configured to perform VR dynamic rendering at a next moment based on the dynamic resolution mapping and the target dynamic quantization parameters corresponding to each rendering area.

[0048] In a third aspect, a computer readable storage medium is provided, and a computer program is stored in the computer readable storage medium, and the computer program is executed by a processor to implement the method for optimizing VR rendering based on eye tracking as described above.

[0049] In a fourth aspect, an electronic device is provided, and the electronic device comprises a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, and the processor implements the method for optimizing VR rendering based on eye tracking as described above when executing the computer program.

[0050] Compared with the prior art, the advantages of the present application are as follows: the gaze center point at the next moment is predicted using a bidirectional LSTM time sequence neural network, and the rendering task is allocated in advance according to the predicted center point, so that the delay perceived by the user is greatly reduced, and the dizziness is avoided. At the same time, dynamic resolution mapping is used for rendering, which can reduce the rendering calculation amount while avoiding the fragmentation caused by traditional partitioning. Moreover, the real-time encoding of network transmission also controls the adaptive quantization parameter according to the gaze area, reduces the code rate of the filtering area and the edge area of cloud transmission, and reduces the bandwidth occupation. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flowchart of an embodiment of a VR rendering method based on eye tracking and cloud edge cooperation of the present application;

[0052] Figure 2 is a structural diagram of a VR rendering system based on eye tracking and cloud edge cooperation of the present application. DETAILED DESCRIPTION

[0053] Reference will now be made in detail to the embodiments of the present application, examples of which are illustrated in the accompanying drawings. While the present application will be described in conjunction with the embodiments, it will be understood that the present application is not limited to the embodiments. Rather, the present application covers any modifications, equivalents, and alternatives falling within the spirit and scope of the present application as defined by the appended claims. It should be noted that the method steps described herein can all be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of both.

[0054] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below in conjunction with the drawings and specific embodiments.

[0055] Note: The examples to be introduced next are only one specific example, and are not as a limitation on the embodiments of the present application, which must be the specific steps, values, conditions, data, sequences, etc. below. Those skilled in the art can use the concept of the present application to construct more embodiments not mentioned in the present specification by reading the present specification.

[0056] Referring to Figure 1 As shown in the drawings, the embodiment of the application provides a VR rendering method based on eye tracking and cloud edge cooperation, comprising the following steps:

[0057] S100, collecting VR data at the current moment, and pre-processing the VR data to obtain target data;

[0058] S200, predicting the gaze center point at the next moment according to the target data and based on a trained bidirectional LSTM time series neural network;

[0059] S300, setting a dynamic resolution mapping of VR rendering according to the gaze center point;

[0060] S400, setting a plurality of rendering areas of VR rendering;

[0061] S500, dynamically adjusting target dynamic quantization parameters corresponding to the plurality of rendering areas respectively based on the current network condition;

[0062] S600, performing VR dynamic rendering at the next moment based on the dynamic resolution mapping and the target dynamic quantization parameters corresponding to each rendering area respectively.

[0063] Specifically, in the embodiment, the application uses a bidirectional LSTM time series neural network to predict the gaze center point at the next moment, and allocates rendering tasks in advance according to the predicted center point, so that the delay perceived by the user is greatly reduced, and dizziness is avoided. At the same time, the rendering adopts dynamic resolution mapping, which can reduce the rendering calculation amount while avoiding the fragmentation caused by traditional partitioning. Moreover, the real-time encoding of network transmission also controls the adaptive quantization parameters according to the gaze area, reduces the code rate of the filtering area and the edge area of cloud transmission, and reduces the bandwidth occupation.

[0064] Preferably, in another embodiment of the application, the S100, collecting VR data at the current moment, and pre-processing the VR data to obtain target data, comprises:

[0065] S110, collecting VR images at the current moment and the previous moment, and calculating the adjacent VR images by using a pyramid Lucas-Kanade algorithm to obtain a current optical flow matrix;

[0066] S120, performing Kalman filtering processing on the angular velocity vector and the acceleration vector of the VR device at the current moment;

[0067] S130, obtaining a unit vector at the current moment based on the original gaze vector of the VR device.

[0068] Specifically, in this embodiment, the pyramid Lucas-Kanade algorithm is a classic algorithm for optical flow estimation in the field of computer vision. The Lucas-Kanade algorithm is based on three basic assumptions: brightness constancy, small motion and spatial consistency. The pyramid Lucas-Kanade algorithm uses image pyramids to handle larger motion on this basis. It constructs the image into a pyramid structure of different resolutions, starting from the top layer (low resolution) of the pyramid to estimate the optical flow, and then gradually propagates the results to the lower layer (high resolution), which can handle larger displacements while reducing computational complexity.

[0069] The adjacent frame images of the user's eyeball movement are collected by the visual sensor, and the pyramid Lucas-Kanade algorithm is used to calculate the adjacent frames to obtain the optical flow matrix F of time t t .

[0070] At the same time, the angular velocity vector ω and the acceleration vector α of the VR device at the current time are obtained from the IMU (inertial measurement unit), and the formula for Kalman filtering and denoising processing of ω and α is as follows:

[0071] ;

[0072] ;

[0073] wherein, , is the Kalman gain coefficient.

[0074] The original gaze vector is obtained from the VR device , and the formula for converting it into a unit vector V is as follows:

[0075] ;

[0076] .

[0077] Preferably, in another embodiment of the present application, the S200 predicts the gaze center point at the next time according to the target data and based on the trained bidirectional LSTM time series neural network, comprising:

[0078] According to the optical flow matrix F t , the unit vector V t , the angular velocity vector ω and the acceleration vector α filtered by Kalman filtering, the input vector X t is obtained as follows:

[0079] ;

[0080] wherein, To flatten the optical flow matrix F t to a vector; d is the dimension.

[0081] ;

[0082] The image width and height are h and w, respectively.

[0083] The trained bidirectional LSTM time series neural network is used to predict the gaze center point at the next time according to the input vector.

[0084] Preferably, in another embodiment of the present application, the trained bidirectional LSTM time series neural network comprises:

[0085] Collect VR data under the previous sequence at the current time, and preprocess the VR data under each previous sequence time;

[0086] An input vector is constructed for each previous sequence time after preprocessing.

[0087] The bidirectional LSTM time series neural network is trained using the corresponding input vectors at the current time and the previous time, and a trained bidirectional LSTM time series neural network is obtained.

[0088] Specifically, in the present embodiment,

[0089] Collect VR data under the previous sequence at the current time, that is, construct input vectors X for the last n time points at the current time, that is, , ,… Input the n input vectors X into the bidirectional LSTM time series neural network to predict the gaze center point P t+1 .

[0090] For data collection and arrangement: collect input vectors X at n time points and arrange them in time sequence to form a training data set: {X1, X2, …, XT}, where T is the total time step.

[0091] Model selection and training: select a bidirectional LSTM time series neural network as a prediction model, and input the arranged training data set into the model for training. The bidirectional LSTM time series neural network can fully utilize the spatio-temporal features in historical data to improve the prediction accuracy. During the training process, the parameters and hyperparameters of the model are adjusted to enable the model to accurately predict the gaze center point at the next time.

[0092] Training data set model selection and training: select a bidirectional LSTM time series neural network as a prediction model. The bidirectional LSTM is composed of a forward LSTM and a backward LSTM, and the update formula of its hidden state is:

[0093]

[0094]

[0095] The output of the final bidirectional LSTM time series neural network is obtained by concatenating the forward and backward hidden states, that is:

[0096]

[0097] The processed training data is input into the bidirectional LSTM time series neural network for training. By minimizing the loss function (such as mean square error), the model parameters are optimized to enable the model to accurately predict the gaze center point at the next time.

[0098] Post-processing optimization: the predicted gaze center point P t+1 is post-processed, such as smoothing, to eliminate noise and outliers in the prediction results, improve the stability and reliability of the prediction results, and ultimately obtain accurate gaze center point prediction results, providing strong support for subsequent virtual reality applications.

[0099] The predicted gaze center point P t+1 is smoothed, for example, using a moving average filter:

[0100] .

[0101] where m is the size of the smoothing window, which can be adjusted according to actual needs to eliminate noise and outliers in the prediction results, improve the stability and reliability of the prediction results.

[0102] Preferably, in another embodiment of the present application,

[0103] Specifically, in this embodiment, the method for setting the dynamic resolution mapping of VR rendering according to the gaze center point S300 is as follows:

[0104] ;

[0105] where, ;

[0106] In the formula, R(r) is the dynamic resolution mapping; R max is the highest resolution of the display; r is the pixel radius from the gaze center point p; is the radian value of the human foveal viewing angle; PPC is the screen pixel density; d is the distance from the human eye to the screen; and λ is the attenuation coefficient.

[0107] Dynamic Resolution Scaling (DRS) is a technique for adjusting rendering resolution in real time, whose core goal is to reduce rendering computation while maintaining visual quality. It dynamically adjusts the resolution of different areas of the picture intelligently, reducing GPU load and avoiding the fragmentation caused by traditional static partition methods (such as fixed area low resolution rendering).

[0108] Preferably, in another embodiment of the present application, the S400, the setting of the plurality of rendering areas of the VR rendering according to the gaze center point comprises:

[0109] S410, setting the view angle range corresponding to the focus area of the VR rendering as less than or equal to a first preset angle according to the gaze center point;

[0110] S420, setting the view angle range corresponding to the transition area of the VR rendering as greater than the first preset angle and less than a second preset angle;

[0111] S430, setting the view angle range corresponding to the edge area of the VR rendering as greater than or equal to a third preset angle.

[0112] Specifically, in the present embodiment, only when the gaze center point of the next moment is determined, the VR rendering picture can be dynamically divided into three areas, so that the rendering task can be allocated in advance, the delay perceived by the user is greatly reduced, and the dizziness is avoided.

[0113] Focus area: view angle range (For example ), area:

[0114]

[0115] Transition area: view angle range (For example );

[0116] Edge area: ;

[0117] The picture of the focus area is rendered using the edge computing device D, and the pictures of the filtering area and the edge area are rendered using the cloud C. At the same time, global illumination and physical engine simulation are also performed on the cloud C.

[0118] Therefore, combining the characteristics of edge computing and cloud computing, according to the change of the gaze area of the picture, the rendering task of the focus area is allocated to the edge computing terminal, the transition area and the edge area are allocated to the cloud, and the cloud is responsible for global illumination and physical engine simulation, which can reduce the computing pressure of the edge terminal while ensuring the real-time performance of the user focus area rendering.

[0119] Preferably, in another embodiment of the present application, the S500 dynamically adjusts the target dynamic quantization parameters corresponding to the plurality of rendering areas based on the current network condition, comprising:

[0120] based on the current real-time available network bandwidth , dynamically setting the target code rate of the rendering picture as follows:

[0121] ;

[0122] according to the target code rate , dynamically adjusting the basic dynamic quantization parameters corresponding to the plurality of rendering areas as follows:

[0123] ;

[0124] calibrating the basic dynamic quantization parameters corresponding to each rendering area to obtain the target dynamic quantization parameters corresponding to each rendering area as follows:

[0125] ;

[0126] wherein, is the highest code rate required by the current rendering picture; K and C are both encoder constants; a is the code rate weight value corresponding to different rendering areas (for example, 0.7 for the focus area, 0.2 for the transition area, and 0.1 for the edge area); b is an empirical scaling coefficient; and T is the texture complexity (image gradient) corresponding to different rendering areas.

[0127] Therefore, the network transmission real-time encoding can also be adaptively quantized according to the code rate weight value corresponding to different rendering areas, reducing the code rate of the filtering area and the edge area in cloud transmission, and reducing the bandwidth occupation.

[0128] Referring to FIG. 1, Figure 2 the embodiment of the present application provides a VR rendering based on eye tracking and cloud edge cooperation 100, comprising:

[0129] a preprocessing module 110, configured to collect VR data at the current time and preprocess the VR data to obtain target data;

[0130] a prediction module 120, in communication connection with the preprocessing module 110, configured to predict the gaze center point at the next time according to the target data and based on the trained bidirectional LSTM time series neural network;

[0131] The mapping module 130 is connected with the prediction module 120, and is configured to set a dynamic resolution mapping of VR rendering according to the gaze center point.

[0132] The rendering area setting module 140 is connected with the prediction module 120, and is configured to set a plurality of rendering areas of VR rendering according to the gaze center point.

[0133] The quantization parameter module 150 is connected with the rendering area setting module 140, and is configured to dynamically adjust target dynamic quantization parameters corresponding to the plurality of rendering areas respectively based on a current network condition.

[0134] The dynamic rendering module 160 is connected with the mapping module 130 and the quantization parameter module 150, and is configured to perform VR dynamic rendering at a next time based on the dynamic resolution mapping and the target dynamic quantization parameters corresponding to each rendering area respectively.

[0135] In summary, the beneficial effects of the present application are as follows: the method uses a multi-modal data fusion model to predict the gaze center point within a certain time window, and allocates rendering tasks in advance according to the predicted center point, so that the delay experienced by the user is greatly reduced, and the dizziness is avoided. At the same time, the rendering adopts dynamic resolution mapping, which can reduce the rendering calculation amount while avoiding the fragmentation caused by traditional partitioning. Combining the characteristics of edge computing and cloud computing, according to the change of the gaze area of the picture, the rendering task of the focus area is allocated to the edge computing terminal, the transition area and the edge area are allocated to the cloud, and the cloud is responsible for global illumination and physical engine simulation, which can reduce the computing pressure of the edge terminal while ensuring the real-time performance of the rendering of the user focus area. The network transmission real-time encoding also controls the adaptive quantization parameter according to the gaze area, reduces the code rate of the filtering area and the edge area of the cloud transmission, and reduces the bandwidth occupation.

[0136] Specifically, the present embodiment corresponds to the above method embodiment one by one, and the functions of each module have been described in detail in the corresponding method embodiment, so they will not be repeated here.

[0137] Based on the same inventive concept, the present embodiment also provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement all method steps or part of the method steps of the above method.

[0138] The present application implements all or part of the processes in the above method, and can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0139] Based on the same inventive concept, the embodiments of the present application also provide an electronic device, including a memory and a processor, the memory stores a computer program running on the processor, and the processor implements all or part of the method steps of the above method when executing the computer program.

[0140] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and connects all parts of the computer device through various interfaces and lines.

[0141] The memory can be used to store computer programs and / or modules, and the processor can realize various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application program (for example, a sound playing function, an image playing function, etc.) required by a function; and the data storage area can store data (for example, audio data, video data, etc.) created according to the use of the mobile phone. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0142] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, a server or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer usable program codes.

[0143] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), servers and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks.

[0144] These computer program instructions can also be stored in a computer readable memory capable of guiding a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer readable memory produce a product including instruction devices, which implement the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks.

[0145] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable devices provide processes for implementing the flow Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or steps of the functions specified in the flow

[0146] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A VR rendering optimization method based on eye tracking, characterized in that, The method comprises the following steps: Collecting VR data at the current moment, and preprocessing the VR data to obtain target data; According to the target data, and based on a trained bidirectional LSTM time series neural network, predicting a gaze center point at the next moment; According to the gaze center point, setting a dynamic resolution mapping for VR rendering; According to the gaze center point, setting a plurality of rendering areas for VR rendering; Based on the current network situation, dynamically adjusting target dynamic quantization parameters corresponding to the plurality of rendering areas respectively; Based on the dynamic resolution mapping and the target dynamic quantization parameters corresponding to each rendering area respectively, performing VR dynamic rendering at the next moment; The method for setting the dynamic resolution mapping for VR rendering according to the gaze center point is as shown in the following formula: ; wherein ; where R(r) is the dynamic resolution map; R max is the highest resolution of the display; r is the pixel radius from the fixation point p; is the radian value of the human foveal angle; PPC is the screen pixel density; d is the distance from the human eye to the screen; and λ is the attenuation coefficient. The method for dynamically adjusting the target dynamic quantization parameters corresponding to the plurality of rendering areas respectively based on the current network situation comprises: based on the network bandwidth currently available in real time dynamically setting a target bit rate for the rendered picture as follows: ; According to the target code rate , dynamically adjusting the basic dynamic quantization parameter QP corresponding to each of the plurality of rendering areas base As shown in the following formula: ; a basic dynamic quantization parameter QP corresponding to each rendering area respectively base calibration to obtain a target dynamic quantization parameter QP corresponding to each rendering area respectively final as shown in the following formula: ; In the formula, is the highest code rate required for the current rendering picture; K and C are both encoder constants; a is the code rate weight value corresponding to different rendering areas; β is an empirical scaling coefficient; and T is the texture complexity corresponding to different rendering areas. The method for setting the plurality of rendering areas for VR rendering according to the gaze center point comprises: According to the gaze center point, setting a viewing angle range corresponding to a focus area of VR rendering to be less than or equal to a first preset angle; According to the gaze center point, setting a viewing angle range corresponding to a transition area of VR rendering to be greater than the first preset angle and less than a second preset angle; According to the gaze center point, setting a viewing angle range corresponding to an edge area of VR rendering to be greater than or equal to a third preset angle.

2. The eye tracking based VR rendering optimization method of claim 1, wherein, The method for collecting VR data at the current moment, and preprocessing the VR data to obtain target data comprises: Collecting VR images at the current moment and at the previous moment, and calculating adjacent VR images by using a pyramid Lucas-Kanade algorithm to obtain an optical flow matrix at the current moment; Performing Kalman filtering processing on an angular velocity vector and an acceleration vector of a VR device at the current moment; Based on an original gaze vector of the VR device, obtaining a unit vector at the current moment.

3. The eye tracking based VR rendering optimization method of claim 2, wherein, The method for predicting a gaze center point at the next moment according to the target data and based on a trained bidirectional LSTM time series neural network comprises: a flow matrix F at a current time t , a unit vector V t , an angular velocity vector after Kalman filtering and an acceleration vector , an input vector X t is obtained as shown in the following equation: ; wherein to flatten the optical flow matrix F t to flatten the optical flow matrix F to flatten the optical flow matrix F Using the trained bidirectional LSTM time series neural network to predict the gaze center point at the next moment according to the input vector.

4. The eye tracking based VR rendering optimization method of claim 1, wherein, The trained bidirectional LSTM time series neural network comprises: Collecting VR data at a previous sequence at the current moment, and preprocessing the VR data at each previous sequence moment; Constructing an input vector for each previous sequence moment of the preprocessed VR data; Using the input vectors corresponding to the current moment and the previous sequence moment to train the bidirectional LSTM time series neural network, to obtain the trained bidirectional LSTM time series neural network.

5. A VR rendering optimization system based on eye tracking, characterized in that, The method comprises: A preprocessing module is configured to collect VR data at the current moment, and preprocess the VR data to obtain target data; A prediction module is in communication connection with the preprocessing module, and is configured to predict a gaze center point at the next moment according to the target data and based on a trained bidirectional LSTM time series neural network; A mapping module is in communication connection with the prediction module, and is configured to set a dynamic resolution mapping for VR rendering according to the gaze center point; A rendering area setting module is in communication connection with the prediction module, and is configured to set a plurality of rendering areas for VR rendering according to the gaze center point; A quantization parameter module, in communication with the rendering area setting module, is configured to dynamically adjust target dynamic quantization parameters corresponding to the plurality of rendering areas based on the current network condition; Furthermore, A dynamic rendering module, in communication with the mapping module and the quantization parameter module, is configured to perform VR dynamic rendering at the next time based on the dynamic resolution mapping and the target dynamic quantization parameters corresponding to each rendering area. The method for setting the dynamic resolution mapping of VR rendering according to the gaze center point is shown in the following formula: ; wherein ; where R(r) is the dynamic resolution map; R max is the highest resolution of the display; r is the pixel radius from the fixation point p; is the radian value of the human foveal angle; PPC is the screen pixel density; d is the distance from the human eye to the screen; and λ is the attenuation coefficient. The method for dynamically adjusting the target dynamic quantization parameters corresponding to the plurality of rendering areas based on the current network condition includes: based on the network bandwidth currently available in real time , dynamically setting a target bit rate of a rendered picture as shown in the following equation: ; According to the target code rate , dynamically adjusting the basic dynamic quantization parameter QP corresponding to each of the plurality of rendering areas base As shown in the following formula: ; a basic dynamic quantization parameter QP corresponding to each rendering area respectively base calibration to obtain a target dynamic quantization parameter QP corresponding to each rendering area respectively final as shown in the following formula: ; In the formula, is the highest code rate required for the current rendering picture; K and C are both encoder constants; a is the code rate weight value corresponding to different rendering areas; β is an empirical scaling coefficient; and T is the texture complexity corresponding to different rendering areas. The method for setting the plurality of rendering areas of VR rendering according to the gaze center point includes: According to the gaze center point, the visual angle range corresponding to the focus area of VR rendering is set to be less than or equal to a first preset angle. The visual angle range corresponding to the transition area of VR rendering is set to be greater than the first preset angle and less than a second preset angle. The visual angle range corresponding to the edge area of VR rendering is set to be greater than or equal to a third preset angle.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the VR rendering optimization method based on eye tracking according to any one of claims 1 to 4.

7. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and operable on the processor, characterized in that, The processor, when running the computer program, implements the VR rendering optimization method based on eye tracking according to any one of claims 1 to 4.

Images