Virtual reality picture rendering method and device

Abstract

The invention provides a virtual reality picture rendering method and device, which can divide a picture region of a virtual reality device into a plurality of processing regions, different processing regions correspond to different field angle ranges and preset picture rendering specifications, and then differentiated rendering is performed on a virtual reality picture. According to the method, a sight line concentration area which is more concerned by a user adopts a higher picture rendering specification, and an area which is less concerned by the user adopts the lowest picture rendering specification, so that the computing resources required by virtual reality rendering are effectively reduced while the picture effect of the high-attention area of the user is ensured.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and more specifically, to a virtual reality image rendering method and apparatus. Background Technology

[0002] Virtual Reality (VR) is a technology that uses computer technology to generate a three-dimensional virtual environment and allows users to interact with it in an immersive way through devices such as head-mounted displays, using visual, auditory, and even tactile senses. Users can interact with the virtual scene through head and hand movements, obtaining a near-realistic immersive experience.

[0003] VR devices typically require high resolution and refresh rates, necessitating strong computing power. Some VR devices with insufficient computing power (such as VR glasses, stand-alone VR headsets, VR devices implemented via smartphones, or PC-based VR devices with weak computing capabilities), especially mobile VR devices, suffer from limited computing and graphics processing capabilities due to constraints such as size, power consumption, or processor strength. This makes it difficult to support sustained high-load rendering of complex scenes, resulting in poor VR content display quality. Summary of the Invention

[0004] In view of this, this application provides a virtual reality screen rendering method and apparatus to solve the problem of poor VR content display effect caused by the weak computing power of VR devices, especially mobile VR devices.

[0005] Specifically, this application is implemented through the following technical solution: In a first aspect, embodiments of this application provide a virtual reality scene rendering method, including: The screen area of ​​the virtual reality device is divided into multiple processing areas; each of the multiple processing areas corresponds to a different field of view range. For a future target frame, the predicted landing position of the target user's eye gaze in the screen area is predicted to obtain the predicted landing position of the target frame. From multiple different preset image rendering specifications, determine the target image rendering specification corresponding to the processing area where the predicted landing point is located; and determine the gaze concentration area corresponding to the predicted landing point. The target image rendering specification is used as the image rendering specification for the area where the view is focused; the lowest image rendering specification is used as the image rendering specification for other image areas besides the area where the view is focused. In response to obtaining the actual landing point of the target user's eye in the screen area under the target frame, and if the actual landing point matches the predicted landing point, the virtual reality screen of the target frame is displayed based on the screen rendering specifications corresponding to the target frame.

[0006] Optionally, the method further includes: Based on the actual landing positions of multiple historical frames, the eye trajectory of the target user is predicted, and at least one user interest region is determined from the screen area based on the prediction results. For any user interest region, determine the transition screen rendering specifications of the user interest region in each transition frame corresponding to the preset transition time; the rendering quality of the transition screen rendering specifications is higher than the rendering quality of the lowest screen rendering specifications, but lower than the preset screen rendering specifications corresponding to the user interest region. The rendering specifications of the transition frame are updated using the aforementioned rendering specifications, and the virtual reality image corresponding to the transition frame is displayed based on the updated rendering specifications.

[0007] Optionally, determining the rendering specifications of the transition screen for the user's region of interest in each transition frame corresponding to the transition time includes: For any of the transition frames, a decay weight is determined based on the position of the transition frame in the preset transition time and a first time constant; and an enhancement weight is determined based on the position of the transition frame in the preset transition time and a second time constant. Based on the product of the attenuation weight and the minimum image rendering specification, and the product of the enhancement weight and the preset image rendering specification corresponding to the user interest region, the buffer image rendering specification of the user interest region in the transition frame is determined.

[0008] Optionally, the method further includes: Based on the target user's line-of-sight movement speed indicated by the prediction result, adjust at least one of the first time constant, the second time constant, and the preset transition time; Based on the confidence level of the prediction results, the start time of the preset transition time is adjusted.

[0009] Optionally, the image rendering specifications include at least one of the following: Rendering priority information; rendering resolution; rendering calculation precision; maximum error for color difference correction; available rendering functions.

[0010] Optionally, predicting the landing position of the target user's eye in the image area for a future target frame, to obtain the predicted landing position of the target frame, includes: Obtain the actual landing point positions of multiple historical frames preceding the target frame; The actual landing positions of the multiple historical frames are input into the landing prediction model to obtain the predicted landing position output by the landing prediction model; the landing prediction model includes a bidirectional gated recurrent unit layer and a multi-head attention mechanism layer.

[0011] Optionally, the method further includes: Determine the distance between the predicted landing point and the nearest region boundary; If the distance is less than or equal to a preset distance, determine the correction parameter corresponding to the boundary of the nearest region; Based on the correction parameters, the boundary of the nearest region is adjusted; Based on the adjusted region boundary, the processing region corresponding to the predicted landing point position is determined.

[0012] Optionally, determining the correction parameter corresponding to the nearest region boundary when the distance is less than or equal to a preset distance includes: Determine the image complexity information of the virtual reality scene within a preset range around the predicted landing point; Based on the image complexity information and the historical motion trajectory information of the target user's eye movement, the correction parameters corresponding to the predicted landing point position are determined.

[0013] Optionally, the plurality of processing regions include a first processing region, a second processing region, and a third processing region; the field of view range corresponding to the first processing region includes (0°, 10°); the field of view range corresponding to the second processing region includes (10°, 20°); and the field of view range corresponding to the third processing region includes (>20°).

[0014] Secondly, embodiments of this application also provide a virtual reality image rendering apparatus, comprising: The partitioning module is used to divide the screen area of ​​the virtual reality device into multiple processing areas; each of the multiple processing areas corresponds to a different field of view range. The prediction module is used to predict the location where the target user's eye gaze will fall in the screen area for a future target frame, and obtain the predicted location of the target frame. The first determining module is used to determine, from multiple different preset image rendering specifications, the target image rendering specification corresponding to the processing area where the predicted landing point is located; and to determine the viewing concentration area corresponding to the predicted landing point. The second determining module is used to use the target image rendering specification as the image rendering specification of the eye-concentration area; and to use the lowest image rendering specification as the image rendering specification of other image areas besides the eye-concentration area. The rendering module is used to respond to the actual landing position of the target user's eye in the screen area under the target frame, and if the actual landing position matches the predicted landing position, to display the virtual reality screen of the target frame based on the screen rendering specifications corresponding to the target frame.

[0015] Optionally, the device further includes a transition module for: Based on the actual landing positions of multiple historical frames, the eye trajectory of the target user is predicted, and at least one user interest region is determined from the screen area based on the prediction results. For any user interest region, determine the transition screen rendering specifications of the user interest region in each transition frame corresponding to the preset transition time; the rendering quality of the transition screen rendering specifications is higher than the rendering quality of the lowest screen rendering specifications, but lower than the preset screen rendering specifications corresponding to the user interest region. The rendering specifications of the transition frame are updated using the aforementioned rendering specifications, and the virtual reality image corresponding to the transition frame is displayed based on the updated rendering specifications.

[0016] Optionally, the transition module is used for: For any of the transition frames, a decay weight is determined based on the position of the transition frame in the preset transition time and a first time constant; and an enhancement weight is determined based on the position of the transition frame in the preset transition time and a second time constant. Based on the product of the attenuation weight and the minimum image rendering specification, and the product of the enhancement weight and the preset image rendering specification corresponding to the user interest region, the buffer image rendering specification of the user interest region in the transition frame is determined.

[0017] Optionally, the transition module is further configured to: Based on the target user's line-of-sight movement speed indicated by the prediction result, adjust at least one of the first time constant, the second time constant, and the preset transition time; Based on the confidence level of the prediction results, the start time of the preset transition time is adjusted.

[0018] Optionally, the image rendering specifications include at least one of the following: Rendering priority information; rendering resolution; rendering calculation precision; maximum error for color difference correction; available rendering functions.

[0019] Optionally, the prediction module is used for: Obtain the actual landing point positions of multiple historical frames preceding the target frame; The actual landing positions of the multiple historical frames are input into the landing prediction model to obtain the predicted landing position output by the landing prediction model; the landing prediction model includes a bidirectional gated recurrent unit layer and a multi-head attention mechanism layer.

[0020] Optionally, the device further includes a boundary adjustment module for: Determine the distance between the predicted landing point and the nearest region boundary; If the distance is less than or equal to a preset distance, determine the correction parameter corresponding to the boundary of the nearest region; Based on the correction parameters, the boundary of the nearest region is adjusted; Based on the adjusted region boundary, the processing region corresponding to the predicted landing point position is determined.

[0021] Optionally, the boundary adjustment module is used for: Determine the image complexity information of the virtual reality scene within a preset range around the predicted landing point; Based on the image complexity information and the historical motion trajectory information of the target user's eye movement, the correction parameters corresponding to the predicted landing point position are determined.

[0022] Optionally, the plurality of processing regions include a first processing region, a second processing region, and a third processing region; the field of view range corresponding to the first processing region includes (0°, 10°); the field of view range corresponding to the second processing region includes (10°, 20°); and the field of view range corresponding to the third processing region includes (>20°).

[0023] Thirdly, embodiments of this application also provide a computer device, which includes a processor and a memory. The memory stores machine-readable instructions executable by the processor. The processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, they perform the steps of the first aspect above, or any possible implementation of the first aspect.

[0024] Fourthly, optional embodiments of this application also provide a computer-readable storage medium storing a computer program that, when run, performs the steps of the first aspect or any possible implementation of the first aspect.

[0025] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application.

[0026] The virtual reality scene rendering method and apparatus provided in this application can divide the screen area of ​​a virtual reality device into multiple processing areas. Different processing areas correspond to different field of view ranges and preset scene rendering specifications, thereby performing differentiated rendering of the virtual reality scene. This allows the area where the user's gaze is more focused to use a higher scene rendering specification, while the area where the user pays less attention to uses the lowest scene rendering specification. While ensuring the scene effect in the area where the user pays high attention, this effectively reduces the computing resources required for virtual reality rendering. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a virtual reality screen rendering method according to an exemplary embodiment of this application; Figure 2 This is one of the schematic diagrams of a screen area shown in an exemplary embodiment of this application; Figure 3 This is a second schematic diagram of the screen area shown in an exemplary embodiment of this application; Figure 4 This is a schematic diagram of a virtual reality screen rendering device shown in an exemplary embodiment of this application; Figure 5 This is a schematic diagram of a computer device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0029] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0030] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0031] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0032] Research has found that some VR devices are limited by size, power consumption, and processor strength, resulting in relatively limited computing and graphics processing capabilities. This makes it difficult to support continuous high-load rendering of complex scenes, leading to poor VR content display.

[0033] In view of this, embodiments of this application provide a virtual reality screen rendering method and apparatus, which can divide the screen area of ​​a virtual reality device into multiple processing areas. Different processing areas correspond to different field of view ranges and preset screen rendering specifications, thereby performing differentiated rendering of virtual reality screens. This allows higher screen rendering specifications to be used for areas where the user's gaze is more focused, while the lowest screen rendering specifications are used for areas where the user's attention is less focused. This effectively reduces the computing resources required for virtual reality rendering while ensuring the screen effect in areas where the user's attention is high.

[0034] The deficiencies of the existing technical solutions are the result of the inventor's practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this application below should be considered as the inventor's contributions to this application.

[0035] To facilitate understanding of this embodiment, the application scenario of the virtual reality image rendering method disclosed in this application embodiment will first be introduced. The execution subject of the virtual reality image rendering method provided in this application embodiment can be a virtual reality device, especially a mobile virtual reality device, such as a head-mounted VR device, VR glasses, or a smartphone with VR functionality. In some possible implementations, the virtual reality image rendering method can be implemented by a processor calling computer-readable instructions stored in memory.

[0036] See Figure 1 The diagram shown is a flowchart illustrating a virtual reality scene rendering method according to an exemplary embodiment of this application. The method includes steps S101 to S105, wherein: S101. Divide the screen area of ​​the virtual reality device into multiple processing areas; the multiple processing areas correspond to different field of view ranges.

[0037] Virtual reality (VR) devices are devices that use computer technology to generate a virtual three-dimensional environment, allowing users to interact with the virtual world through hardware devices such as head-mounted displays, motion controllers, and position trackers. In this application embodiment, VR devices may include computer-based VR devices, host-based VR devices, all-in-one VR devices, and mobile VR devices.

[0038] The aforementioned screen area refers to the image display area on the virtual reality device's screen used to present the virtual scene. This screen typically covers the entire display area of ​​the user's field of vision, such as a horizontal field of view of approximately 100° to 120°.

[0039] See Figure 2 The image shown is one of the schematic diagrams of the screen area shown in an exemplary embodiment of this application. Figure 2 In this scenario, the distance between the target user's eyeball and the display screen is 10mm, and the area within the dotted line in the center of the display screen is the image area.

[0040] In this step, the image area can be divided into multiple processing areas. The image area can be divided according to a preset field of view to obtain multiple processing areas.

[0041] The aforementioned field of view range can be set according to the visual characteristics of the human eye and the optical characteristics of the system. Research has found that the fovea region of the human retina (approximately 2° from the center of the field of view) has the highest density of cone cells and the highest perception accuracy. Visual acuity outside the fovea region decreases sharply in a radial pattern, with a sensitivity outside 15° of the field of view being only about 20% of that at the center. In eye-tracking analysis, over 80% of the user's gaze is distributed within the central region of the VR field of view (±15° range). During normal use, users rarely directly gaze at the peripheral areas, instead shifting their focus to the central field of view by turning their head. The peripheral field of view primarily provides environmental perception and motion detection, with lower requirements for fine details and color accuracy. The optical distortion of VR lenses increases non-linearly from the center to the edges. In standard Fresnel lens systems, the distortion and chromatic aberration complexity in the peripheral areas are high, making correction difficult. Even with significant computational resources, the improvement in correction effectiveness in the peripheral areas is limited, exhibiting physical limitations.

[0042] See Figure 3The image shown is a second schematic diagram of the screen area shown in an exemplary embodiment of this application. Figure 3 In the image, the screen area is divided into a first processing area 1, a second processing area 2, and a third processing area 3. The field of view range corresponding to the first processing area 1 includes (0°, 10°); the field of view range corresponding to the second processing area 2 includes (10°, 20°); and the field of view range corresponding to the third processing area 3 includes (>20°).

[0043] For example, the first processing area 1 corresponds to the fovea region of the retina, which is the area that users pay the most attention to; the second processing area 2 corresponds to the periretinal region, which is the area that users pay the second most attention to; and the edge area corresponds to the peripheral region of the retina, which is the area that users pay less attention to.

[0044] After obtaining multiple processing regions, a corresponding preset screen rendering specification can be configured for each processing region. When the user focuses on different regions, the corresponding rendering screen specification is used to achieve differentiated rendering of the screen. High-quality screens are rendered in areas of high user attention, and low-quality screens are rendered in areas of low user attention, thereby reducing the computational resource requirements for screen rendering while maintaining the screen quality of high-attention areas.

[0045] S102. For a future target frame, predict the landing position of the target user's eye gaze in the screen area to obtain the predicted landing position of the target frame.

[0046] In this step, the location where the target user's eye falls on the screen area can be predicted in advance to determine the high-precision rendering area, thereby improving rendering efficiency and reducing latency.

[0047] The target frame can be one or more frames after the current frame. For example, if the current frame is frame t, the target frame can include frame t+1, frame t+2, etc.

[0048] Eye gaze refers to the spatial vector of the direction of a target user's gaze, used to determine the user's gaze position in a virtual scene. It can be detected in real time by an eye-tracking module (such as an eye sensor) to obtain the direction angle (pitch angle, yaw angle) or spatial coordinates of the user's gaze.

[0049] The point where the eye's gaze falls on the screen area can be referred to as the intersection of the target user's line of sight and the virtual reality display plane, that is, the projection point of the gaze on the screen, also known as the "focus point" or "focal point". This position determines the center of the target user's visual attention.

[0050] For example, the predicted landing point can be obtained through machine learning models, deep learning models, interpolation calculations, etc.

[0051] In some embodiments, the actual landing positions of multiple historical frames preceding the target frame can be obtained; then the actual landing positions of the multiple historical frames are input into the landing prediction model to obtain the predicted landing position output by the landing prediction model; the landing prediction model includes a bidirectional gated recurrent unit layer and a multi-head attention mechanism layer.

[0052] In this step, the actual focal point can refer to the user's real gaze point on the virtual reality display plane. The predicted focal point, on the other hand, is the predicted assumed focal point of the user's gaze on the real plane.

[0053] The aforementioned landing point prediction model can be a neural network model. This landing point prediction model can include a Bidirectional Gated Recurrent Unit (Bi-GRU) layer to capture more complete contextual information. By updating and resetting the gates, it controls the flow of information, effectively handles long sequences and short-term dependencies, and outputs a hidden state vector by processing the input data, which can contain rich temporal features.

[0054] For example, a bidirectional gated recurrent unit (GRU) layer can contain two stacked GRU layers. The first layer can contain 128 GRU units (64 in each direction) to process the raw sequence features; the second layer can contain 64 GRU units (32 in each direction) to extract high-level temporal features. Residual connections can be added between the two GRU layers to prevent information decay. A normalization layer can be added after each GRU layer to accelerate training and improve stability.

[0055] The aforementioned landing point prediction model can also incorporate an attention mechanism to assign different weights to different time points in the input sequence, capturing different types of temporal dependencies. For example, a four-head attention mechanism can be used, with each attention head focusing on features in different dimensions. For instance, the number of historical frames can be eight, and the attention heads can include: a head focusing on short-term rapid changes, paying attention to the positions in the most recent 2-3 frames; a head focusing on medium-term trends, paying attention to the positions in the past 4-6 frames; a head focusing on motion patterns, paying attention to the pattern features of the entire sequence; and a head focusing on key turning points, paying attention to features such as points of change in direction or sudden changes in velocity.

[0056] By integrating multi-head information, the model can simultaneously consider multiple time scales and dependency types, making it particularly effective for complex eye movement predictions, capturing both short-term velocity changes and long-term motion patterns.

[0057] The above-mentioned landing point prediction model can also be equipped with a multi-step position predictor, which can use a two-layer fully connected network with ReLU activation to predict the eye view in the next few frames and output it linearly.

[0058] The aforementioned landing point prediction model may also include a time-coding layer to add relative timestamps to the input data, representing the sampling time interval. Optionally, a random deactivation Dropout layer to prevent overfitting may also be included, with the deactivation value set to 0.2.

[0059] S103. From multiple different preset image rendering specifications, determine the target image rendering specification corresponding to the processing area where the predicted landing point is located; and determine the gaze concentration area corresponding to the predicted landing point.

[0060] In this step, the preset image rendering specifications can correspond to the processing area. For example, the image rendering specifications may include at least one of the following: rendering priority information; rendering resolution; rendering calculation accuracy; maximum color difference correction error; available rendering functions.

[0061] In some implementations, the preset rendering specifications corresponding to the first processing area may include: high rendering priority weight (e.g., 70%), full resolution rendering, using a high-precision processing model (e.g., FP16), a maximum color difference correction error of 0.1 pixels (ΔE < 1.5), using GPU acceleration, using cache loading, using fast processing, etc. The preset rendering specifications corresponding to the second processing area may include: medium rendering priority weight (e.g., 25%), half resolution rendering, using a processing model with medium calculation accuracy (e.g., FP8), and a maximum color difference correction error of 0.5 pixels (ΔE < 3). The preset rendering specifications used in the third processing area may include: low rendering priority weight (e.g., 5%), quarter-resolution rendering, use of a low-precision processing model (e.g., INT8), and a maximum color difference correction error of 1 pixel (ΔE < 6).

[0062] Based on the processing area where the predicted landing point is located, and the mapping relationship between the processing area and the preset image rendering specifications, the target image rendering specifications corresponding to the processing area can be determined.

[0063] The aforementioned gaze concentration area can refer to an area with a certain size around the predicted gaze point. For example, the gaze concentration area can be a rectangular area with a side length of 700 pixels centered on the predicted gaze point, or a circular area with a diameter of 700 pixels.

[0064] When determining the processing area where the predicted landing point is located, the predicted landing point may be near the boundary of the processing area, which may cause the processing area where the predicted landing point is located to change multiple times in a short period of time, resulting in frequent switching of rendering specifications. Therefore, the boundary position of the processing area can be dynamically adjusted to reduce the rate of change of rendering specifications, prevent the rendering of the image from jumping or the resolution from changing abruptly, and thus stabilize the virtual reality image.

[0065] For example, the distance between the predicted landing point and the nearest region boundary can be determined; if the distance is less than or equal to a preset distance, a correction parameter corresponding to the nearest region boundary can be determined; the nearest region boundary can be adjusted based on the correction parameter; and the processing area corresponding to the predicted landing point can be determined based on the adjusted region boundary.

[0066] Specifically, when determining the correction parameters, the image complexity information of the virtual reality screen within a preset range around the predicted landing point can be determined; based on the image complexity information and the historical movement trajectory information of the target user's eye movement, the correction parameters corresponding to the predicted landing point position can be determined.

[0067] In this way, by adjusting the parameters, the boundary lines of the original processing area are moved or softened, thereby optimizing the visual continuity near the boundary.

[0068] For example, the image complexity information mentioned above may include content complexity, texture density, edge strength, etc. When near the boundary between the first processing region and the second processing region, the boundary can be adjusted based on the content complexity and the historical motion trajectory information of the target user's eye movement; when near the boundary between the second processing region and the third processing region, the boundary can be adjusted based on the motion intensity and the historical motion trajectory information of the target user's eye movement.

[0069] The historical movement trajectory information of the target user's eye can reflect the target user's behavior. Based on the target user's behavior, correction weights can be determined, and the image complexity information can be adjusted using the correction weights.

[0070] S104. Use the target image rendering specification as the image rendering specification for the area where the view is concentrated; use the lowest image rendering specification as the image rendering specification for other image areas besides the area where the view is concentrated.

[0071] In this step, the area where the predicted landing point is located, which is the region of greatest interest to the user, can be rendered using higher specifications to ensure visual clarity and image quality. However, users' perception of different processing areas varies, so the preset image rendering specifications for the corresponding processing area can be used as the target image rendering specifications for rendering the area of ​​greatest interest.

[0072] Aside from the areas where the user's attention is focused, the remaining areas have lower user attention, so the lowest rendering specifications can be used to save computing resources.

[0073] S105. In response to obtaining the actual landing point of the target user's eye in the screen area under the target frame, and if the actual landing point matches the predicted landing point, display the virtual reality screen of the target frame based on the screen rendering specifications corresponding to the target frame.

[0074] In this step, when the target frame arrives, the actual gaze point position of the target user under the target frame (i.e., the actual landing point position) can be collected in real time. If the actual landing point matches the previously predicted landing point (within the allowable range of error), it means that the prediction is accurate and the frame can be rendered using the predicted rendering specifications. Since the rendering specifications of the frame have been predetermined, they can be used directly, which can effectively reduce the latency of the rendering.

[0075] In one possible implementation, after determining the rendering specifications of the target frame, rendering can be performed directly according to the rendering specifications, and the rendering results can be stored in a cache for subsequent direct use of the rendering results to display virtual reality images.

[0076] If the actual landing point does not match the predicted landing point (e.g., the error exceeds the allowable range), the image can be rendered according to the preset rendering specifications.

[0077] In some embodiments, the gaze trajectory of the target user can be predicted based on the actual landing positions of multiple historical frames, and at least one user interest region can be determined from the screen area based on the prediction results; for any user interest region, the transition screen rendering specifications of the user interest region are determined in each transition frame corresponding to a preset transition time; the rendering quality of the transition screen rendering specifications is higher than the rendering quality of the lowest screen rendering specifications, but lower than the preset screen rendering specifications corresponding to the user interest region; the screen rendering specifications of the transition frames are updated using the transition screen rendering specifications, and the virtual reality screen corresponding to the transition frames is displayed based on the updated screen rendering specifications.

[0078] In this way, by rendering the possible user interest areas in a transitional manner, the viewer's gaze can smoothly transition from low to high image quality.

[0079] For example, when determining the rendering specifications of a transition frame, for any one of the transition frames, a decay weight can be determined based on the position of the transition frame in the preset transition time and a first time constant; and an enhancement weight can be determined based on the position of the transition frame in the preset transition time and a second time constant; and the buffer rendering specifications of the user interest region in the transition frame can be determined based on the product of the decay weight and the lowest rendering specifications, and the product of the enhancement weight and the preset rendering specifications corresponding to the user interest region.

[0080] The aforementioned attenuation weights are used to control the contribution of low-specification rendering to the current frame to gradually decrease over time during the transition process. The first time constant determines the attenuation rate, that is, the rate at which the influence of low-specification rendering decreases with the number of frames. The enhancement weights are used to gradually increase the influence of the target specification in the region of interest, so that the rendering effect smoothly transitions to the final high specification. The second time constant controls the enhancement rate, that is, the rate at which the influence of high specification increases.

[0081] In some embodiments, an exponential or linear function may be used to gradually increase the enhancement weight from 0 to 1 and gradually decrease the decay weight from 1 to 0.

[0082] For example, the rendering specifications for transition scenes can be determined using the following formula: ; ; .

[0083] in, For transitional scene rendering strategies, The current rendering strategy for frame t. The final rendering specifications required for the transition process (such as the preset rendering specifications mentioned above). The attenuation weight for frame t. Here, represents the enhancement weight for frame t, where t is the position of frame t within the preset transition time, and T is the length of the preset transition time. The first time constant, This is the second time constant.

[0084] For example, the initial value of the preset transition time can be 50ms.

[0085] In some implementations, at least one of the first time constant, the second time constant, and the preset transition time can be adjusted based on the target user's line-of-sight movement speed indicated by the prediction result; and the start time of the preset transition time can be adjusted based on the confidence level of the prediction result.

[0086] This allows for control over the intensity of the difference in image rendering quality during the transition phase.

[0087] For example, when the target user's gaze movement speed indicated by the prediction result is greater than 15° / s, the enhancement weight can be increased to speed up the transition; when the gaze movement speed is less than 5° / s, the transition time can be extended to improve smoothness; when the confidence level is greater than 0.8, the start time of the preset transition time can be advanced to start the transition rendering earlier; when the confidence level is less than 0.5, the start time of the preset transition time can be delayed to maintain a longer transition phase.

[0088] The virtual reality scene rendering method provided in this application can divide the screen area of ​​the virtual reality device into multiple processing areas. Different processing areas correspond to different field of view ranges and preset scene rendering specifications, thereby performing differentiated rendering of the virtual reality scene. This allows the area where the user's gaze is more focused to use a higher scene rendering specification, while the area where the user pays less attention to uses the lowest scene rendering specification. While ensuring the scene effect in the area where the user pays high attention, this method effectively reduces the computing resources required for virtual reality rendering.

[0089] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0090] Based on the same inventive concept, this application also provides a virtual reality rendering device corresponding to the virtual reality rendering method. Since the principle of the device in this application is similar to the virtual reality rendering method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0091] See Figure 4 The diagram shown is a schematic representation of a virtual reality image rendering apparatus according to an exemplary embodiment of this application. The apparatus includes: The partitioning module 410 is used to divide the screen area of ​​the virtual reality device into multiple processing areas; the multiple processing areas correspond to different field of view ranges. The prediction module 420 is used to predict the landing position of the target user's eye gaze in the screen area for a future target frame, and obtain the predicted landing position of the target frame. The first determining module 430 is used to determine, from multiple different preset image rendering specifications, the target image rendering specification corresponding to the processing area where the predicted landing point is located; and to determine the viewing concentration area corresponding to the predicted landing point. The second determining module 440 is used to use the target image rendering specification as the image rendering specification of the view concentration area; and to use the lowest image rendering specification as the image rendering specification of other image areas besides the view concentration area. The rendering module 450 is used to respond to obtaining the actual landing point position of the target user's eye in the screen area under the target frame, and if the actual landing point position matches the predicted landing point position, to display the virtual reality screen of the target frame based on the screen rendering specifications corresponding to the target frame.

[0092] Optionally, the device further includes a transition module 460 for: Based on the actual landing positions of multiple historical frames, the eye trajectory of the target user is predicted, and at least one user interest region is determined from the screen area based on the prediction results. For any user interest region, determine the transition screen rendering specifications of the user interest region in each transition frame corresponding to the preset transition time; the rendering quality of the transition screen rendering specifications is higher than the rendering quality of the lowest screen rendering specifications, but lower than the preset screen rendering specifications corresponding to the user interest region. The rendering specifications of the transition frame are updated using the aforementioned rendering specifications, and the virtual reality image corresponding to the transition frame is displayed based on the updated rendering specifications.

[0093] Optionally, the transition module 460 is used for: For any of the transition frames, a decay weight is determined based on the position of the transition frame in the preset transition time and a first time constant; and an enhancement weight is determined based on the position of the transition frame in the preset transition time and a second time constant. Based on the product of the attenuation weight and the minimum image rendering specification, and the product of the enhancement weight and the preset image rendering specification corresponding to the user interest region, the buffer image rendering specification of the user interest region in the transition frame is determined.

[0094] Optionally, the transition module 460 is further configured to: Based on the target user's line-of-sight movement speed indicated by the prediction result, adjust at least one of the first time constant, the second time constant, and the preset transition time; Based on the confidence level of the prediction results, the start time of the preset transition time is adjusted.

[0095] Optionally, the image rendering specifications include at least one of the following: Rendering priority information; rendering resolution; rendering calculation precision; maximum error for color difference correction; available rendering functions.

[0096] Optionally, the prediction module 420 is used for: Obtain the actual landing point positions of multiple historical frames preceding the target frame; The actual landing positions of the multiple historical frames are input into the landing prediction model to obtain the predicted landing position output by the landing prediction model; the landing prediction model includes a bidirectional gated recurrent unit layer and a multi-head attention mechanism layer.

[0097] Optionally, the device further includes a boundary adjustment module 470, used for: Determine the distance between the predicted landing point and the nearest region boundary; If the distance is less than or equal to a preset distance, determine the correction parameter corresponding to the boundary of the nearest region; Based on the correction parameters, the boundary of the nearest region is adjusted; Based on the adjusted region boundary, the processing region corresponding to the predicted landing point position is determined.

[0098] Optionally, the boundary adjustment module 470 is used for: Determine the image complexity information of the virtual reality scene within a preset range around the predicted landing point; Based on the image complexity information and the historical motion trajectory information of the target user's eye movement, the correction parameters corresponding to the predicted landing point position are determined.

[0099] Optionally, the plurality of processing regions include a first processing region, a second processing region, and a third processing region; the field of view range corresponding to the first processing region includes (0°, 10°); the field of view range corresponding to the second processing region includes (10°, 20°); and the field of view range corresponding to the third processing region includes (>20°).

[0100] The virtual reality rendering apparatus provided in this application can divide the screen area of ​​a virtual reality device into multiple processing areas. Different processing areas correspond to different field of view ranges and preset screen rendering specifications, thereby performing differentiated rendering of the virtual reality screen. This allows higher screen rendering specifications to be used in areas where the user's gaze is more focused, while the lowest screen rendering specifications are used in areas where the user's attention is less focused. This effectively reduces the computing resources required for virtual reality rendering while ensuring the screen effect in areas where the user's attention is high.

[0101] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0102] This application also provides a computer device, such as... Figure 5 The diagram shown is a schematic representation of a computer device structure according to an exemplary embodiment of this application. The computer device includes: A processor 51 and a memory 52; the memory 52 stores machine-readable instructions executable by the processor 51, and the processor 51 executes the machine-readable instructions stored in the memory 52. ​​When the machine-readable instructions are executed by the processor 51, the processor 51 performs the following steps: The screen area of ​​the virtual reality device is divided into multiple processing areas; each of the multiple processing areas corresponds to a different field of view range. For a future target frame, the predicted landing position of the target user's eye gaze in the screen area is predicted to obtain the predicted landing position of the target frame. From multiple different preset image rendering specifications, determine the target image rendering specification corresponding to the processing area where the predicted landing point is located; and determine the gaze concentration area corresponding to the predicted landing point. The target image rendering specification is used as the image rendering specification for the area where the view is focused; the lowest image rendering specification is used as the image rendering specification for other image areas besides the area where the view is focused. In response to obtaining the actual landing point of the target user's eye in the screen area under the target frame, and if the actual landing point matches the predicted landing point, the virtual reality screen of the target frame is displayed based on the screen rendering specifications corresponding to the target frame.

[0103] Optionally, processor 51 is also used to perform: Based on the actual landing positions of multiple historical frames, the eye trajectory of the target user is predicted, and at least one user interest region is determined from the screen area based on the prediction results. For any user interest region, determine the transition screen rendering specifications of the user interest region in each transition frame corresponding to the preset transition time; the rendering quality of the transition screen rendering specifications is higher than the rendering quality of the lowest screen rendering specifications, but lower than the preset screen rendering specifications corresponding to the user interest region. The rendering specifications of the transition frame are updated using the aforementioned rendering specifications, and the virtual reality image corresponding to the transition frame is displayed based on the updated rendering specifications.

[0104] Optionally, determining the rendering specifications of the transition screen for the user's region of interest in each transition frame corresponding to the transition time includes: For any of the transition frames, a decay weight is determined based on the position of the transition frame in the preset transition time and a first time constant; and an enhancement weight is determined based on the position of the transition frame in the preset transition time and a second time constant. Based on the product of the attenuation weight and the minimum image rendering specification, and the product of the enhancement weight and the preset image rendering specification corresponding to the user interest region, the buffer image rendering specification of the user interest region in the transition frame is determined.

[0105] Optionally, processor 51 is also used to perform: Based on the target user's line-of-sight movement speed indicated by the prediction result, adjust at least one of the first time constant, the second time constant, and the preset transition time; Based on the confidence level of the prediction results, the start time of the preset transition time is adjusted.

[0106] Optionally, the image rendering specifications include at least one of the following: Rendering priority information; rendering resolution; rendering calculation precision; maximum error for color difference correction; available rendering functions.

[0107] Optionally, predicting the landing position of the target user's eye in the image area for a future target frame, to obtain the predicted landing position of the target frame, includes: Obtain the actual landing point positions of multiple historical frames preceding the target frame; The actual landing positions of the multiple historical frames are input into the landing prediction model to obtain the predicted landing position output by the landing prediction model; the landing prediction model includes a bidirectional gated recurrent unit layer and a multi-head attention mechanism layer.

[0108] Optionally, processor 51 is also used to perform: Determine the distance between the predicted landing point and the nearest region boundary; If the distance is less than or equal to a preset distance, determine the correction parameter corresponding to the boundary of the nearest region; Based on the correction parameters, the boundary of the nearest region is adjusted; Based on the adjusted region boundary, the processing region corresponding to the predicted landing point position is determined.

[0109] Optionally, determining the correction parameter corresponding to the nearest region boundary when the distance is less than or equal to a preset distance includes: Determine the image complexity information of the virtual reality scene within a preset range around the predicted landing point; Based on the image complexity information and the historical motion trajectory information of the target user's eye movement, the correction parameters corresponding to the predicted landing point position are determined.

[0110] Optionally, the plurality of processing regions include a first processing region, a second processing region, and a third processing region; the field of view range corresponding to the first processing region includes (0°, 10°); the field of view range corresponding to the second processing region includes (10°, 20°); and the field of view range corresponding to the third processing region includes (>20°).

[0111] The aforementioned memory 52 includes a main memory 521 and an external memory 522; the main memory 521, also known as internal memory, is used to temporarily store the computational data in the processor 51, as well as the data exchanged with external memory 522 such as a hard disk. The processor 51 exchanges data with the external memory 522 through the main memory 521.

[0112] The specific execution process of the above instructions can be referred to the steps of the virtual reality screen rendering method described in the embodiments of this application, and will not be repeated here.

[0113] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0114] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the virtual reality image rendering method described in the above method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0115] This application also provides a computer program product, including a computer program / instruction, which, when executed by the computer program / instruction processor, implements the virtual reality screen rendering method provided in the various embodiments of this application.

[0116] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0117] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0118] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0119] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0120] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0122] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A virtual reality image rendering method, characterized in that, The method includes: The screen area of ​​the virtual reality device is divided into multiple processing areas; each of the multiple processing areas corresponds to a different field of view range. For a future target frame, the predicted landing position of the target user's eye gaze in the screen area is predicted to obtain the predicted landing position of the target frame. From multiple different preset image rendering specifications, determine the target image rendering specification corresponding to the processing area where the predicted landing point is located; and determine the gaze concentration area corresponding to the predicted landing point. The target image rendering specification is used as the image rendering specification for the area where the view is focused; the lowest image rendering specification is used as the image rendering specification for other image areas besides the area where the view is focused. In response to obtaining the actual landing point of the target user's eye in the screen area under the target frame, and if the actual landing point matches the predicted landing point, the virtual reality screen of the target frame is displayed based on the screen rendering specifications corresponding to the target frame.

2. The method according to claim 1, characterized in that, The method further includes: Based on the actual landing positions of multiple historical frames, the eye trajectory of the target user is predicted, and at least one user interest region is determined from the screen area based on the prediction results. For any user interest region, determine the transition screen rendering specifications of the user interest region in each transition frame corresponding to the preset transition time; the rendering quality of the transition screen rendering specifications is higher than the rendering quality of the lowest screen rendering specifications, but lower than the preset screen rendering specifications corresponding to the user interest region. The rendering specifications of the transition frame are updated using the aforementioned rendering specifications, and the virtual reality image corresponding to the transition frame is displayed based on the updated rendering specifications.

3. The method according to claim 2, characterized in that, The determination of the rendering specifications for the transition screen of the user's region of interest in each transition frame corresponding to the transition time includes: For any of the transition frames, a decay weight is determined based on the position of the transition frame in the preset transition time and a first time constant; and an enhancement weight is determined based on the position of the transition frame in the preset transition time and a second time constant. Based on the product of the attenuation weight and the minimum image rendering specification, and the product of the enhancement weight and the preset image rendering specification corresponding to the user interest region, the buffer image rendering specification of the user interest region in the transition frame is determined.

4. The method according to claim 3, characterized in that, The method further includes: Based on the target user's line-of-sight movement speed indicated by the prediction result, adjust at least one of the first time constant, the second time constant, and the preset transition time; Based on the confidence level of the prediction results, the start time of the preset transition time is adjusted.

5. The method according to claim 1, characterized in that, The image rendering specifications include at least one of the following: Rendering priority information; rendering resolution; rendering calculation precision; maximum error for color difference correction; available rendering functions.

6. The method according to claim 1, characterized in that, The process of predicting the location of the target user's eye movement within the image area for a future target frame, to obtain the predicted location of the target frame, includes: Obtain the actual landing point positions of multiple historical frames preceding the target frame; The actual landing positions of the multiple historical frames are input into the landing prediction model to obtain the predicted landing position output by the landing prediction model; the landing prediction model includes a bidirectional gated recurrent unit layer and a multi-head attention mechanism layer.

7. The method according to claim 1, characterized in that, The method further includes: Determine the distance between the predicted landing point and the nearest region boundary; If the distance is less than or equal to a preset distance, determine the correction parameter corresponding to the boundary of the nearest region; Based on the correction parameters, the boundary of the nearest region is adjusted; Based on the adjusted region boundary, the processing region corresponding to the predicted landing point position is determined.

8. The method according to claim 7, characterized in that, The step of determining the correction parameter corresponding to the nearest region boundary when the distance is less than or equal to a preset distance includes: Determine the image complexity information of the virtual reality scene within a preset range around the predicted landing point; Based on the image complexity information and the historical motion trajectory information of the target user's eye movement, the correction parameters corresponding to the predicted landing point position are determined.

9. The method according to claim 1, characterized in that, The plurality of processing regions include a first processing region, a second processing region, and a third processing region; the field of view range corresponding to the first processing region includes (0°, 10°); the field of view range corresponding to the second processing region includes (10°, 20°); and the field of view range corresponding to the third processing region includes (>20°).

10. A virtual reality image rendering device, characterized in that, The device includes: The partitioning module is used to divide the screen area of ​​the virtual reality device into multiple processing areas; each of the multiple processing areas corresponds to a different field of view range. The prediction module is used to predict the location where the target user's eye gaze will fall in the screen area for a future target frame, and obtain the predicted location of the target frame. The first determining module is used to determine, from multiple different preset image rendering specifications, the target image rendering specification corresponding to the processing area where the predicted landing point is located; and to determine the viewing concentration area corresponding to the predicted landing point. The second determining module is used to use the target image rendering specification as the image rendering specification of the eye-concentration area; and to use the lowest image rendering specification as the image rendering specification of other image areas besides the eye-concentration area. The rendering module is used to respond to the actual landing position of the target user's eye in the screen area under the target frame, and if the actual landing position matches the predicted landing position, to display the virtual reality screen of the target frame based on the screen rendering specifications corresponding to the target frame.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 9.

12. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 9.

Images