A method, apparatus, medium and device for frame image rendering

Abstract

The application relates to the technical field of image rendering, and particularly provides a frame image rendering method, device, medium and equipment. The method can comprise the following steps: in the case that color channel data in a current frame image is refreshed by a display driver, predicting a next frame image of the current frame image; wherein the color channel data comprises R channel data, G channel data or B channel data; rendering the color channel data of the next frame image by a graphics processor to obtain a rendering result of the color channel data. The embodiment of the application can realize low-delay rendering.

Description

Technical Field

[0001] This application relates to the field of image rendering technology, and more specifically, to a method, apparatus, medium, and device for rendering frame images. Background Technology

[0002] Extended Reality (XR) is a collective term for technologies such as Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR). It creates a seamless interactive environment that blends the virtual and real worlds through wearable devices (such as headsets and sensors), forming a continuous technological spectrum from completely virtual to completely real. Taking AR as an example, in the AR field, latency refers to the time difference between a user's action (such as head movement) and the image being displayed on the AR device. This latency includes sensor data acquisition, data processing, rendering, and display. High latency often causes virtual objects to float, destroying the realism of the virtual-real fusion. Currently, to reduce latency during rendering, the frame buffer is divided into upper and lower sub-buffers. While the display device reads the upper part, the image processor can draw the lower part. However, this method cannot divide the frame buffer into upper and lower parts for reading and writing, resulting in read conflicts and hindering high-quality, low-latency rendering. Summary of the Invention

[0003] The purpose of some embodiments of this application is to provide a method, apparatus, medium and device for frame image rendering. The technical solutions of the embodiments of this application can achieve high-quality, low-latency rendering of frame images and effectively avoid frame buffer read conflicts.

[0004] In a first aspect, some embodiments of this application provide a method for rendering a frame image, comprising: predicting the next frame image of the current frame image after the display driver has completed refreshing the color channel data in the current frame image; wherein the color channel data includes R channel data, G channel data or B channel data; rendering the color channel data of the next frame image by a graphics processor to obtain the rendering result of the color channel data.

[0005] Some embodiments of this application predict the next frame image after the display driver has finished refreshing the color channel data of the current frame image, and then use the image processor to render the color channel data of the next frame image to obtain the rendering result of the color channel data. Embodiments of this application can precisely control the time when each color channel data is drawn to the frame buffer, avoiding conflicts caused by the display driver reading from the frame buffer. Furthermore, by controlling the processing order between the display driver and the image processor, low-latency rendering can be achieved.

[0006] In some embodiments, predicting the next frame image of the current frame image includes: predicting the pose data at the next moment based on the pose data at the current moment in the current frame image; and generating the next frame image corresponding to the pose data at the next moment.

[0007] Some embodiments of this application predict the pose data at the next moment by using the pose data at the current moment in the current frame image, and then generate the corresponding next frame image, thereby achieving accurate prediction of the next frame image and providing effective support for low-latency rendering.

[0008] In some embodiments, the display driver refreshes the color channel data in the current frame image by: refreshing the color channel data twice within one refresh cycle; wherein the one refresh cycle is the time interval between the current frame image and the next frame image.

[0009] Some embodiments of this application refresh the color channel data twice within one refresh cycle by the display driver, which can precisely control the time when the color channel data is drawn to the final frame buffer and avoid read conflicts.

[0010] In some embodiments, the time taken by the graphics processor to render the color channel data is less than or equal to one-sixth of a refresh cycle.

[0011] Some embodiments of this application can achieve low-latency rendering by controlling the rendering time of the graphics processor.

[0012] In some embodiments, rendering the color channel data of the next frame image using a graphics processor to obtain the rendering result of the color channel data includes: obtaining the pose change amount between the next frame image and the current frame image; constructing a correction matrix based on the pose change amount and the projection matrix; determining the texture coordinates in the color channel data through the correction matrix, and generating the rendering result by combining the pixel value of the current frame image at the texture coordinates.

[0013] Some embodiments of this application construct a correction matrix by measuring the pose change between the next frame image and the current frame image, and then combine the correction matrix with the pixel values ​​in the image to generate a rendering result, thereby achieving accurate rendering of the next frame image and improving the rendering effect.

[0014] Secondly, some embodiments of this application provide a frame image rendering apparatus, including: a prediction module, configured to predict the next frame image of the current frame image after the display driver has completed refreshing the color channel data in the current frame image; wherein the color channel data includes R channel data, G channel data or B channel data; and a rendering module, configured to render the color channel data of the next frame image through a graphics processor to obtain the rendering result of the color channel data.

[0015] In some embodiments, the prediction module is used to: predict the pose data at the next moment based on the pose data at the current moment in the current frame image; and generate the next frame image corresponding to the pose data at the next moment.

[0016] In some embodiments, the prediction module is configured to: refresh the color channel data twice within one refresh cycle using the display driver; the one refresh cycle being the time interval between the current frame image and the next frame image.

[0017] In some embodiments, the rendering module is used to obtain the pose change between the next frame image and the current frame image; construct a correction matrix based on the pose change and the projection matrix; determine the texture coordinates in the color channel data through the correction matrix, and generate the rendering result by combining the pixel value of the current frame image at the texture coordinates.

[0018] Thirdly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0019] Fourthly, some embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method as described in any embodiment of the first aspect.

[0020] Fifthly, some embodiments of this application provide a computer program product, the computer program product including a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of some embodiments of this application, the accompanying drawings used in some embodiments of this application will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of a frame image rendering method provided for some embodiments of this application; Figure 2 A schematic diagram of frame image rendering provided for some embodiments of this application; Figure 3 A block diagram of an apparatus for frame image rendering is provided for some embodiments of this application; Figure 4 A schematic diagram of an electronic device provided for some embodiments of this application. Detailed Implementation

[0023] The technical solutions of some embodiments of this application will now be described with reference to the accompanying drawings.

[0024] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0025] To facilitate understanding of the implementation process of the embodiments of this application, the technical terms involved below will be explained first.

[0026] Time Warp: A technique used to reduce the delay between head movement and image display. Since rendering takes time, the displayed image actually corresponds to the head posture a few milliseconds ago. Time Warp adjusts the image quickly to match the latest head position.

[0027] Line Sequential Display: Writes and displays all pixel information of a frame of image sequentially, line by line, on the screen.

[0028] Color Sequential Display: A complete color image is divided into multiple monochrome component images (usually red, green, and blue) according to color channels, and the display time of a frame is divided into multiple corresponding sub-field sequences. In each sub-field sequence, the display shows only one color component image, and the corresponding color backlight is turned on simultaneously. Due to the persistence of vision in the human eye, when this switching speed is fast enough, the brain will fuse the continuously appearing red, green, and blue monochrome images into a complete color image.

[0029] Single Buffer: Traditional rendering often uses a double or triple buffer mechanism. For example, with a double buffer, at any given moment, the GPU (Graphics Processing Unit) is writing to one buffer while the display is reading from another. Once completed, the two buffers are swapped. Single buffers often cause screen tearing due to simultaneous read and write operations on the same buffer.

[0030] In the AR field, high rendering latency often causes virtual objects to float, disrupting the realism of virtual-real integration. More seriously, it can trigger dizziness and nausea. Therefore, reducing latency is crucial in AR. A typical optimization solution for Line Sequential Display devices divides the Frame Buffer into two sub-buffers. While the display device is reading the upper half of the Frame Buffer for final display, the GPU is drawing to the lower half. In this case, low-latency rendering can be achieved by controlling the rendering time. However, in typical Color Sequential Display technology, the Frame Buffer is divided into three buffers (R, G, B) according to color channels and displayed sequentially. For this display method, it's impossible to divide the Frame Buffer into two parts for reading and writing. As can be seen from the above, current technologies offer few optimization solutions for Color Sequential Display, and there are no technologies for achieving low-latency rendering in Color Sequential Display.

[0031] In view of this, some embodiments of this application provide a method for rendering frame images, which combines ColorSequential Display technology and Time Warp technology to achieve low-latency rendering of frame images. Specifically, after the display driver refreshes the color channel data of the current frame image, the next frame image is predicted. Then, the graphics processor can render the color channel data of the next frame image to obtain the rendering result. Embodiments of this application can achieve low-latency rendering under ColorSequential Display technology.

[0032] The following is in conjunction with the appendix Figure 1 The present application provides an exemplary embodiment of the implementation process of frame image rendering performed by an AR device.

[0033] Please see the appendix Figure 1 , Figure 1 A flowchart of a frame image rendering method is provided for some embodiments of this application. The frame image rendering method may include: S110, after the display driver has finished refreshing the color channel data in the current frame image, predict the next frame image of the current frame image; wherein, the color channel data includes R channel data, G channel data or B channel data.

[0034] For example, in a specific embodiment of this application, the GPU separates the frame image into three channels: R, G, and B, thereby obtaining R channel data, G channel data, and B channel data. After refreshing the color channel data of the current frame image using the Display Driver (as a specific example of a display driver), it is necessary to predict the next frame image to facilitate subsequent rendering operations. The frame image can be an image related to the head pose.

[0035] In some embodiments of this application, S110 may include: refreshing the color channel data twice within a refresh cycle using the display driver; the refresh cycle being the time interval between the current frame image and the next frame image.

[0036] For example, in a specific embodiment of this application, it is assumed that the AR display device refreshes data twice within one VSync cycle (i.e., the refresh cycle of the display's vertical synchronization), and the Display Driver and GPU can simultaneously read and write the same FrameBuffer. Therefore, the Display Driver needs to refresh the color channel data of the (N-1)th frame (as a specific example of the current frame image) twice within one VSync cycle. The time corresponding to the (N-1)th frame image is T1, and the time corresponding to the Nth frame image (as a specific example of the next frame image) is T2. The time interval between T1 and T2 is one VSync cycle.

[0037] In some embodiments of this application, S110 may include: predicting the pose data at the next moment based on the pose data at the current moment in the current frame image; and generating the next frame image corresponding to the pose data at the next moment.

[0038] For example, in a specific embodiment of this application, before the image processor renders, it is first necessary to predict the pose data at time T2. A pre-trained prediction model or prediction algorithm can be used to predict the pose data at the next time (i.e., time T2) based on the current pose data at time T1, so as to generate the corresponding Nth frame image.

[0039] S120, the graphics processor renders the color channel data of the next frame image to obtain the rendering result of the color channel data. The time taken for the graphics processor to render the color channel data is less than or equal to one-sixth of a refresh cycle.

[0040] For example, in a specific embodiment of this application, after the Display Driver refreshes the color channel data of the second (N-1)th frame, based on the predicted Nth frame image, the GPU begins to draw the color channel data of the Nth frame image onto the FrameBuffer to obtain the rendering result. Since the Display Driver refreshes the data twice within one VSync cycle (i.e., refreshes the data of the R, G, and B channels twice respectively), the Display Driver and the GPU can read and write the same FrameBuffer simultaneously. Therefore, the time for the GPU to draw one color channel data is less than or equal to 1 / 6 of the VSync cycle.

[0041] In some embodiments of this application, S120 may include: obtaining the pose change amount between the next frame image and the current frame image; constructing a correction matrix based on the pose change amount and the projection matrix; determining the texture coordinates in the color channel data through the correction matrix, and generating the rendering result by combining the pixel value of the current frame image at the texture coordinates.

[0042] For example, in a specific embodiment of this application, the GPU needs to perform warp correction on the (N-1)th frame image during rendering to obtain an accurate rendering result. Specifically, the pose change of the head posture between the Nth frame image and the (N-1)th frame image is first calculated. For example, if the current pose data of the (N-1)th frame image is pose1 and the pose data of the Nth frame image is pose2, the pose change between the two is deltapose = pose2. The process begins with `inverse(pose1)`. Next, a warp matrix (as a concrete example of a correction matrix) is constructed using `deltapose` and the projection matrix used for rendering. The source UV coordinates (u, v) (i.e., texture coordinates) of the pixels in frame N in frame N-1 are calculated using the inverse of the warp matrix. Pixel values ​​at positions (u, v) in frame N-1 are sampled using bilinear interpolation, and these sampled pixel values ​​are assigned to the position (x, y) of each pixel in frame N, thus generating the corresponding rendering result.

[0043] The following is in conjunction with the appendix Figure 2 The specific process of frame image rendering provided by some embodiments of this application is illustrated by way of example.

[0044] Please see the appendix Figure 2 , Figure 2 This is a schematic diagram of frame image rendering provided for some embodiments of this application. Figure 2 The different colored boxes in the image represent the R, G, and B color channels (data) of the standard image.

[0045] Depend on Figure 2 As can be seen, when rendering a frame image, after the Display Driver refreshes the R channel data of the second N-1 frames (i.e., the red box) in one VSync cycle, the GPU starts drawing the R channel data of the Nth frame onto the Frame Buffer. When drawing, it is necessary to first predict the pose at time T2 (i.e., the pose data at the next time step), and then use the GPU to warp and render the image to obtain the rendering result under the R channel data.

[0046] Then, the same operation is performed on the G channel data. That is, when the Display Driver refreshes the G channel data of the N-1th frame for the second time, the GPU starts to draw the G channel data of the Nth frame onto the Frame Buffer. When drawing, it is also necessary to predict the pose at T2+1 / 6VSync time in advance. Then the GPU is used to warp and render the image to obtain the rendering result under the G channel data.

[0047] Finally, the same operation is performed on the B channel data. That is, when the Display Driver finishes refreshing the B channel data of the second N-1 frame, the GPU starts drawing the B channel data of the Nth frame onto the Frame Buffer. When drawing, it is also necessary to predict the pose at T2+2 / 6VSync time in advance. Then, the GPU is used to warp and render the image to obtain the rendering result under the B channel data.

[0048] It can be seen that the application provided Figure 1 The method implementation can be a loop operation, executed repeatedly. Figure 1 The implementation process can render the R, G, and B channel data sequentially, and finally obtain the rendering result of the complete frame image.

[0049] It is understood that the core idea of ​​the embodiments provided in this application is to separate the frame image into three channels (R, G, and B) (i.e., color channel data) on the GPU side, and precisely control the time when each color channel image is drawn to the final frame buffer through the event signals of the Display Driver, avoiding conflicts with the Display Driver reading from the frame buffer. Combined with Time Warp technology, low-latency rendering can be achieved. For example, the color channel data is refreshed twice on the display device within one VSync cycle. When the Display Driver refreshes the R channel image of the N-1th frame for the second time, the GPU starts drawing the R channel image of the Nth frame. After 1 / 3 of a VSync time, the R channel image of the Nth frame is refreshed and lit on the display screen. Then, combined with Time Warp, Time Warp only needs to adjust and match the head pose (i.e., pose data) after 1 / 3 of a VSync time. The shorter the time for Time Warp to adjust and match the head pose, the better the effect. If the optimization method provided in this application is not adopted, Time Warp needs to adjust and match the head pose after 2 VSync times, which will reduce the rendering effect. Therefore, the solution provided in this paper can reduce rendering latency while ensuring rendering quality.

[0050] Please refer to Figure 3 , Figure 3 The diagram illustrates a block diagram of a frame image rendering apparatus provided in some embodiments of this application. It should be understood that this frame image rendering apparatus corresponds to the method embodiments described above and is capable of performing the various steps involved in the method embodiments. The specific functions of this frame image rendering apparatus can be found in the description above; detailed descriptions are omitted here to avoid repetition.

[0051] Figure 3The frame image rendering apparatus includes at least one software function module that can be stored in memory or embedded in the frame image rendering apparatus in the form of software or firmware. The frame image rendering apparatus includes: a prediction module 310, used to predict the next frame image of the current frame image after the display driver has completed refreshing the color channel data in the current frame image; wherein the color channel data includes R channel data, G channel data or B channel data; and a rendering module 320, used to render the color channel data of the next frame image through a graphics processor to obtain the rendering result of the color channel data.

[0052] In some embodiments of this application, the prediction module 310 is used to: predict the pose data at the next moment based on the pose data at the current moment in the current frame image; and generate the next frame image corresponding to the pose data at the next moment.

[0053] In some embodiments of this application, the prediction module 310 is configured to: refresh the color channel data twice within one refresh cycle using the display driver; the one refresh cycle being the time interval between the current frame image and the next frame image.

[0054] In some embodiments of this application, the rendering module 320 is used to obtain the pose change amount between the next frame image and the current frame image; construct a correction matrix based on the pose change amount and the projection matrix; determine the texture coordinates in the color channel data through the correction matrix, and generate the rendering result by combining the pixel value of the current frame image at the texture coordinates.

[0055] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0056] Some embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can perform the operation of any of the methods corresponding to the methods provided in the above embodiments.

[0057] Some embodiments of this application also provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operation of any of the methods corresponding to the above embodiments provided in the above embodiments.

[0058] like Figure 4As shown, some embodiments of this application provide an electronic device 400, which includes a memory 410, a processor 420, and a computer program stored in the memory 410 and executable on the processor 420. When the processor 420 reads the program from the memory 410 via a bus 430 and executes the program, it can implement the methods of any of the above embodiments.

[0059] Processor 420 can process digital signals and may include various computing architectures. For example, it may be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 420 may be a microprocessor.

[0060] Memory 410 can be used to store instructions executed by processor 420 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 420 of this disclosure embodiment can be used to execute instructions in memory 410 to implement the methods shown above. Memory 410 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0061] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0062] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0063] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for rendering a frame image, characterized in that, include: Once the display driver has finished refreshing the color channel data in the current frame image, predict the next frame image of the current frame image; wherein, the color channel data includes R channel data, G channel data, or B channel data; The color channel data of the next frame image is rendered by a graphics processor to obtain the rendering result of the color channel data; The display driver refreshes the color channel data in the current frame image by: refreshing the color channel data twice within one refresh cycle; the refresh cycle is the time interval between the current frame image and the next frame image, and the time for the graphics processor to render the color channel data is less than or equal to one-sixth of the refresh cycle.

2. The method as described in claim 1, characterized in that, The prediction of the next frame image of the current frame image includes: Based on the pose data at the current moment in the current frame image, predict the pose data at the next moment; Generate the next frame image corresponding to the attitude data at the next moment.

3. The method as described in claim 1 or 2, characterized in that, The step of rendering the color channel data of the next frame image using a graphics processor to obtain the rendering result of the color channel data includes: Obtain the pose change between the next frame image and the current frame image; Based on the attitude change and projection matrix, a correction matrix is ​​constructed; The texture coordinates in the color channel data are determined by the correction matrix, and the rendering result is generated by combining the pixel value of the current frame image at the texture coordinates.

4. An apparatus for rendering frame images, characterized in that, include: The prediction module is used to predict the next frame image of the current frame image after the display driver has completed refreshing the color channel data in the current frame image; wherein the color channel data includes R channel data, G channel data or B channel data; The rendering module is used to render the color channel data of the next frame image through the graphics processor and obtain the rendering result of the color channel data; The display driver refreshes the color channel data in the current frame image by: refreshing the color channel data twice within one refresh cycle; the refresh cycle is the time interval between the current frame image and the next frame image, and the time for the graphics processor to render the color channel data is less than or equal to one-sixth of the refresh cycle.

5. The apparatus as described in claim 4, characterized in that, The prediction module is used for: Based on the pose data at the current moment in the current frame image, predict the pose data at the next moment; Generate the next frame image corresponding to the attitude data at the next moment.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is executed by a processor to perform the method as described in any one of claims 1-3.

7. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the computer program is executed by the processor to perform the method as described in any one of claims 1-3.

Images