Image processing method and device, display device, and medium

Abstract

Provided are an image processing method and device, a display device, and a medium. The image processing method comprises: on the basis of the time consumed by each rendering process within a predetermined time period prior to a current frame, determining a statistical rendering time consumed; on the basis of the statistical rendering time consumed, determining a selected image processing mode for an image to be rendered of the current frame from among a first mode, a second mode and a third mode, wherein the first mode, the second mode and the third mode are respectively associated with different time sequences between a rendering process in a rendering thread and a timewarp process in an asynchronous timewarp thread; and on the basis of the selected image processing mode, asynchronously performing a rendering process and a timewarp process on the image to be rendered of the current frame, to obtain output image data of the current frame.

Description

Image processing method and device, display device and medium Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method and device, a display device, and a medium. Background Art

[0002] Virtual Reality (VR) is a computer simulation method that can create and experience a virtual world. It fuses multi-source information to provide users with an interactive three-dimensional dynamic vision and simulates user behavior to immerse users in the simulated environment.

[0003] Virtual reality technology encompasses various aspects, including environmental simulation and user behavior perception. Environmental simulation refers to the use of computers to generate realistic, dynamic, three-dimensional images in real time. User behavior perception, on the other hand, means that an ideal virtual reality system can acquire all of a user's senses. In addition to visual perception generated by computer graphics, other senses include hearing, touch, force, motion, and even smell and taste, also known as multi-sensory perception.

[0004] The widespread promotion and application of virtual reality technology requires that VR applications themselves maintain a high system frame rate to provide the best rendering quality.

[0005] Summary of the Invention

[0006] According to one aspect of the present application, an image processing method is provided, the method comprising: determining a statistically calculated rendering time based on the time consumed by each rendering process within a predetermined time period before a current frame; selecting a selected image processing mode for an image to be rendered of the current frame from a first mode, a second mode, and a third mode based on the statistically calculated rendering time, wherein the first mode, the second mode, and the third mode are respectively associated with different timings between a rendering process in a rendering thread and a time warp process in an asynchronous time warp thread; and asynchronously performing rendering processing and time warp processing on the image to be rendered of the current frame according to the selected image processing mode to obtain output image data of the current frame.

[0007] According to another aspect of the present application, an image processing device is provided, comprising: a statistics module for determining a statistically calculated rendering time based on the time consumed by each rendering process within a predetermined time period before a current frame; a mode selection module for selecting a selected image processing mode for an image to be rendered of the current frame from a first mode, a second mode, and a third mode based on the statistically calculated rendering time, wherein the first mode, the second mode, and the third mode are respectively associated with different timings between the rendering process in the rendering thread and the time warp process in the asynchronous time warp thread; and a processing module for performing rendering processing and time warp processing on the image to be rendered of the current frame according to the selected image processing mode for the image to be rendered of the current frame, to obtain output image data of the current frame.

[0008] According to another aspect of the present application, an image processing device is provided, comprising: a processor and a memory, wherein the memory stores computer-executable instructions, and when the computer-executable instructions are executed by the processor, the image processing method as described above is performed.

[0009] According to another aspect of the present application, a display device is provided, including: a display; and the image processing device as described above.

[0010] According to another aspect of the present application, a non-volatile storage medium is provided, storing computer-executable instructions, wherein the computer-executable instructions are executed by a processor to perform the image processing method as described above.

[0011] The embodiments of the present application also provide an idea of ​​dynamically adjusting the system timing according to the real-time duration of rendering processing. By automatically switching to the optimal timing of rendering processing and time warping processing after real-time statistics of the time spent on rendering processing, it can not only improve the utilization efficiency of the processor, but also improve the delay and jitter problems caused by the long time spent on rendering processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative dimensions of the following figures may not be drawn to scale. Like reference numerals and names in the various drawings indicate like elements.

[0013] FIG1 is a schematic diagram showing the rendering and display process of a VR display device when a user quickly turns his head.

[0014] FIG2 is a schematic diagram showing the rendering and display process of a VR display device when a user turns his head quickly, using asynchronous time warping.

[0015] FIG3 shows a more specific implementation of the asynchronous time warping scheme.

[0016] FIG4 shows a schematic flow chart of an image processing method according to an embodiment of the present application.

[0017] FIG. 5 shows more details of step 430 when the image to be rendered is rendered as a whole and the current processing mode is the first mode.

[0018] FIG. 6 shows a schematic timing diagram of asynchronously performing rendering processing and time warping processing on the to-be-rendered image of the current frame in FIG. 5 .

[0019] FIG. 7 shows more details of step 430 when the image to be rendered is rendered as a whole and the current processing mode is the second mode.

[0020] FIG8 is a schematic timing diagram showing the asynchronous rendering process and the time warping process for the to-be-rendered image of the current frame in FIG7 .

[0021] FIG. 9 shows more details of step 430 when the image to be rendered is rendered as a whole and the current processing mode is the third mode.

[0022] FIG. 10 is a schematic timing diagram showing the asynchronous rendering process and the time warping process for the to-be-rendered image of the current frame in FIG. 9 .

[0023] FIG11 shows more details of step 430 when the image to be rendered is divided into a left-eye image and a right-eye image and the current processing mode is the first mode (low-time mode).

[0024] FIG. 12A is a schematic timing diagram showing the asynchronous rendering process and the time warping process for the to-be-rendered image of the current frame in FIG. 11 .

[0025] FIG. 12B shows an operation flow chart corresponding to the schematic timing diagram in FIG. 12A .

[0026] FIG13 shows more details of step 430 when the image to be rendered is divided into a left-eye image and a right-eye image and the selected image processing mode is the second mode (intermediate mode).

[0027] FIG. 14 is a schematic timing diagram showing the asynchronous rendering process and the time warping process for the to-be-rendered image of the current frame in FIG. 13 .

[0028] FIG. 15 is a schematic timing diagram showing asynchronous rendering and time warping of an image to be rendered of a current frame based on the timing of the prior art.

[0029] FIG16 shows more details of step 430 when the image to be rendered is divided into a left-eye image and a right-eye image and the current processing mode is the third mode (high time consumption mode).

[0030] FIG. 17 is a schematic timing diagram showing the asynchronous rendering process and the time warping process for the to-be-rendered image of the current frame in FIG. 16 .

[0031] FIG. 18 is a block diagram illustrating an image processing apparatus according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present disclosure more apparent, example embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the example embodiments described herein. Based on the embodiments of the present disclosure described in this disclosure, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present disclosure.

[0033] It should be noted that, in this specification and the drawings, substantially the same steps and elements are denoted by the same reference numerals, and repeated description of these steps and elements will be omitted.

[0034] According to exemplary embodiments, a virtual reality display device can generate an immersive virtual world environment based on the user's movements, perceptions, and other information for display to the user. The virtual reality display device may include motion sensors such as gyroscopes and accelerometers, a central processing unit (CPU), memory, a graphics processing unit (GPU), a communication device, and the like. Image processing and display operations, such as those described herein, can be performed by the virtual reality display device's central processing unit or graphics processing unit in conjunction with corresponding computer program code. The processor in the virtual reality display device can utilize various suitable operating systems, such as Android, Linux, Unix, or Windows.

[0035] Currently, mainstream commercial-grade VR systems are divided into two categories: tethered and mobile. Tethered VR systems offload complex rendering and computing tasks to a powerful host computer, and then transmit each frame of image rendered by the host computer to a separate virtual reality (VR) display device (e.g., a head-mounted device such as a virtual reality helmet or virtual reality glasses) via a high-speed data cable (such as HDMI). Mobile VR systems, on the other hand, do away with the host computer and data cables, using the smartphone's own processor (e.g., central processing unit (CPU) or graphics processing unit (GPU)) and display screen as the renderer and head-mounted device (including the display) to provide an immersive VR experience. In this case, the smartphone can serve as a virtual reality display device. The virtual reality display device can provide users with a single-screen image or a dual-screen image (e.g., when used with dual lenses, also called a left-eye image and a right-eye image).

[0036] Typically, VR systems offer a wide viewing angle, such as a 360° panorama. Consequently, in practical applications, the rendering and provision of scene image data for the corresponding portion of the scene must be tailored to the user's current posture. When using a virtual reality headset, rapid user movement can cause delays in scene image rendering. Consequently, the scene image the user sees may not match the scene image they expect at their new location, leading to delays in displaying the scene image and a feeling of dizziness.

[0037] FIG1 is a schematic diagram illustrating the rendering and display process of a VR display device when a user quickly turns their head. As shown in FIG1 , if a new image frame has not been rendered when the Nth frame vertical synchronization signal Vsync (indicating the start of the Nth display frame (in the context of this application, a "display frame" refers to a system frame, also referred to herein as a "frame")) arrives, that is, the central processing unit (CPU) or graphics processing unit (GPU) in the VR display device has not completed rendering the image of the N-1th frame, then the already rendered N-2th frame will be repeatedly displayed, resulting in the image of the N-2th frame being displayed twice. This does not cause any abnormality when the user keeps their head still. However, when the user rotates from the old head position to the new position, the image corresponding to the old head position is still displayed on the screen of the display device. At this time, the brain's reaction has switched to the new head position, but the eyes receive the image of the old head position. The mismatch between the two information can cause the user to feel dizzy, and the greater the rotation angle, the more severe the dizziness.

[0038] A vertical synchronization signal (Vsync) is a synchronization signal applied between two adjacent display frames of a display device, indicating the end of the previous display frame and the beginning of the next display frame. This signal is valid once before each frame scan and can be used to determine the field frequency of the display device, that is, the number of times the screen refreshes per second, also known as the refresh rate of the display device. For example, the refresh rate of a display device can be 60Hz, 120Hz, etc., meaning it refreshes 60 times, 120 times, etc. per second, with a display frame time of 1 / 60 second, 1 / 120 second, etc. When a display device includes two or more screens, each screen can correspond to a different vertical synchronization signal based on its own display timing and can be associated with (e.g., synchronized with) the vertical synchronization signal of the system frame. This vertical synchronization signal is generated, for example, by a display driver (e.g., a graphics card) and is used to synchronize the gate drive signals and data signals required to display a frame of image during the display process.

[0039] To reduce image rendering latency in a displayed scene (e.g., the time difference between the start of image rendering and the final display of the image), some virtual reality display devices employ timewarping (TW) technology. Timewarping is a technique that corrects image frames. It addresses scene image rendering latency by distorting (or correcting) rendered scene image data based on changes in user actions after rendering. Because timewarping occurs closer to the display time, the resulting image is closer to the image the user desires. Furthermore, because timewarping only processes two-dimensional images, similar to affine transformations in image processing, it does not impose excessive system workload overhead. Typically, timewarping and rendering are performed in the same thread, resulting in lengthy processing times for that thread and hindering the resolution of image latency. Furthermore, existing TW technologies perform both rendering and timewarping in the same thread. If the graphics card performance is low, the rendering and TW operations for the current frame cannot be completed sequentially within the rendering time of a single frame.

[0040] To this end, some virtual reality display devices also use asynchronous timewarp (ATW) technology, which uses asynchronous timewarp technology to further optimize the rendering image delay problem. Specifically, asynchronous timewarp technology can further optimize the above-mentioned timewarp technology by arranging rendering and timewarp processing in two different threads, so that rendering and timewarp processing can be performed asynchronously, thereby reducing the overall running time of rendering and timewarp processing. For example, when the virtual reality application cannot maintain a sufficient system frame rate, asynchronous timewarp processing will reprocess the previously rendered scene image according to the current user posture to produce a frame image (intermediate frame) that matches the current user posture, thereby reducing image jitter and further reducing latency.

[0041] FIG2 is a schematic diagram showing the rendering and display process of a VR display device when a user turns his head quickly, using asynchronous time warping.

[0042] As shown in Figure 2, before the arrival of each frame's vertical synchronization signal Vsync (indicating the start of frame N), time warping is performed to generate a new frame based on the most recently rendered frame. As shown in Figure 2, time warping is performed on frame N-1 in the period before the arrival of frame N's vertical synchronization signal Vsync, and on frame N in the period before the arrival of frame N+1's synchronization signal Vsync. Because the rendering of frame N-1 was not complete by the time the vertical synchronization signal Vsync of frame N arrives, the CPU or GPU cannot output a new rendered frame. Therefore, when the CPU or GPU performs time warping on frame N-1, it corrects the rendered image of frame N-2, which has already been rendered, based on the new head position, thereby generating an image corresponding to the new head position (which serves as the output image data of frame N-1 after ATW processing and is displayed within frame N). Next, because the rendering process of frame N has already been completed before the time warping process for frame N begins, the CPU or GPU can correct the rendered image of frame N, which has already been rendered, based on the new head position, thereby generating an image corresponding to the new head position (as the output image data of frame N after ATW processing, and displayed within the time of frame N+1). This method ensures that a new frame is generated based on the latest head position before the vertical synchronization signal (Vsync) signal of each frame arrives. This can greatly reduce the user's feeling of dizziness and improve the user's comfort in VR products.

[0043] In the asynchronous time warp scheme adopted in FIG. 2 , the timing of the rendering process and the asynchronous time warp process is merely an example. Since they are asynchronous, the timing can be implemented in various ways as will be described later.

[0044] To ensure that the image presented by a VR display device reflects all objects observable within the user's current field of view (based on the user's current posture), all objects within the user's field of view must typically be rendered. Furthermore, different images must be presented to the user's left and right eyes (i.e., parallax). The user's brain then synthesizes the different images for the left and right eyes, thereby presenting a visually three-dimensional image to the user. In one implementation, the object can be rendered as a whole before the rendered output is output to the first and second half-screens via an affine transformation, respectively, to be presented to the user as the left-eye image and the right-eye image, respectively. In another implementation, each rendering process in the rendering thread includes rendering the current frame's to-be-rendered image separately for both the left and right eyes (e.g., based on different user posture data). This approach yields more accurate and more three-dimensional images compared to the first embodiment. Alternatively, in other applications, only a single-screen image may be provided to the user. In this case, the scene image corresponding to the current user posture must be rendered as a whole before the rendered output is output.

[0045] Figure 3 illustrates a more specific implementation of the asynchronous time warping scheme. In Figure 3, the to-be-rendered images (including the left-eye image and the right-eye image) of the current display frame are rendered and time-warped. In this context, the left-eye image and the right-eye image may contain the same content as the to-be-rendered image obtained based on the predicted user gesture data at time b. In other words, the left-eye image and the right-eye image are identical.

[0046] For example, in the example of FIG3 , considering the general time consumption of the rendering process and the asynchronous time warp process, the rendering process and the asynchronous time warp process are operated in the following fixed timing sequence: the rendering process of the left-eye image and the right-eye image of the N-1th frame is started at the middle time of the N-1th frame, the rendering process of the left-eye image and the right-eye image of the N-1th frame is started at the middle time of the Nth frame, and the time warp process of the rendered left-eye image of the N-1th frame is started at the same time to obtain the left-eye output image data of the N-1th frame, and the time warp process of the rendered right-eye image of the N-1th frame is started at the beginning time of the N+1th frame. Time warping processing is performed (to obtain the right eye output image data of the N-1th frame), and the left eye screen scan can be started (that is, the left eye output image data of the Nth frame is displayed on the corresponding area of ​​the display (for example, the left screen)); at the middle moment of the N+1th frame, the left eye image and right eye image of the N+1th frame are rendered, and the time warping processing of the rendered left eye image of the Nth frame is started to obtain the left eye output image data of the Nth frame, and the right eye screen scan can be started (that is, the right eye output image data of the N-1th frame is displayed on the corresponding area of ​​the display (for example, the right screen)). The duration of each left eye screen scan (and right eye screen scan) is the time of one frame.

[0047] As shown in FIG3 , at the middle moment of the N-1th frame, the rendering of the image to be rendered (including the left eye image and the right eye image) of the N-1th frame begins (MRT processing begins). At this time, the image to be rendered can depend on the predicted user posture data at time point b, where time point b is closer to the actual display time. For example, it can be any time between the time when the left eye image of the N-1th frame is output and displayed (the time when the left eye screen scan ends, that is, time point a below) and the time when the right eye image of the N-1th frame is output and displayed (the time when the right eye screen scan ends, that is, time point c below), for example, an intermediate time. For example, as shown in FIG3 , time point b is located at 1 / 4 frame from the N+2th vertical synchronization signal, that is, the duration T1 = 2.75 frames from the start of the rendering process of the left eye image of the N-1th frame to time point b. In the example of FIG3 , since the timing of the MRT processing and the AWT processing is fixed and determined, time point b and time points a and c described below can be easily determined.

[0048] In addition, in the ATW thread, the time warping (ATW) process corrects and dedistorts the image rendered by the MRT process for each eye based on the predicted user posture data at a closer time (closer to the actual display time), obtaining left-eye output image data and right-eye output image data. For example, at the middle of frame N, when the ATW process obtains the rendered left-eye image of frame N-1 transmitted by the MRT process, it performs ATW processing on the rendered left-eye image of frame N-1 based on the predicted user posture data at time a (point a is the time when the left-eye image of frame N-1 is output and displayed), obtaining left-eye output image data. The duration from the start of ATW processing on the rendered left-eye image of frame N-1 to time a is T2 = 1.5 frames. The left-eye output image data is then sent to the backlight module. At the beginning of frame N+1, the left-eye output image data obtained by the ATW process begins scanning the screen, and the left-eye backlight is simultaneously displayed. At the same time, at the start of frame N+1, the rendered right-eye image of frame N-1 is subjected to ATW processing based on the predicted user gesture data at point c (point c is the moment when the right-eye image of frame N-1 is output and displayed), obtaining right-eye output image data. The duration from the start of ATW processing of the rendered right-eye image of frame N-1 to point c is T3 = T2 = 1.5 frames, and the right-eye output image data is sent to the backlight module. At the middle of frame N+1, the screen begins to be scanned based on the right-eye output image data obtained through ATW processing, and the right-eye backlight is simultaneously displayed.

[0049] It can be seen that under the refresh display mechanism of the traditional VR system, it takes a total of 3 frames (absolute delay) from the moment the rendering processing of the to-be-rendered image begins (i.e., the middle moment of the N-1th frame in Figure 3) to the moment when the to-be-rendered image is finally displayed on the screen after image processing (i.e., time point c in Figure 3).

[0050] As previously mentioned, in current asynchronous time warp schemes, rendering and asynchronous time warp processing are performed according to a unified or fixed sequence. However, the scene complexity of different VR applications varies significantly, resulting in significant differences in the rendering time required for each frame. This means that the existing fixed sequence does not take into account the actual rendering time. This results in low processor efficiency when rendering simple scenes, as the actual rendering time is short, but the processor waits for a certain period before performing asynchronous time warp processing. Furthermore, when rendering complex scenes, the longer rendering time leads to a lower processor output rate, which can cause frame drops and significant jitter. Therefore, to ensure that no frames are dropped and thus prevent jitter, ATW processing is generally performed after the rendering of each frame of the image to be rendered is completed, which in turn causes excessive delays. Furthermore, as previously described in conjunction with FIG3 , when operating based on a fixed sequence, regardless of the complexity of the scene image to be rendered, the absolute delay is as long as three frames to meet the typical large rendering time. This can cause dizziness, affect the VR experience, and reduce user enthusiasm for VR.

[0051] To this end, the embodiments of the present application also provide an idea of ​​dynamically adjusting the system timing according to the real-time duration spent on rendering processing. By automatically switching to the optimal timing of rendering processing and time warping processing after real-time statistics of the time spent on rendering processing, it can not only improve the utilization efficiency of the processor, but also improve the delay and jitter problems caused by the long time spent on rendering processing.

[0052] Figure 4 shows a schematic flow chart of an image processing method according to an embodiment of the present application. The method of Figure 4 can be executed by a processing module (eg, a CPU or a GPU) in a virtual reality display device.

[0053] As shown in FIG. 4 , in step S410 , the statistical rendering time is determined based on the time consumed by each rendering process within a predetermined time period before the current frame.

[0054] In this application, the current frame can be any display frame in the scene image displayed by the VR display device, and there can be other display frames before or after it.

[0055] For example, the predetermined time period can be M frames adjacent to the start time of the current frame, which is equivalent to setting a leading time window for the start time of the current frame, or it can be M frames that are not adjacent to the start time of the current frame, where M is greater than or equal to 1. Of course, in order to more realistically count the rendering processing time near the current frame, the M frames should not be too far apart from the start time of the current frame. The time taken for each rendering process can be the time required to render each frame of the image. Alternatively, the predetermined time period can also be a predetermined time period (for example, the time period when the VR scene was previously used). Optionally, the statistical rendering time can be the average, maximum, minimum or median value of the time taken for each rendering process within the predetermined time period. When the number of display frames before the current frame is less than M, the rendering process and time warping process are first performed according to the default timing, and the actual time taken for each rendering process is counted and recorded.

[0056] Of course, in actual VR applications, the rendering times of various scene images corresponding to consecutive user gestures in the same VR scene generally vary little. Therefore, for the same application scene, the statistical rendering times determined based on the actual rendering times of each rendering process every M frames are substantially the same, and thus the image processing mode selected each time as described below is substantially the same. Of course, even if the statistical rendering times determined based on the actual rendering times of each rendering process every M frames vary for the same application scene, the selected image processing mode can be switched in real time.

[0057] In step S420, based on the statistically calculated rendering duration, a selected image processing mode for the to-be-rendered image of the current frame is selected from a first mode, a second mode, and a third mode, wherein the first mode, the second mode, and the third mode are respectively associated with different timings between the rendering process in the rendering thread and the time warp process in the asynchronous time warp thread.

[0058] For example, as mentioned earlier, the scene complexity of different VR applications varies greatly, resulting in significant differences in the rendering time required for images used in different application scenarios. Therefore, different image processing modes can be dynamically selected based on the different rendering times. Different image processing modes correspond to different timings for rendering and time warping. That is, they are no longer fixed to a unified timing as shown in Figure 3, but can be switched dynamically. Different timings can, for example, involve performing rendering and time warping at different times within a frame or multiple adjacent frames.

[0059] For example, when the statistical rendering time is less than or equal to the first threshold, the first mode is selected as the image processing mode; and / or when the statistical rendering time is greater than the second threshold, the third mode is selected as the image processing mode; and / or when the statistical rendering time is greater than the first threshold and less than or equal to the second threshold, the second mode is selected as the image processing mode. In this example, the first mode can be called a low-time mode because the statistical rendering time is small, the third mode can be called a high-time mode because the statistical rendering time is large, and the second mode can be called an intermediate mode because the statistical rendering time is moderate.

[0060] In step S430 , according to the selected image processing mode, the image to be rendered of the current frame is asynchronously rendered and time-warped to obtain output image data of the current frame.

[0061] For example, the selected image processing mode corresponds to a specific timing of the rendering process and the time warping process, so that the above two processes are performed on the image to be rendered of the current frame according to the specific timing.

[0062] In the present application, the image to be rendered of the current frame can be determined based on the first user posture data predicted after a first predetermined time has passed from the start moment of the rendering process of the image to be rendered of the current frame, and the end moment of the first predetermined time is the expected display completion moment of the image to be rendered of the current frame.

[0063] Alternatively, in the case where the image to be rendered includes a left eye image and a right eye image, the end moment of the first predetermined time is any moment between the moment when the left eye image of the current frame is displayed and the moment when the right eye image of the current frame is displayed, for example, the middle moment (point b as shown in Figure 3).

[0064] Optionally, the prediction operation of the first user posture data can be performed by analyzing the motion data fed back by motion sensors such as accelerometers and gyroscopes to obtain the motion state of the current user's head, such as the motion direction, speed, acceleration, etc., and then predicting the head position that the user will reach at the next predetermined moment (for example, point b), thereby predicting the scene image that the user will see. For example, the first user posture data and other user posture data involved in the following description may include one or a combination of the direction and angle toward which the user is facing, the motion direction and angle, speed, acceleration, etc. For example, in an embodiment of the present disclosure, the prediction operation can calculate or predict the user posture after a given time based on the current user posture and the historical data of the user posture, thereby obtaining the user posture data at the display moment. The calculation or prediction method includes but is not limited to prediction using a neural network.

[0065] More details of asynchronously performing rendering processing and time warping processing on the to-be-rendered image of the current frame according to the selected image processing mode will be described in detail later.

[0066] Optionally, after the rendered image is rendered and the asynchronous time warp process obtains output image data, the CPU or GPU can send the output image data to the display module, i.e., throw the output image data on the display screen. The display module may include a virtual reality display and other circuits related to display drive and control. In addition, depending on the type of display, if backlight is required, the CPU or GPU can also calculate backlight data based on the output image data (e.g., based on LocalDiming (LD) algorithm), and send the backlight data to the backlight module (e.g., including backlight control circuit and backlight generation circuit) via SPI or USB data transmission mode, so that the backlight data of the backlight module is displayed synchronously with the output image data (screen scanning) of the display module.

[0067] According to the image processing method shown in reference Figure 4, by automatically switching to the optimal timing of rendering processing and time warping processing after real-time statistics of rendering time, the processor utilization efficiency can be improved, and the delay and jitter problems caused by the long time spent on rendering processing can be improved.

[0068] As described above, when rendering the image to be rendered of the current frame, in one embodiment, the image to be rendered can be rendered as a whole first, and then the rendering output results can be affine transformed to obtain the left eye image and the right eye image respectively and provided to the user; in another embodiment, each rendering process in the rendering thread includes rendering the first half-screen scene image data (for the left eye) and the second half-screen scene image data (for the right eye) of the current frame respectively.

[0069] FIG. 5 shows more details of step 430 when the image to be rendered is rendered as a whole and the current processing mode is the first mode.

[0070] As shown in FIG. 5 , in step S510 , at a first time point in the current frame, the rendering process of the image to be rendered of the current frame begins.

[0071] In step S520 , at a second time point that is a first time period from the first time point, time warping processing is started on the rendered image in the current frame, wherein the first time period is longer than or equal to the statistical rendering time.

[0072] For example, the duration of the first time period can be equal to the duration corresponding to the first threshold. Generally, for the convenience of timing, for example, the start time and the middle time of each frame are generally easy to determine and can usually be used as a timing reference. Therefore, the first time point can be the start time or the middle time of the current frame, and the first threshold can be 0.5 / FPS, and the second threshold can be 1 / FPS, where FPS is the system frame rate, and 1 / FPS is the time length of one frame. In this way, the duration of the first time period can be 0.5 / FPS. In addition, under normal circumstances, the duration of the ATW processing is assumed to be less than 0.25 / FPS in this application. Of course, when the duration of the ATW processing is greater than 0.25 / FPS, the corresponding timing can also be easily determined based on the various embodiments of this application (for example, when the duration of the ATW processing is 0.5 / FPS, the first time period can be 1 / FPS in the case shown in Figure 5).

[0073] Of course, these timings are merely examples, and it is only necessary to perform the corresponding time warping process as soon as possible after the image rendering process is completed.

[0074] For example, when the first time point is the start moment of the current frame, the second time point may be the middle moment of the current frame; and when the first time point is the middle moment of the current frame, the second time point may be the start moment of the next frame of the current frame.

[0075] Figure 6 shows a schematic timing diagram of asynchronously rendering and time warping the image to be rendered of the current frame in Figure 5. In Figure 6, the first time point is the start time of the current frame, the second time point is the middle time of the current frame, the first threshold is 0.5 / FPS, and the second threshold is 1 / FPS as an example.

[0076] As shown in Figure 6, VsyncN-1, VsyncN, VsyncN+1, and VsyncN+2 represent the vertical synchronization signal of the N-1th frame, the vertical synchronization signal of the Nth frame, the vertical synchronization signal of the N-1th frame, the vertical synchronization signal of the Nth frame, the vertical synchronization signal of the N+1th frame, and the vertical synchronization signal of the N+2th frame, respectively. The moment of receiving the vertical synchronization signal of the N-1th frame is defined as the start moment of the N-1th frame, the moment of receiving the vertical synchronization signal of the Nth frame is defined as the end moment of the N-1th frame or the start moment of the Nth frame, and so on. The middle moment between the start moment of the N-1th frame and the end moment of the N-1th frame is defined as the middle moment of the N-1th frame, the middle moment between the start moment of the Nth frame and the end moment of the Nth frame is defined as the middle moment of the Nth frame, and so on.

[0077] At the start time of the N-1th frame (herein referred to as the start time of the current frame), rendering begins on the image to be rendered of the N-1th frame (e.g., obtained based on the predicted user pose data at time point b), and at the middle time of the N-1th frame, time warping begins on the rendered image of the N-1th frame (i.e., the image to be rendered that has been rendered in the current frame) to obtain output image data for the N-1th frame. Then, the current frame is updated to the Nth frame in the figure, and at the start time of the Nth frame (herein referred to as the start time of the updated current frame), rendering begins on the image to be rendered of the Nth frame (e.g., obtained based on the predicted updated user pose data at time point b as described above), and at the middle time of the Nth frame, time warping begins on the rendered image of the Nth frame (i.e., the image to be rendered that has been rendered in the current frame) to obtain output image data for the Nth frame. The current frame is then updated to the N+1th frame in the figure, and this process is repeated.

[0078] Optionally, as described above, after obtaining the output image data, it can also be sent to the display module (screen scan) and the backlight data can be calculated and sent to the backlight module, so that the backlight data is displayed synchronously with the output image data of the display module. For example, in the first mode, and / or in the second mode and the third mode described later, the corresponding output image data is sent to the display module after each time warping process is completed, so that the display module starts to display the corresponding output image data 0.5 / FPS after the time warping process of the corresponding image starts, for example, the screen scan is started (the screen scan time is the time of one frame), and during this screen scan, the backlight data is displayed synchronously. In the case shown in Figure 6, the absolute delay is defined as the delay from the moment when the rendering process of the N-1th frame to be rendered is started to the moment when the corresponding output image is displayed, which is 2 frames. In this example, there is no left eye image and right eye image, so a, b, and c can be regarded as the same point.

[0079] FIG. 7 shows more details of step 430 when the image to be rendered is rendered as a whole and the current processing mode is the second mode.

[0080] As shown in Figure 7, in step S710, at a first time point in the current frame, the image to be rendered of the current frame is rendered. For example, for the convenience of timing, the first time point can be the beginning or middle of the current frame.

[0081] In step S720, at a second time point that is a second time period from the first time point, rendering processing is started on the image to be rendered of the next frame of the current frame, and time warping processing is performed on the rendered image of the current frame, wherein the duration of the second time period is greater than or equal to the statistical rendering time.

[0082] For example, the duration of the second time period may be equal to the duration corresponding to the second threshold. Similarly, for the convenience of timing, the first time point may be the start or middle of the current frame, and the first threshold may be 0.5 / FPS, and the second threshold may be 1 / FPS. Thus, the duration of the second time period may be 1 / FPS.

[0083] For example, when the first time point is the start moment of the current frame, the second time point may be the start moment of the next frame of the current frame; and when the first time point is the middle moment of the current frame, the second time point may be the middle moment of the next frame of the current frame.

[0084] Of course, these timings are merely examples; it is sufficient to perform the corresponding time warping process as quickly as possible after the rendering process is completed. FIG8 illustrates a schematic timing diagram of asynchronously performing rendering and time warping on the image to be rendered of the current frame in FIG7 . FIG8 illustrates an example in which the first time point is the start time of the current frame, and the second time point is the start time of the frame following the current frame.

[0085] As shown in FIG8 , at the start time of the N-1th frame (herein, the start time of the current frame), rendering of the image to be rendered of the N-1th frame begins, and at the start time of the Nth frame, rendering of the image to be rendered of the Nth frame begins, and time warping is performed on the rendered image of the N-1th frame (i.e., the image to be rendered that has been rendered in the current frame) to obtain output image data of the N-1th frame. Then, the current frame is updated to the Nth frame, and the above process is repeated. For example, at the start time of the N+1th frame, rendering of the image to be rendered of the N+1th frame begins, and time warping is performed on the rendered image of the Nth frame to obtain output image data of the Nth frame.

[0086] Furthermore, in the scenario shown in FIG8 , after obtaining the output image data for the N-1th frame, the corresponding output image data is sent to the display module, causing the display module to begin displaying the corresponding output image data 0.5 / FPS after the time warping process for the corresponding left-eye image begins (i.e., at the middle of the Nth frame). For example, a screen scan is initiated (the screen scan duration is the duration of one frame). During this screen scan, the backlight data is synchronously displayed, and the absolute delay is 2.5 frames. It should be understood that displaying the image on the screen can be performed based on vertical synchronization signals (not shown) occurring at the various middle moments shown in the figure, and the subsequent left-eye and right-eye screen scan operations are similarly performed based on their respective vertical synchronization signals.

[0087] FIG. 9 shows more details of step 430 when the image to be rendered is rendered as a whole and the current processing mode is the third mode.

[0088] As shown in Figure 9, in step S910, at a first time point in the current frame, the rendering process of the image to be rendered of the current frame begins. For example, for the convenience of timing, the first time point can be the beginning or middle of the current frame.

[0089] In step S920, at a second time point that is one frame time (1 / FPS) later than the first time point, rendering processing is started on the image to be rendered of the next frame of the current frame. For example, when the first time point is the start time of the current frame, the second time point may be the start time of the next frame of the current frame; and when the first time point is the middle time of the current frame, the second time point may be the middle time of the next frame of the current frame.

[0090] In step S930, at a third time point after the rendering of the current frame to be rendered image is completed, time warping is started on the rendered image of the current frame. For example, the approximate rendering completion time can be determined based on statistical rendering time consumption.

[0091] For example, because the rendering time of the current frame's to-be-rendered image exceeds the duration of one frame, it is necessary to perform time warping on the rendered image of the current frame after the rendering of the to-be-rendered image is completed. Optionally, for ease of timing, the third time point can be the closest time point after the current frame's rendering completion time point among P equally spaced time points within the frame containing the current frame's rendering completion time point.

[0092] Figure 10 shows a schematic timing diagram of asynchronous rendering and time warping of the image to be rendered of the current frame in Figure 9. In Figure 10, the rendering time is 1.25 frames, the first time point is the start time of the current frame, and P is 4 as an example.

[0093] As shown in Figure 10, at the start of frame N-1 (herein, the current frame), rendering begins for the image to be rendered of frame N-1, and at the start of frame N (herein, the next frame to the current frame), rendering begins for the image to be rendered of frame N. At 0.25 frames of frame N, the rendering process of frame N-1 has been completed. Therefore, at the middle of frame N, the rendered image of frame N-1 is time-warped to obtain the output image data of frame N-1. Then, the current frame is updated to frame N, and the above process is repeated. For example, at the start of frame N+1, rendering begins for the image to be rendered of frame N+1, and at the middle of frame N+1, the rendered image of frame N is time-warped to obtain the output image data of frame N.

[0094] In addition, in the case shown in Figure 10, after the output image data of the N-1th frame is obtained, the corresponding output image data is sent to the display module, so that the display module starts to display the corresponding output image data 0.5 / FPS after the start time of the time warping processing of the corresponding image (that is, the start time of the N+1th frame), for example, starts to scan the screen (the scanning time is the length of one frame), and during this scanning period, the backlight data is displayed synchronously, and the absolute delay is 3 frames at this time.

[0095] In this case, it may be necessary to perform simultaneous rendering processing on to-be-rendered images of two adjacent frames in the rendering thread. In this case, two or more buffers may be set to implement parallel implementation of multiple rendering processes.

[0096] The above describes an implementation method for rendering an image as a whole. While ensuring that each frame of the image to be rendered is rendered and time-warped, and the output image data is displayed (avoiding jitter caused by dropped frames), the absolute latency varies in different modes. Therefore, the most appropriate mode can be selected based on statistically analyzed rendering times. This implementation method is suitable for single-screen image output, as well as for applications where the entire image is rendered first and then the rendered output is affine transformed to generate left-eye and right-eye output image data.

[0097] In other implementations, the left eye image and the right eye image included in the image to be rendered can be rendered separately, where the left eye image and the right eye image can include the same content as the image to be rendered obtained based on the predicted user posture data at point b, that is, the left eye image and the right eye image are the same. At this time, the statistical rendering time in step S410 can further include the following steps: based on the first time of each rendering process for the left eye image and the second time of each rendering process for the right eye image within the predetermined time period before the current frame, determine the first statistical rendering time for the left eye image and the second statistical rendering time for the right eye image, wherein the first statistical rendering time is the same as the second statistical rendering time, and the statistical rendering time includes the first statistical rendering time and the second statistical rendering time. That is, the left eye image and the right eye image are processed separately (i.e., rendering processing and time warping processing). Optionally, in the rendering thread, the rendering processing of the left-eye image and the right-eye image for each frame can be parallel (e.g., based on multiple buffers) or serial (e.g., based on a single buffer), and in the asynchronous time warp thread, the time warp processing of the rendered left-eye image and the rendered right-eye image for each frame can be serial.

[0098] At this time, in the mode selection process in step S420, it is necessary to compare the first statistical rendering time or the second statistical rendering time with 1 / 2 of the first threshold (for example, 0.25 / FPS) and 1 / 2 of the second threshold (for example, 0.5 / FPS) to determine the selected image processing mode.

[0099] FIG11 shows more details of step 430 when the image to be rendered is divided into a left-eye image and a right-eye image and the current processing mode is the first mode (low-time mode).

[0100] As shown in Figure 11, in step S1110, at a first time point of the current frame, the left eye image of the current frame is rendered. For example, the first time point may be the beginning or middle of the current frame.

[0101] In step S1120, at a second time point that is a first time period away from the first time point, the right eye image of the current frame is started to be rendered, and the rendered left eye image of the current frame that has been rendered is started to be time-warped to obtain the left eye output image data of the current frame.

[0102] In step S1130, at a third time point after the first time period from the second time point, time warping processing is started on the rendered right eye image of the current frame that has been rendered to obtain right eye output image data of the current frame, wherein the duration of the first time period is greater than or equal to the first statistical rendering time and the second statistical rendering time.

[0103] For example, the duration of the first time period may be equal to the duration corresponding to 1 / 2 of the first threshold. Similarly, for the convenience of timing, the first time point may be the start or middle of the current frame, and the first threshold may be 0.5 / FPS, and the second threshold may be 1 / FPS. Thus, the first time period may be 0.25 / FPS.

[0104] For example, when the first time point is the start moment of the current frame, the second time point may be the 0.25 / FPS moment of the current frame, and the third time point may be the 0.5 / FPS moment of the current frame; and when the first time point is the middle moment of the current frame, the second time point may be the 0.75 / FPS moment of the current frame, and the third time point may be the start moment of the next frame of the current frame.

[0105] Optionally, as described above, after obtaining the left-eye output image data and the right-eye output image data of the current frame, the data may be sent to the display module (screen scanning) and backlight data may be calculated and sent to the backlight module, so that the backlight data is refreshed synchronously with the display of the output image data of the display module. For example, in the first mode, and / or in the second and third modes described below, after each time-warping process of the left-eye image is completed, the corresponding left-eye output image data is sent to the display module, so that the display module begins to display the corresponding left-eye output image data 0.25 / FPS after the time-warping process of the corresponding left-eye image is started; and after each time-warping process of the right-eye image is completed, the corresponding right-eye output image data is sent to the display module, so that the display module begins to display the corresponding right-eye output image data 0.25 / FPS after the time-warping process of the corresponding left-eye image is started. In the first mode, it is feasible to start displaying the corresponding output image data for both the left and right eyes after 0.5 / FPS from the start of time warping processing, or to start displaying the corresponding output image data after 0.5 / FPS for one of the left eye and 0.25 / FPS for the other, etc., which can also shorten various delays.

[0106] Figure 12A shows a schematic timing diagram of asynchronously rendering and time warping the image to be rendered of the current frame in Figure 11. Figure 12B shows a corresponding operation flow chart of the schematic timing diagram in Figure 12A. In Figure 12A, the first time point is taken as the middle moment of the current frame as an example.

[0107] As shown in Figure 12A, at the middle moment of the N-1th frame (here as the starting moment of the current frame), the left eye image of the image to be rendered of the N-1th frame starts to be rendered, and at the 0.75 / FPS moment of the N-1th frame, the time warping processing of the rendered left eye image of the N-1th frame starts (the output image data of the left eye image, i.e., the left eye output image data, can be obtained), and the right eye image of the image to be rendered of the N-1th frame starts to be rendered, and then at the starting moment of the Nth frame, the time warping processing of the rendered right eye image of the Nth frame starts (the output image data of the right eye image, i.e., the right eye output image data, can be obtained), and then the current frame is updated to the Nth frame, and the above process is repeated, for example, at the middle moment of the Nth frame, the left eye image of the Nth frame starts to be rendered, and so on.

[0108] Here, when performing time warping processing on the rendered left-eye image and the rendered right-eye image, user gesture data is also similarly involved.

[0109] For example, when performing time warping processing on the rendered left-eye image of the current frame, it is also possible to predict the second user posture data of the expected display completion moment of the left-eye image of the current frame (for example, the moment at point a shown in the figure) and the third user posture data of the expected display completion moment of the right-eye image of the current frame (for example, the moment at point c shown in the figure) in a manner similar to the prediction of the first user posture data. Based on the second user posture data, time warping processing is performed on the rendered left-eye image of the current frame that has been rendered; and based on the third user posture data, time warping processing is performed on the rendered right-eye image of the current frame that has been rendered.

[0110] Furthermore, in the case shown in FIG12A , after obtaining the left-eye output image data / right-eye output image data of the N-1th frame, the corresponding left-eye output image data / right-eye output image data are respectively sent to the display module, so that the display module begins displaying the corresponding left-eye output image data / right-eye output image data 0.25 / FPS (or 0.5 / FPS) after the time warping processing of the corresponding rendered left-eye image / rendered right-eye image begins. For example, a left-eye screen scan / right-eye screen scan is respectively initiated (the scan duration is the duration of one frame), and during this scan, the backlight data is synchronously displayed. In this case, the absolute delay from the start of rendering the left-eye image of the N-1th frame to the moment when the corresponding left-eye and right-eye output image data are both displayed (time point c) is 1.75 frames (or 2 frames), and as defined in FIG3 , T1 = 1.625 frames (or 1.75 frames) and T2 = 1.25 frames. The reduction of these delays compared to the solution in Figure 3 makes the posture data of each user obtained based on time prediction more accurate.

[0111] In addition, as shown in Figure 12B, in response to obtaining the vertical synchronization signal of the N-1th frame, it is determined in process 1 whether there is a rendered left eye image of the N-2th frame (i.e., whether the N-1th frame is the 1st frame). If not, enter process 2, and if yes, enter process 3.

[0112] In process 2 (for the case where the N-1th frame is the first frame), determine whether the middle moment of the N-1th frame (expressed as VsyncN-1*0.5) has been reached. If so, proceed to process 4, otherwise wait. In process 4, start rendering the left eye image of the N-1th frame at the middle moment of the N-1th frame; in process 5, determine whether the 0.75 / FPS moment of the N-1th frame (expressed as Vsync(N-1)*0.75) has been reached. If so, proceed to process 6, otherwise wait. In process 6, start rendering the right eye image of the N-1th frame at 0.75 / FPS of the N-1th frame, and at the same time start time warping processing of the rendered left eye image of the N-1th frame, and calculate the left eye backlight data based on the obtained left eye output image data and send it to the backlight module. In process 7, determine whether the vertical synchronization signal of the Nth frame (expressed as Vsync(N)) has been obtained. If so, proceed to process 8, otherwise wait. In process 8, at the moment (start moment) when the vertical synchronization signal of the Nth frame is obtained, the time warping process of the rendered right eye image of the N-1th frame is started, and at the same time, the left eye output image data of the N-1th frame is sent to the display module for left eye scanning (the left eye backlight data is displayed synchronously with the left eye output image data), and the right eye backlight data is calculated and sent to the backlight module. In process 9, it is determined whether the middle moment of the Nth frame is reached (expressed as VsyncN*0.5). If so, proceed to process 10, otherwise wait. In process 10, at the middle moment of the Nth frame, the right eye output image data of the N-1th frame is sent to the display module for right eye scanning (the right eye backlight data is displayed synchronously with the right eye output image data), and at the same time, the left eye image of the Nth frame is started to be rendered, and then N is replaced by N-1, and processes 5-10 are repeated.

[0113] In process 3 (for the case where the N-1th frame is not the first frame), determine whether the middle moment of the N-1th frame (expressed as VsyncN-1*0.5) has been reached. If so, proceed to process 11, otherwise wait. In process 11, start rendering the left eye image of the N-1th frame at the middle moment of the N-1th frame, and start sending the right eye output image data of the N-2th frame to the display module for right eye screen scanning; in process 12, determine whether the 0.75 / FPS moment of the N-1th frame (expressed as VsyncN-1*0.75) has been reached. If so, proceed to process 13, otherwise wait. In process 13, start rendering the right eye image of the N-1th frame at the 0.75 / FPS moment of the N-1th frame, and start time-warping processing on the rendered left eye image of the N-1th frame at the same time, and calculate the left eye backlight data based on the obtained left eye output image data and send it to the backlight module. In process 14, a determination is made as to whether the vertical synchronization signal (Vsync(N)) for frame N has been obtained. If so, the process proceeds to process 15; otherwise, the process waits. In process 15, at the moment the vertical synchronization signal for frame N has been obtained (the start moment), time warping of the rendered right-eye image for frame N-1 begins. Simultaneously, the left-eye output image data for frame N-1 begins to be sent to the display module for left-eye scanning. Right-eye backlight data is calculated and sent to the backlight module. N is then replaced with N-1, and processes 3 and 11-15 are repeated.

[0114] FIG13 shows more details of step 430 when the image to be rendered is divided into a left-eye image and a right-eye image and the selected image processing mode is the second mode (intermediate mode).

[0115] As shown in Figure 13, in step S1310, at a first time point of the current frame, the left eye image of the current frame is rendered. For example, the first time point may be the beginning or middle of the current frame.

[0116] In step S1320 , at a second time point that is a second time period later than the first time point, the right eye image of the current frame is rendered, and the rendered left eye image of the current frame is time-warped.

[0117] In step S1330, at a third time point after a second time period from the second time point, time warping processing is started on the rendered right eye image of the current frame, and rendering processing is performed on the left eye image of the next frame of the current frame, wherein the duration of the second time period is greater than or equal to the first statistical rendering time and the second statistical rendering time.

[0118] For example, the duration of the second time period may be equal to the duration corresponding to 1 / 2 of the second threshold. Similarly, for the convenience of timing, the first time point may be the start or middle of the current frame, and the first threshold may be 0.5 / FPS, and the second threshold may be 1 / FPS. Thus, the second time period may be 0.5 / FPS.

[0119] For example, when the first time point is the start moment of the current frame, the second time point may be the middle moment of the current frame, and the third time point may be the start moment of the next frame of the current frame; and when the first time point is the middle moment of the current frame, the second time point may be the start moment of the next frame of the current frame, and the third time point may be the middle moment of the next frame of the current frame.

[0120] Fig. 14 shows a schematic timing diagram of asynchronously rendering and time warping the image to be rendered of the current frame in Fig. 13. In Fig. 14, the first time point is taken as an example as the middle moment of the current frame.

[0121] As shown in FIG14 , at the middle of the N-1th frame (the current frame), the rendering process begins for the left-eye image of the image to be rendered in the N-1th frame. At the start of the Nth frame, the rendering process begins for the right-eye image of the image to be rendered in the N-1th frame, and time warping begins for the rendered left-eye image of the N-1th frame. Then, at the middle of the Nth frame, the rendering process begins for the left-eye image of the image to be rendered in the Nth frame, and time warping begins for the rendered right-eye image of the Nth frame. At the start of the Nth frame, the current frame is updated to the Nth frame, and the above process is repeated. For example, at the middle of the Nth frame, the rendering process begins for the left-eye image of the image to be rendered in the Nth frame.

[0122] In addition, in the case shown in FIG14 , after obtaining the left eye output image data / right eye output image data of the N-1th frame respectively, the corresponding left eye output image data / right eye output image data are sent to the display module respectively, so that the display module starts to display the corresponding left eye output image data / right eye output image data 0.25 / FPS (or 0.5 / FPS) after the time warping processing of the corresponding rendered left eye image / rendered right eye image is started. For example, the left eye screen scan / right eye screen scan is started respectively (the screen scan duration is the duration of one frame), and during this screen scan, the backlight data is displayed synchronously, and the absolute delay at this time is 2.25 frames, and as defined in FIG3 , T1 = 1.75 frames and T2 = 1.25 frames. The reduction of these delays relative to the scheme in FIG3 makes the individual user posture data obtained based on time prediction more accurate.

[0123] The operation flow chart for FIG. 14 is similar in content to the operation flow chart shown in FIG. 12B , and thus will not be described again here.

[0124] In some cases, such as when VR applications use complex scenes, rendering can take longer than a single frame. This means the left-eye and right-eye rendering times are both greater than 0.5 / FPS, potentially resulting in a 4-frame delay. Given that rendering can take significantly longer, for example, more than two frames, this can lead to even greater delays.

[0125] Specifically, as shown in FIG15 , based on the existing timing, rendering of the image to be rendered begins at the middle of frame N−1. If the rendered image (rendered left-eye image and rendered right-eye image) of frame N−1 is not obtained at the middle of frame N, the time warping process will be performed on the rendered left-eye image of frame N−1 at the middle of frame N+1. Time warping of the rendered right-eye image of frame N−2 will begin at the beginning of frame N+2, and the rendered left-eye image after time warping will begin to be displayed (left-eye screen scan). Display of the rendered right-eye image after time warping of frame N−2 will begin at the middle of frame N+2, and the display will be completed at the middle of frame N+3. In this way, for the image to be rendered of frame N−1, there is an absolute delay of 4 frames from acquisition to final display on the screen, with T1 = 3.75 frames and T2 = 1.5 frames. Alternatively, even if the start time of the first ATW process in the figure is shifted to the left by 0.5 frames, there is still an absolute delay of 3.5 frames. In addition, if the rendering time is longer, this timing may also cause frame drops, which will cause more serious jitter problems. Therefore, in this application, a third mode is proposed to address such situations where the rendering time is longer.

[0126] FIG16 shows more details of step 430 when the image to be rendered is divided into a left-eye image and a right-eye image and the current processing mode is the third mode (high time consumption mode).

[0127] As shown in FIG. 16 , in step S1610 , at the first time point of the current frame, the left-eye image and the right-eye image of the current frame are rendered in parallel.

[0128] In step S1620, at a second time point after the rendering of the left-eye image and the right-eye image of the current frame is completed, time warping of the rendered left-eye image of the current frame is initiated. For example, the approximate rendering completion time can be determined based on the first statistical rendering time for the left-eye image and the second statistical rendering time for the right-eye image. Optionally, for timing purposes, the second time point can correspond to the closest moment after the rendering completion time of the current frame among P equally spaced moments within the frame containing the rendering completion time of the current frame.

[0129] In step S1630, at a third time point one frame later than the first time point, time warping processing is started on the rendered right eye image of the current frame, and rendering processing is started on the left eye image and right eye image of the next frame of the current frame in parallel.

[0130] Similarly, the first threshold can be 0.5 / FPS, and the second threshold can be 1 / FPS. In this way, when the first and second statistical rendering times are both less than 0.75 / FPS, when the first time point is the start time of the current frame, the second time point can be the 0.75 / FPS time of the current frame, and the third time point can be the start time of the next frame after the current frame. When the first time point is the middle time of the current frame, the second time point can be the 0.25 / FPS time of the next frame after the current frame, and the third time point can be the middle time of the next frame after the current frame. Figure 17 shows a schematic timing diagram of asynchronous rendering processing and time warping processing of the image to be rendered of the current frame in Figure 16. In Figure 17, the rendering time of the left eye image and the right eye image is 0.7 frames, the first time point is the middle time of the current frame, P=4, the second time point is the 0.25 / FPS time of the next frame after the current frame, and the third time point is the middle time of the second frame as an example.

[0131] As shown in Figure 17, at the middle of frame N-1, the left-eye and right-eye images of the image to be rendered of frame N-1 begin to be rendered in parallel. At 0.25 / FPS of frame N, time warping begins on the rendered left-eye image of frame N-1, and at the middle of frame N, time warping begins on the rendered right-eye image of frame N-1, while rendering begins simultaneously on the left-eye and right-eye images of frame N. At the start of frame N, the current frame is updated to frame N, and the above process repeats.

[0132] In addition, in the case shown in FIG17 , after obtaining the left eye output image data / right eye output image data of the N-1th frame respectively, the corresponding left eye output image data / right eye output image data are sent to the display module respectively, so that the display module starts to display the corresponding left eye output image data / right eye output image data 0.25 / FPS (or 0.5 / FPS) after the time warping processing of the corresponding rendered left eye image / rendered right eye image starts. For example, the left eye screen scan / right eye screen scan is started respectively (the screen scan duration is the duration of one frame), and during this screen scan, the backlight data is displayed synchronously, and the absolute delay at this time is 2.25 frames, and as defined in FIG3 , T1 = 1.75 frames, T2 = 1.25 frames. The reduction of these delays relative to the scheme in FIG15 makes the various user posture data obtained based on time prediction more accurate.

[0133] The content of the operation flowchart for FIG16 is similar to that of the operation flowchart shown in FIG12B , and therefore will not be repeated here.

[0134] The above is a description of an implementation method for separately rendering and time-warping the left-eye image and the right-eye image of the rendered image. This implementation method can be applied to dual-screen image output scenarios (e.g., virtual reality glasses), and the image-processed left-eye output image data and right-eye output image data can be output to the left and right display areas (e.g., two lenses), respectively.

[0135] FIG. 18 is a block diagram illustrating an image processing apparatus according to an embodiment of the present disclosure.

[0136] 18 , an image processing apparatus 1800 according to some embodiments of the present disclosure includes a statistics module 1810 , a mode selection module 1820 , and a processing module 1830 .

[0137] The statistics module 1810 is configured to determine a statistically calculated rendering duration based on the duration of each rendering process within a predetermined time period before the current frame.

[0138] The mode selection module 1820 is configured to select, based on the statistically calculated rendering time, a selected image processing mode for the to-be-rendered image of the current frame from a first mode, a second mode, and a third mode, wherein the first mode, the second mode, and the third mode are respectively associated with different timings between the rendering process in the rendering thread and the time warp process in the asynchronous time warp thread.

[0139] The processing module 1830 is configured to perform rendering processing and time warping processing on the image to be rendered of the current frame according to the selected image processing mode, to obtain output image data of the current frame.

[0140] According to different functional division methods, the image processing device 1800 can perform module division in different ways. For example, each module can be further divided into sub-modules, or can also include other modules.

[0141] More details of the operations implemented by the above modules may be similar to those described with reference to Figures 4-17, so they will not be repeated here.

[0142] Each module in the above-mentioned image processing device 1800 can be implemented by software, hardware, firmware, or any combination thereof. The method steps of other embodiments of the present disclosure can also be implemented by software, hardware, firmware, or any combination thereof.

[0143] In addition, the image processing device 1800 may be included in a virtual reality display device, for example, as at least a part of its processor, and, for example, combined with a display module and a backlight module, etc., to provide users with various scene images, thereby providing an immersive virtual world environment.

[0144] In addition, some embodiments of the present disclosure further provide an image processing apparatus, comprising a processor and a memory, wherein computer program instructions are stored in the memory. When the computer executable instructions are executed by a computer, the image processing method according to the embodiments of the present disclosure can be implemented.

[0145] Therefore, another aspect of the present application further provides a display device, comprising a display and the image processing device as described above.

[0146] The display device provided herein may include, but is not limited to, a smartphone, a tablet computer, a media player, and the like. It should be noted that for clarity, the entire structure of the device is not shown. To achieve the necessary functions of the device, those skilled in the art may configure other structures not shown according to specific application scenarios, and the present invention does not impose any limitations thereto.

[0147] The computer-readable storage medium may be any combination of one or more computer-readable storage media, for example, one computer-readable storage medium may contain a computer-readable program code for randomly generating a sequence of action instructions, and another computer-readable storage medium may contain a computer-readable program code for performing facial activity recognition.

[0148] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0149] The processor in the embodiments of the present disclosure may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor, and may be an X86 architecture or an ARM architecture.

[0150] The computer-readable storage medium in the embodiments of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0151] The image processing method and apparatus of the disclosed embodiments can dynamically adjust the system timing based on the real-time duration of rendering processing. By automatically switching to the optimal timing for rendering processing and time warping processing after real-time statistics of the time spent on rendering processing, the method and apparatus can not only improve the efficiency of processor utilization, but also alleviate the delay and jitter problems caused by the excessive time spent on rendering processing.

[0152] The exemplary embodiments of the present invention described in detail above are merely illustrative and not restrictive. Those skilled in the art will appreciate that various modifications, combinations or sub-combinations may be made to these embodiments without departing from the principles and spirit of the present invention, and such modifications should fall within the scope of the present invention.

Claims

1. An image processing method, comprising: Determine the statistical rendering time based on the time consumed by each rendering process within a predetermined time period before the current frame; Based on the statistical rendering time, select a selected image processing mode for the to-be-rendered image of the current frame from a first mode, a second mode, and a third mode, wherein the first mode, the second mode, and the third mode are respectively associated with different timings between a rendering process in a rendering thread and a time warp process in an asynchronous time warp thread; as well as According to the selected image processing mode, the image to be rendered of the current frame is asynchronously rendered and time-warped to obtain output image data of the current frame.

2. The method according to claim 1, wherein: Based on the statistical rendering time consumption, selecting a selected image processing mode for the image to be rendered of the current frame from the first mode, the second mode, and the third mode, including: When the statistical rendering time is less than or equal to a first threshold, using the first mode as the selected image processing mode; and / or In a case where the statistical rendering time is greater than a second threshold, using the third mode as the selected image processing mode; and / or When the statistical rendering time is greater than the first threshold and less than or equal to the second threshold, the second mode is used as the selected image processing mode.

3. The method according to claim 2, wherein: When the selected image processing mode is the first mode, according to the selected image processing mode, asynchronously performing rendering processing and time warping processing on the image to be rendered of the current frame, including: At a first time point in the current frame, start rendering the image to be rendered of the current frame; and At a second time point after a first time period from the first time point, time warping processing is started on the rendered image of the current frame, wherein the duration of the first time period is greater than or equal to the counted rendering time.

4. The method according to claim 2, in, When the selected image processing mode is the second mode, according to the selected image processing mode, asynchronously performing rendering processing and time warping processing on the image to be rendered of the current frame, including: At a first time point in the current frame, start rendering the image to be rendered of the current frame; and At a second time point after a second time period from the first time point, rendering processing is started on the image to be rendered of the next frame of the current frame, and time warping is performed on the rendered image of the current frame, wherein the duration of the second time period is greater than or equal to the statistical rendering time.

5. The method according to claim 2, wherein: When the selected image processing mode is the third mode, according to the selected image processing mode, asynchronously performing rendering processing and time warping processing on the image to be rendered of the current frame, including: At a first time point in the current frame, start rendering the image to be rendered of the current frame; At a second time point after one frame of time from the first time point, starting to render the image to be rendered of the next frame of the current frame; and At a third time point after the rendering process of the to-be-rendered image of the current frame is completed, the time warping process of the rendered image of the current frame begins.

6. The method according to claim 2, wherein: The image to be rendered for each frame includes a left-eye image and a right-eye image. The statistical rendering time consumption is determined based on the time consumption of each rendering process in a predetermined time period before the current frame, including: Determine a first statistical rendering time for the left-eye image and a second statistical rendering time for the right-eye image based on a first time consumption for each rendering process for the left-eye image and a second time consumption for each rendering process for the right-eye image in the predetermined time period before the current frame, respectively; The first statistical rendering time is the same as the second statistical rendering time, and the statistical rendering time includes the first statistical rendering time and the second statistical rendering time.

7. The method according to claim 6, wherein: When the first and second statistical rendering times are less than or equal to 1 / 2 of the first threshold, so that the selected image processing mode is the first mode, rendering processing and time warping processing are performed on the image to be rendered of the current frame according to the selected image processing mode, including: At the first time point of the current frame, start rendering the left eye image of the current frame; At a second time point after a first time period from the first time point, starting to render the right eye image of the current frame, and starting to time warp the rendered left eye image of the current frame; At a third time point from the second time point through the first time period, start time warping the rendered right eye image of the current frame, wherein the duration of the first time period is greater than or equal to the first statistical rendering time and the second statistical rendering time.

8. The method according to claim 6, wherein: When the first and second statistical rendering times are greater than 1 / 2 of the first threshold and less than or equal to 1 / 2 of the second threshold, so that the selected image processing mode is the second mode, according to the selected image processing mode, asynchronously performing rendering processing and time warping processing on the to-be-rendered image of the current frame, including: At a first time point of the current frame, start rendering the left eye image of the current frame; and At a second time point after a second time period from the first time point, starting to render the right eye image of the current frame, and performing a time warping process on the rendered left eye image of the current frame that has been rendered; and At a third time point that is after the second time point and the second time period, start to perform time warping on the rendered right eye image of the current frame that has been rendered, and perform rendering on the left eye image of the next frame of the current frame, wherein the duration of the second time period is greater than or equal to the first statistical rendering time and the second statistical rendering time.

9. The method according to claim 6, wherein: When the first and second statistical rendering times are greater than 1 / 2 of the second threshold, so that the selected image processing mode is the third mode, according to the selected image processing mode, asynchronously performing rendering processing and time warping processing on the to-be-rendered image of the current frame, including: At a first time point of the current frame, starting to render the left eye image and the right eye image of the current frame in parallel; At a second time point after the rendering process of the left-eye image and the right-eye image of the current frame is completed, starting to perform time warping process on the rendered left-eye image of the current frame; At a third time point after one frame from the first time point, the current The rendered right-eye image of the frame is time-warped, and the rendering process of the left-eye image and the right-eye image of the next frame begins in parallel.

10. The method according to claim 6, further comprising: Sending the output image data of the current frame to a display module; Backlight data is generated based on the output image data of the current frame and sent to a backlight module, so that the backlight data of the backlight module is displayed synchronously with the output image data of the display module.

11. The method according to claim 10, wherein: Sending the output image data of the current frame to the display module includes: Sending corresponding left-eye output image data to the display module after each time warping process of the left-eye image is completed, so that the display module starts to display the corresponding left-eye output image data after 0.25 / FPS from the start time of the time warping process of the corresponding left-eye image; and After each time warping of the right eye image is completed, the corresponding right eye output image data is sent to the display module, so that the display module starts to display the corresponding right eye output image data 0.25 / FPS after the time warping of the corresponding left eye image starts.

12. The method according to any one of claims 3 to 5, further comprising: predicting first user gesture data after a first predetermined time has passed since the start of rendering processing of the to-be-rendered image of the current frame, and Determine the image to be rendered of the current frame based on the first user posture data, The end time of the first predetermined time is the expected display completion time of the to-be-rendered image of the current frame.

13. The method according to any one of claims 7 to 9, further comprising: predicting first user gesture data after a first predetermined time has passed since the start of rendering processing of the to-be-rendered image of the current frame, and Determine the image to be rendered of the current frame based on the first user posture data, The end time of the first predetermined time is any time between the expected display completion time of the left eye image of the current frame and the expected display completion time of the right eye image of the current frame.

14. The method according to claim 13, wherein: According to the selected image processing mode, rendering processing and time warping processing are performed on the image to be rendered of the current frame, and further comprising: Predicting the second user gesture data of the expected display completion time of the left eye image of the current frame, and predicting the third user gesture data of the expected display completion time of the right eye image of the current frame, Performing a time warping process on the rendered left-eye image of the current frame based on the second user gesture data; and A time warping process is performed on the rendered right-eye image of the current frame based on the third user gesture data.

15. An image processing device, comprising: A statistics module, used to determine the statistical rendering time based on the time consumed by each rendering process within a predetermined time period before the current frame; a mode selection module, configured to select, based on the statistical rendering time, a selected image processing mode for the to-be-rendered image of the current frame from a first mode, a second mode, and a third mode, wherein the first mode, the second mode, and the third mode are respectively associated with different timings between a rendering process in a rendering thread and a time warp process in an asynchronous time warp thread; as well as The processing module is used to perform rendering processing and time warping processing on the image to be rendered of the current frame according to the selected image processing mode for the image to be rendered of the current frame, so as to obtain output image data of the current frame.

16. An image processing device, comprising: A processor and a memory, wherein the memory stores computer executable instructions, and when the computer executable instructions are executed by the processor, the image processing method according to any one of claims 1 to 14 is performed.

17. A display device, comprising: monitor; as well as The image processing apparatus according to claim 16.

18. A non-volatile storage medium storing computer executable instructions, wherein: When the computer executable instructions are executed by a processor, the image processing method according to any one of claims 1 to 14 is performed.

Images