Image processing method and device, display device and medium

Abstract

The invention provides an image processing method and device, a display device and a medium. The image processing method comprises the following steps: determining counted rendering time consumption based on time consumption of each rendering processing in a preset time period before a current frame; based on the counted rendering time consumption, selecting a selected image processing mode for the to-be-rendered image of the current frame from a first mode, a second mode and a third mode, wherein the first mode, the second mode and the third mode are respectively associated with different time sequences between rendering processing in the rendering thread and time warping processing in the asynchronous time warping thread; and according to the selected image processing mode, asynchronously performing rendering processing and time warping processing on the to-be-rendered image of the current frame to obtain output image data of the current frame.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method and device, a display device and a medium. Background Art

[0002] Virtual Reality (VR) is a computer simulation method that can create and experience a virtual world. It fuses multi-source information to provide users with an interactive three-dimensional dynamic vision and simulates user behavior to immerse users in the simulated environment.

[0003] Virtual reality technology includes various aspects such as environment simulation technology and user behavior perception technology. Environment simulation technology refers to the generation of real-time dynamic three-dimensional realistic images by computer. User behavior perception technology means that an ideal virtual reality system can know all the perceptions of the user. In addition to the visual perception generated by computer image technology, there are also perceptions such as hearing, touch, force, movement, and even smell and taste, which are also called multi-perception.

[0004] The widespread promotion and application of virtual reality technology requires that the VR application itself maintain a higher system frame rate to provide the best rendering quality. Summary of the invention

[0005] According to one aspect of the present application, an image processing method is provided, the method comprising: determining a statistically calculated rendering time based on the time consumed by each rendering process within a predetermined time period before a current frame; selecting a selected image processing mode for an image to be rendered of the current frame from a first mode, a second mode, and a third mode based on the statistically calculated rendering time, wherein the first mode, the second mode, and the third mode are respectively associated with different timings between a rendering process in a rendering thread and a time warp process in an asynchronous time warp thread; and according to the selected image processing mode, asynchronously performing rendering processing and time warp processing on the image to be rendered of the current frame to obtain output image data of the current frame.

[0006] According to another aspect of the present application, an image processing device is provided, comprising: a statistics module, for determining a statistically calculated rendering time based on the time consumed by each rendering process within a predetermined time period before a current frame; a mode selection module, for selecting a selected image processing mode for an image to be rendered of the current frame from a first mode, a second mode and a third mode based on the statistically calculated rendering time, wherein the first mode, the second mode and the third mode are respectively associated with different timings between a rendering process in a rendering thread and a time warp process in an asynchronous time warp thread; and a processing module, for performing a rendering process and a time warp process on the image to be rendered of the current frame according to the selected image processing mode for the image to be rendered of the current frame, so as to obtain output image data of the current frame.

[0007] According to another aspect of the present application, an image processing device is provided, comprising: a processor and a memory, wherein the memory stores computer executable instructions, and when the computer executable instructions are executed by the processor, the image processing method as described above is performed.

[0008] According to another aspect of the present application, a display device is provided, including: a display; and the image processing device as described above.

[0009] According to another aspect of the present application, a non-volatile storage medium is provided, storing computer executable instructions, wherein the computer executable instructions are executed by a processor to perform the image processing method as described above.

[0010] The embodiments of the present application also provide an idea of ​​dynamically adjusting the system timing according to the real-time duration spent on rendering processing. By automatically switching to the optimal timing of rendering processing and time warping processing after real-time statistics of the time spent on rendering processing, the processor utilization efficiency can be improved, and the delay and jitter problems caused by the long time spent on rendering processing can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and in the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative sizes of the following figures may not be drawn to scale. The same reference numerals and names in the various drawings indicate the same elements.

[0012] Figure 1 A schematic diagram showing the rendering and display process of a VR display device when a user turns his head quickly.

[0013] Figure 2 A schematic diagram showing the rendering and display process of a VR display device when a user turns his head quickly when asynchronous time warping is used.

[0014] Figure 3 A more specific implementation of the asynchronous time warp scheme is shown.

[0015] Figure 4 A schematic flow chart of an image processing method according to an embodiment of the present application is shown.

[0016] Figure 5 More details of step 430 are shown when the image to be rendered is rendered as a whole and the current processing mode is the first mode.

[0017] Figure 6 Shows Figure 5 A schematic timing diagram of asynchronously performing rendering processing and time warping processing on the to-be-rendered image of the current frame.

[0018] Figure 7 More details of step 430 are shown when the image to be rendered is rendered as a whole and the current processing mode is the second mode.

[0019] Figure 8 Shows Figure 7 A schematic timing diagram of asynchronously performing rendering processing and time warping processing on the to-be-rendered image of the current frame.

[0020] Fig. 9 More details of step 430 are shown when the image to be rendered is rendered as a whole and the current processing mode is the third mode.

[0021] Fig.10 Shows Fig. 9 A schematic timing diagram of asynchronously performing rendering processing and time warping processing on the to-be-rendered image of the current frame.

[0022] Fig.11 More details of step 430 are shown when the image to be rendered is divided into a left-eye image and a right-eye image and the current processing mode is the first mode (low-time consumption mode).

[0023] Fig. 12A Shows Fig.11 A schematic timing diagram of asynchronously performing rendering processing and time warping processing on the to-be-rendered image of the current frame.

[0024] Fig. 12B Shows Fig. 12A The operation flow chart corresponding to the schematic timing diagram in .

[0025] Fig.13 More details of step 430 are shown when the image to be rendered is divided into a left-eye image and a right-eye image and the selected image processing mode is the second mode (intermediate mode).

[0026] Fig.14 Shows Fig.13 A schematic timing diagram of asynchronously performing rendering processing and time warping processing on the to-be-rendered image of the current frame.

[0027] Fig.15 A schematic timing diagram showing asynchronous rendering and time warping of an image to be rendered of a current frame based on the timing of the prior art is shown.

[0028] Fig.16 More details of step 430 are shown when the image to be rendered is divided into a left-eye image and a right-eye image and the current processing mode is the third mode (high time consumption mode).

[0029] Fig.17 Shows Fig.16 A schematic timing diagram of asynchronously performing rendering processing and time warping processing on the to-be-rendered image of the current frame.

[0030] Fig.18 is a block diagram showing an image processing apparatus according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein. Based on the embodiments of the present disclosure described in the present disclosure, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present disclosure.

[0032] It should be noted that, in the present specification and the drawings, substantially the same steps and elements are denoted by the same reference numerals, and repeated description of these steps and elements will be omitted.

[0033] According to an exemplary embodiment, the virtual reality display device can generate an immersive virtual world environment based on the user's actions, perceptions, and other information to display to the user. The virtual reality display device may include motion sensors such as gyroscopes and accelerometers, a central processing unit (CPU), a memory, a graphics processing unit (GPU), a communication device, and the like. The image processing and display operations described herein may be performed, for example, by the central processing unit or the graphics processing unit of the virtual reality display device in conjunction with corresponding computer program codes, and the processor in the virtual reality display device may use various appropriate operating systems, such as an Android system, a Linux system, a Unix system, or a Windows system, and the like.

[0034] The current mainstream commercial-grade VR systems are divided into two categories: wired and mobile. The wired VR system delegates the complex rendering and computing work to a powerful host, and then transmits each frame of the image rendered by the host to a separate virtual reality (VR) display device (e.g., head-mounted device, such as a virtual reality helmet, virtual reality glasses, etc.) through a high-speed data cable (HDMI, etc.). The mobile VR system abandons the host and data cable, and uses the smartphone's own processor (e.g., central processing unit (CPU) or image processing unit (GPU)) and display screen as a renderer and head-mounted device (including a display) to provide an immersive VR experience. At this time, the smartphone can be used as a virtual reality display device. The virtual reality display device can provide users with single-screen images or dual-screen images (e.g., when used in conjunction with dual lenses, also called left-eye images and right-eye images).

[0035] Generally, the scene viewing angle of a VR system is generally large, such as a 360° panoramic view, so in practical applications, it is necessary to render and provide the corresponding part of the scene image data according to the current user posture. When using a virtual reality head-mounted device, if the user moves too fast, it will cause a delay in the rendering of the scene image. Therefore, the scene image seen by the user is inconsistent with the scene image of the new position he expects, resulting in a delay in displaying the scene image and the user's dizziness.

[0036] Figure 1 FIG. 1 is a schematic diagram showing the rendering and display process of a VR display device when a user turns his head quickly. Figure 1As shown, if there is no new image frame output that has been rendered when the Nth frame vertical synchronization signal Vsync (indicating the start of the Nth display frame (in the context of this application, "display frame" refers to the system frame, also referred to as "frame" here)) arrives, that is, the central processing unit (CPU) or image processing unit (GPU) in the VR display device has not been able to complete the rendering of the image of the N-1th frame, then the already rendered N-2th frame will be repeatedly displayed, resulting in the image of the N-2th frame being displayed twice. When the user keeps the head still, this will not cause abnormal phenomena. However, when the user rotates from the old head position to the new position, the image corresponding to the old head position is still displayed on the screen of the display device. At this time, the brain reaction has switched to the new head position, but the eyes receive the image of the old head position. The mismatch between the two information will cause the user to feel dizzy, and the greater the rotation angle, the stronger the dizziness will be.

[0037] The vertical synchronization signal (Vsync) is a synchronization signal applied between two adjacent display frames of the display device, which indicates the end of the previous display frame and the beginning of the next display frame. This signal is valid once before each frame scan. The field frequency of the display device can be determined by this signal, that is, the number of times the screen is refreshed per second, which can also be called the refresh frequency of the display device. For example, the refresh frequency of the display device can be 60Hz, 120Hz, etc., that is, it is refreshed 60 times, 120 times, etc. per second, and a display frame time is 1 / 60 second, 1 / 120 second, etc. When the display device includes two screens or more screens, each screen can correspond to a different vertical synchronization signal according to its own display timing, and can be associated with the vertical synchronization signal of the system frame (for example, synchronized). The vertical synchronization signal is generated by a display driver (such as a graphics card) and is used to synchronize the gate drive signal, data signal, etc. required to display a frame of the picture during the display process.

[0038] In order to reduce the delay of rendering images in the display scene (for example, the time difference between the start of the rendering process of the image and the final display of the image), some virtual reality display devices use time warp (TW) technology. Time warp technology is a technology for correcting image frames. It solves the problem of scene image rendering delay by distorting (or correcting) the scene image data after rendering based on the change of user actions after rendering. Since the time when the time warp process is performed is closer to the display time, the new display image obtained after the time warp process is closer to the image that the user wants to see. At the same time, since the time warp technology only processes two-dimensional images, it is similar to the affine transformation in image processing, and it does not bring too much system workload overhead. Usually, the time warp process and the rendering process are in the same thread, which leads to a long processing time of the thread, affecting the solution to the image delay problem. In addition, the rendering process and the time warp process of the existing TW technology are performed in the same thread. If the graphics card performance is relatively low, it cannot complete the current frame image rendering process and the current frame image TW processing operations in sequence within one frame rendering time.

[0039] To this end, some virtual reality display devices also use asynchronous time warp (ATW) technology, which uses asynchronous time warp technology to further optimize the rendering image delay problem. Specifically, the asynchronous time warp technology can further optimize the above-mentioned time warp technology, which arranges the rendering and time warp processing in two different threads respectively, so that the rendering processing and the time warp processing can be performed asynchronously, thereby reducing the overall running time of the rendering processing and the time warp processing. For example, when the virtual reality application cannot maintain a sufficient system frame rate, the asynchronous time warp processing will re-process the previously rendered scene image according to the current user posture to generate a frame picture (intermediate frame) that conforms to the current user posture, thereby reducing the jitter of the picture to better reduce the delay.

[0040] Figure 2 A schematic diagram showing the rendering and display process of a VR display device when a user turns his head quickly when asynchronous time warping is used.

[0041] like Figure 2 As shown, before the vertical synchronization signal Vsync (indicating the start of the Nth frame) of each frame arrives, the time warping process generates a new frame based on the frame that has recently completed the rendering process. Figure 2As shown, the time warping process is performed on the N-1th frame in a time period before the arrival of the vertical synchronization signal Vsync of the Nth frame, and the time warping process is performed on the Nth frame in a time period before the arrival of the N+1th synchronization signal Vsync. Since the rendering process of the N-1th frame is not completed when the vertical synchronization signal Vsync of the Nth frame arrives, that is, the CPU or GPU cannot output a new frame that has been rendered, when the CPU or GPU performs the time warping process on the N-1th frame, it will correct the rendered image of the N-2th frame that has been rendered based on the new head position, thereby generating an image corresponding to the new head position (as the output image data of the N-1th frame that has been processed by ATW, and displayed within the time of the Nth frame). Next, since the rendering process of the Nth frame has been completed before the time warping process for the Nth frame is started, that is, the CPU or GPU can correct the rendered image of the Nth frame that has been rendered based on the new head position, thereby generating an image corresponding to the new head position (as the output image data of the Nth frame that has been processed by ATW, and displayed within the time of the N+1th frame). This method ensures that a new frame will be generated according to the latest head position before the vertical synchronization signal Vsync signal of each frame arrives. This can greatly reduce the user's dizziness and improve the user's comfort in VR products.

[0042] Figure 2 In the asynchronous time warp scheme adopted in , the timing of the rendering process and the asynchronous time warp process is only an example. Since they are asynchronous, the timing can be in many ways as will be described later.

[0043] In order to enable the user to use the VR display device, the image presented by the VR display device can be all objects that the user can observe in the current field of view (based on the current user posture), generally all objects within the user's field of view need to be rendered. At the same time, it is necessary to present different images to the user's left eye and right eye (that is, to form parallax), and then the user's brain synthesizes the different images of the left and right eyes, so as to present a visually three-dimensional image to the user. In one implementation, the object can be rendered as a whole first, and then the rendering output result can be output to the first half screen and the second half screen respectively through affine transformation, so as to be provided to the user as a left eye image and a right eye image respectively. In another implementation, each rendering process in the rendering thread includes rendering the image to be rendered of the current frame for both the left eye and the right eye (for example, based on different user posture data). In this way, compared with the first implementation, a more accurate and more three-dimensional image can be obtained. Alternatively, in other applications, only a single-screen image can be provided to the user, in which case the scene image corresponding to the current user posture needs to be rendered as a whole and then the rendering output result is output.

[0044] Figure 3 A more specific implementation of the asynchronous time warp scheme is shown. Figure 3 In the present embodiment, the image to be rendered (including the left eye image and the right eye image) of the current display frame is rendered and time-warped. In this paper, the left eye image and the right eye image may include the same content as the image to be rendered obtained based on the predicted user posture data at point b, that is, the left eye image and the right eye image are the same.

[0045] For example, in Figure 3 In the example of FIG. 1 , considering the general time consumption of the rendering process and the asynchronous time warp process, the rendering process and the asynchronous time warp process are operated in the following fixed timing: the rendering process of the left eye image and the right eye image of the N-1th frame is started at the middle time of the N-1th frame, the rendering process of the left eye image and the right eye image of the N-1th frame is started at the middle time of the N frame, and the time warp process of the rendered left eye image of the N-1th frame is started at the same time to obtain the left eye output image data of the N-1th frame, and the time warp process of the rendered right eye image of the N-1th frame is started at the beginning time of the N+1th frame. The left eye image and the right eye image of the N+1 frame are rendered at the middle of the N+1 frame, and the time warping processing is started on the rendered left eye image of the N+1 frame to obtain the left eye output image data of the N-1 frame, and the right eye screen scanning can be started (that is, the right eye output image data of the N-1 frame is displayed on the corresponding area of ​​the display (for example, the right screen)). The duration of each left eye screen scanning (and right eye screen scanning) is one frame.

[0046] like Figure 3 As shown, at the middle moment of the N-1th frame, the image to be rendered (including the left eye image and the right eye image) of the N-1th frame starts to be rendered (MRT processing starts). At this time, the image to be rendered may depend on the predicted user posture data at the time point b, where the time point b is closer to the actual display time, for example, it can be any time between the time when the left eye image of the N-1th frame is output and displayed (the time when the left eye screen scan ends, that is, the time point a in the following text) and the time when the right eye image of the N-1th frame is output and displayed (the time when the right eye screen scan ends, that is, the time point c in the following text), for example, the middle time. For example, as Figure 3 As shown, the time point b is located at 1 / 4 frame from the N+2th vertical synchronization signal, that is, the time length T1 from the start of the rendering process of the left eye image of the N-1th frame to the time point b is 2.75 frames. Figure 3In the example, since the timing of the MRT process and the AWT process is fixed and determined, the time of point b and the time of point a and point c described later can be easily determined.

[0047] In addition, in the ATW thread, the time warping (ATW) process corrects and dedistorts the image rendered by the MRT process for the left and right eyes based on the predicted user posture data at a relatively recent moment (closer to the actual display moment), and obtains the left eye output image data and the right eye output image data. For example, at the middle moment of the Nth frame, when the ATW process obtains the rendered left eye image of the N-1th frame transmitted by the MRT process, the ATW process is performed on the rendered left eye image of the N-1th frame based on the predicted user posture data at time point a (point a is the time when the output and display of the left eye image of the N-1th frame is completed), and the left eye output image data is obtained, wherein the time length from the time when the ATW process of the rendered left eye image of the N-1th frame starts to the time point a is T2=1.5 frames, and the left eye output image data is sent to the backlight module. At the beginning of the N+1th frame, the left eye output image data obtained based on the ATW process starts to scan the screen, and the left eye backlight is displayed synchronously at the same time. At the same time, at the beginning of the N+1th frame, the rendered right eye image of the N-1th frame is subjected to ATW processing based on the predicted user posture data at point c (point c is the moment when the right eye image of the N-1th frame is output and displayed), and the right eye output image data is obtained, wherein the duration from the moment when the ATW processing of the rendered right eye image of the N-1th frame is started to the point c is T3=T2=1.5 frames, and the right eye output image data is sent to the backlight module. At the middle moment of the N+1th frame, the screen is scanned based on the right eye output image data obtained by the ATW processing, and the right eye backlight is displayed synchronously at the same time.

[0048] It can be seen that under the traditional VR system refresh display mechanism, from the moment when the rendering process of the image to be rendered begins (i.e., Figure 3 The time from the middle moment of the N-1th frame in the image to be rendered to the moment when the image processing is finally completed and the image to be rendered is displayed on the screen (that is, Figure 3 It takes a total of 3 frames (absolute delay) to reach the time point c in the figure.

[0049] As mentioned above, in the current asynchronous time warp scheme, the rendering process and the asynchronous time warp process are operated according to a unified or fixed timing. However, the scene complexity of different VR applications varies greatly, which results in a large difference in the rendering time for each frame of the image. That is, the existing fixed timing does not take into account the actual rendering time. As a result, when the rendering scene is simple, the actual rendering time is short, but the processor waits for a certain period of time before performing asynchronous time processing, resulting in low processor utilization efficiency. When the rendering scene is complex, the long rendering time leads to a low processor output rate, which may cause frame drops, resulting in more obvious jitter problems. Therefore, in order to ensure that no frames are dropped and thus prevent jitter, it is generally necessary to perform ATW processing after the rendering process of each frame of the image to be rendered is completed, which will cause excessive delays. In addition, as mentioned above in combination with Figure 3 As described, based on fixed timing operations, regardless of how complex the scene image to be rendered is, in order to meet the usual longer rendering time, the absolute delay is as long as 3 frames, which can cause dizziness, affect the VR experience, and reduce the user's enthusiasm for VR experience.

[0050] To this end, an embodiment of the present application also provides an idea of ​​dynamically adjusting the system timing according to the real-time duration spent on rendering processing. By automatically switching to the optimal timing of rendering processing and time warping processing after real-time statistics of the time spent on rendering processing, the processor utilization efficiency can be improved, and the delay and jitter problems caused by the long time spent on rendering processing can be improved.

[0051] Figure 4 A schematic flow chart of an image processing method according to an embodiment of the present application is shown. Figure 4 The method may be executed by a processing module (eg, a CPU or a GPU) in a virtual reality display device.

[0052] like Figure 4 As shown, in step S410, the statistical rendering time is determined based on the time consumed by each rendering process within a predetermined time period before the current frame.

[0053] In the present application, the current frame may be any display frame in which a VR display device displays a scene image, and there may be other display frames before or after it.

[0054] For example, the predetermined time period may be M frames adjacent to the start time of the current frame, which is equivalent to setting a leading time window for the start time of the current frame, or may be M frames not adjacent to the start time of the current frame, where M is greater than or equal to 1. Of course, in order to more realistically count the rendering processing time near the current frame, the M frames should not be too far away from the start time of the current frame. The time consumed for each rendering process may be the time required to render each frame of the image. Alternatively, the predetermined time period may also be a predetermined time period (for example, a time period in which the VR scene was previously used). Optionally, the statistical rendering time consumption may be the average, maximum, minimum or median value of the time consumption for each rendering process within the predetermined time period, etc. When the number of display frames before the current frame is less than M, the rendering process and time warping process are first performed according to the default timing, and the actual time consumption of each rendering process is counted and recorded.

[0055] Of course, in actual VR applications, the rendering time of each scene image corresponding to continuous user gestures in the same VR scene is generally not much different. Therefore, for the same application scene, the statistical rendering time determined based on the actual time of each rendering process of every M frames is basically the same, so the image processing mode selected each time as described below is basically the same. Of course, even if the statistical rendering time determined based on the actual time of each rendering process of every M frames is different for the same application scene, the selected image processing mode can be switched in real time.

[0056] In step S420, based on the statistically calculated rendering time, a selected image processing mode for the image to be rendered of the current frame is selected from a first mode, a second mode, and a third mode, wherein the first mode, the second mode, and the third mode are respectively associated with different timings between the rendering process in the rendering thread and the time warp process in the asynchronous time warp thread.

[0057] For example, as mentioned above, the scene complexity of different VR applications varies greatly, which makes the rendering time of images used in different application scenarios vary greatly. Therefore, different image processing modes can be dynamically selected according to different rendering times. Different image processing modes correspond to different timings of rendering processing and time warping processing, that is, no longer like Figure 3 The timing is not fixed to a unified timing as shown, but can be switched dynamically. Different timings may, for example, involve rendering and time warping at different moments in a frame or a plurality of adjacent frames.

[0058] For example, when the statistical rendering time is less than or equal to the first threshold, the first mode is selected as the image processing mode; and / or when the statistical rendering time is greater than the second threshold, the third mode is selected as the image processing mode; and / or when the statistical rendering time is greater than the first threshold and less than or equal to the second threshold, the second mode is selected as the image processing mode. In this example, the first mode can also be called a low-time mode because the statistical rendering time is small, the third mode can correspond to a high-time mode because the statistical rendering time is large, and the second mode can also be called an intermediate mode because the statistical rendering time is moderate.

[0059] In step S430, according to the selected image processing mode, the image to be rendered of the current frame is asynchronously rendered and time-warped to obtain output image data of the current frame.

[0060] For example, the selected image processing mode corresponds to a specific timing of the rendering process and the time warping process, so that the above two processes are performed according to the specific timing for the image to be rendered of the current frame.

[0061] In the present application, the image to be rendered of the current frame can be determined based on the first user posture data predicted after a first predetermined time has passed from the start time of the rendering process of the image to be rendered of the current frame, and the end time of the first predetermined time is the expected display completion time of the image to be rendered of the current frame.

[0062] Alternatively, in the case where the image to be rendered includes a left-eye image and a right-eye image, the end time of the first predetermined time is any time between the time when the display of the left-eye image of the current frame is completed and the time when the display of the right-eye image of the current frame is completed, for example, an intermediate time (such as Figure 3 point b shown).

[0063] Optionally, the prediction operation of the first user posture data can be performed by, for example, analyzing the motion data fed back by motion sensors such as accelerometers and gyroscopes to obtain the motion state of the current user's head, such as the motion direction, speed, acceleration, etc., and then predicting the head position that the user will reach at the next predetermined moment (for example, point b), thereby predicting the scene image that the user will see. For example, the first user posture data and other user posture data involved in the following description may include one or a combination of the direction and angle toward which the user is facing, the direction and angle of motion, speed, acceleration, etc. For example, in an embodiment of the present disclosure, the prediction operation can calculate or predict the user posture after a given time based on the current user posture and the historical data of the user posture, thereby obtaining the user posture data at the display moment. The method of calculation or prediction includes but is not limited to prediction using a neural network.

[0064] According to the selected image processing mode, more details of asynchronously performing rendering processing and time warping processing on the to-be-rendered image of the current frame will be described in detail later.

[0065] Optionally, after rendering the image to be rendered and asynchronous time warping to obtain output image data, the CPU or GPU can send the output image data to the display module, that is, throw the output image data on the display screen. The display module may include a virtual reality display and other circuits related to display drive and control. In addition, depending on the type of display, if backlight is required, the CPU or GPU can also calculate backlight data based on the output image data (for example, based on LocalDiming (LD) algorithm), and send the backlight data to the backlight module (for example, including a backlight control circuit and a backlight generation circuit) via SPI or USB data transmission, so that the backlight data of the backlight module is displayed synchronously with the output image data display (screen scanning) of the display module.

[0066] According to reference Figure 4 The image processing method shown can improve the processor utilization efficiency and improve the delay and jitter problems caused by the long rendering time by automatically switching to the optimal timing of rendering processing and time warping processing after real-time statistics of rendering time.

[0067] As described above, when rendering the image to be rendered of the current frame, in one embodiment, the image to be rendered can be rendered as a whole first, and then the rendering output results can be provided to the user by obtaining the left eye image and the right eye image through affine transformation; in another embodiment, each rendering process in the rendering thread includes rendering the first half-screen scene image data (for the left eye) and the second half-screen scene image data (for the right eye) of the current frame respectively.

[0068] Figure 5 More details of step 430 are shown when the image to be rendered is rendered as a whole and the current processing mode is the first mode.

[0069] like Figure 5 As shown, in step S510, at a first time point in the current frame, the rendering process of the image to be rendered of the current frame begins.

[0070] In step S520, at a second time point after a first time period from the first time point, a time warping process is started on the rendered image in the current frame, wherein the duration of the first time period is greater than or equal to the statistical rendering time.

[0071] For example, the duration of the first time period may be equal to the duration corresponding to the first threshold. Usually, in order to facilitate timing, for example, the start time and the middle time of each frame are generally easy to determine and can usually be used as a timing reference. Therefore, the first time point may be the start time or the middle time of the current frame, and the first threshold may be 0.5 / FPS, and the second threshold may be 1 / FPS, where FPS is the system frame rate, and 1 / FPS is the time length of a frame. In this way, the duration of the first time period may be 0.5 / FPS. In addition, under normal circumstances, in this application, it is assumed that the duration of ATW processing is less than 0.25 / FPS. Of course, when the duration of ATW processing is greater than 0.25 / FPS, the corresponding timing may be easily determined based on the various embodiments of the present application (for example, when the duration of ATW processing is 0.5 / FPS, in Figure 5 In the case shown, the first time period can be 1 / FPS).

[0072] Of course, these timings are merely examples, and it is only necessary to perform the corresponding time warping process as soon as possible after the image rendering process is completed.

[0073] For example, when the first time point is the start time of the current frame, the second time point may be the middle time of the current frame; and when the first time point is the middle time of the current frame, the second time point may be the start time of the next frame of the current frame.

[0074] Figure 6 Shows Figure 5 A schematic timing diagram of asynchronously rendering and time warping the image to be rendered of the current frame. Figure 6 In the example, the first time point is the start time of the current frame, the second time point is the middle time of the current frame, the first threshold is 0.5 / FPS, and the second threshold is 1 / FPS.

[0075] like Figure 6 As shown, VsyncN-1, VsyncN, VsyncN+1, and VsyncN+2 represent the vertical synchronization signal of the N-1th frame, the vertical synchronization signal of the Nth frame, the vertical synchronization signal of the N-1th frame, the vertical synchronization signal of the Nth frame, the vertical synchronization signal of the N+1th frame, and the vertical synchronization signal of the N+2th frame, respectively. The moment of receiving the vertical synchronization signal of the N-1th frame is defined as the start moment of the N-1th frame, and the moment of receiving the vertical synchronization signal of the Nth frame is defined as the end moment of the N-1th frame or the start moment of the Nth frame, and so on. The middle moment between the start moment of the N-1th frame and the end moment of the N-1th frame is defined as the middle moment of the N-1th frame, and the middle moment between the start moment of the Nth frame and the end moment of the Nth frame is defined as the middle moment of the N-1th frame, and the middle moment between the start moment of the Nth frame and the end moment of the Nth frame is defined as the middle moment of the Nth frame, and so on.

[0076] At the start time of the N-1th frame (here as the start time of the current frame), the image to be rendered of the N-1th frame (for example, obtained based on the predicted user posture data at the time of point b) is rendered, and at the middle time of the N-1th frame, the rendered image of the N-1th frame (that is, the image to be rendered after rendering in the current frame) is time-warped to obtain the output image data of the N-1th frame. Then, the current frame is updated to the Nth frame in the figure, and at the start time of the Nth frame (here as the start time of the updated current frame), the image to be rendered of the Nth frame (for example, obtained based on the predicted updated user posture data at the time of point b) is rendered, and at the middle time of the Nth frame, the rendered image of the Nth frame (that is, the image to be rendered after rendering in the current frame) is time-warped to obtain the output image data of the Nth frame, and the current frame is updated to the N+1th frame in the figure, and the process is repeated.

[0077] Optionally, as described above, after the output image data is obtained, it can also be sent to the display module (screen scanning) and the backlight data can be calculated and sent to the backlight module, so that the backlight data display is synchronized with the output image data display of the display module. For example, in the first mode, and / or in the second mode and the third mode described below, the corresponding output image data is sent to the display module after each time warping process is completed, so that the display module starts to display the corresponding output image data after 0.5 / FPS from the start of the time warping process of the corresponding image, for example, starts to scan the screen (the scanning duration is the duration of one frame), and during this scanning period, the backlight data is displayed synchronously. Figure 6 In the case shown, the absolute delay is defined as the delay from the moment when the rendering process of the image to be rendered of the N-1th frame begins to the moment when the corresponding output image is displayed, which is 2 frames. In this example, there is no left eye image and right eye image, so a, b, and c can be regarded as the same point.

[0078] Figure 7 More details of step 430 are shown when the image to be rendered is rendered as a whole and the current processing mode is the second mode.

[0079] like Figure 7 As shown, in step S710, at a first time point in the current frame, the image to be rendered of the current frame is rendered. For example, for the convenience of timing, the first time point may be the start time or the middle time of the current frame.

[0080] In step S720, at a second time point that is a second time period from the first time point, rendering processing is started on the image to be rendered of the next frame of the current frame, and time warping processing is performed on the rendered image of the current frame, wherein the duration of the second time period is greater than or equal to the statistical rendering time.

[0081] For example, the duration of the second time period may be equal to the duration corresponding to the second threshold. Similarly, for the convenience of timing, the first time point may be the start time or the middle time of the current frame, and the first threshold may be 0.5 / FPS, and the second threshold may be 1 / FPS, so that the duration of the second time period may be 1 / FPS.

[0082] For example, when the first time point is the start time of the current frame, the second time point may be the start time of the next frame of the current frame; and when the first time point is the middle time of the current frame, the second time point may be the middle time of the next frame of the current frame.

[0083] Of course, these timings are only examples, and it is only necessary to perform the corresponding time warping process as soon as possible after the rendering process is completed. Figure 8 Shows Figure 7 A schematic timing diagram of asynchronously rendering and time warping the image to be rendered of the current frame. Figure 8 In the example, the first time point is the start time of the current frame, and the second time point is the start time of the next frame of the current frame.

[0084] like Figure 8 As shown, at the start time of the N-1th frame (here as the start time of the current frame), the image to be rendered of the N-1th frame starts to be rendered, and at the start time of the Nth frame, the image to be rendered of the Nth frame starts to be rendered, and the rendered image of the N-1th frame (that is, the image to be rendered that has been rendered in the current frame) is time-warped to obtain the output image data of the N-1th frame. Then, the current frame is updated to the Nth frame, and the above process is repeated. For example, at the start time of the N+1th frame, the image to be rendered of the N+1th frame starts to be rendered, and the rendered image of the Nth frame is time-warped to obtain the output image data of the Nth frame.

[0085] In addition, Figure 8In the case shown, after the output image data of the N-1th frame is obtained, the corresponding output image data is sent to the display module, so that the display module starts to display the corresponding output image data after 0.5 / FPS (i.e., the middle moment of the Nth frame) from the start moment of the time warping process of the corresponding left eye image, for example, starts to scan the screen (the scanning duration is the duration of one frame), during which the backlight data is synchronously displayed, and the absolute delay is 2.5 frames. It should be understood that the display of the image on the screen can be performed based on the vertical synchronization signal (not shown) occurring at each middle moment shown in the figure, and the left eye scanning and right eye scanning operations described later are also performed similarly based on their respective vertical synchronization signals.

[0086] Fig. 9 More details of step 430 are shown when the image to be rendered is rendered as a whole and the current processing mode is the third mode.

[0087] like Fig. 9 As shown, in step S910, at a first time point in the current frame, the image to be rendered of the current frame is rendered. For example, for the convenience of timing, the first time point may be the start time or the middle time of the current frame.

[0088] In step S920, at a second time point that is one frame time (1 / FPS) from the first time point, the rendering process of the image to be rendered of the next frame of the current frame begins. For example, when the first time point is the start time of the current frame, the second time point may be the start time of the next frame of the current frame; and when the first time point is the middle time of the current frame, the second time point may be the middle time of the next frame of the current frame.

[0089] In step S930, at a third time point after the rendering process of the to-be-rendered image of the current frame is completed, the time warping process is started on the rendered image of the current frame. For example, the approximate rendering process completion time can be determined based on the statistical rendering time consumption.

[0090] For example, since the rendering time of the image to be rendered of the current frame exceeds the duration of one frame, it is necessary to perform time warping on the rendered image of the current frame after the rendering of the image to be rendered is completed. Optionally, for the convenience of timing, the third time point may be a time after and closest to the time when the rendering process of the current frame is completed among P equidistant time points in the frame where the rendering process of the current frame is completed.

[0091] Fig.10 Shows Fig. 9 A schematic timing diagram of asynchronously rendering and time warping the image to be rendered of the current frame. Fig.10In the example, the rendering time is 1.25 frames, the first time point is the start time of the current frame, and P is 4.

[0092] like Fig.10 As shown, at the start time of the N-1th frame (here as the current frame), the image to be rendered of the N-1th frame starts to be rendered, and at the start time of the Nth frame (here as the next frame of the current frame), the image to be rendered of the Nth frame starts to be rendered. At 0.25 frames of the Nth frame, the rendering process of the N-1th frame has been completed, so at the middle time of the Nth frame, the rendered image of the N-1th frame is time-warped to obtain the output image data of the N-1th frame. Then, the current frame is updated to the Nth frame, and the above process is repeated. For example, at the start time of the N+1th frame, the image to be rendered of the N+1th frame starts to be rendered, and at the middle time of the N+1th frame, the rendered image of the Nth frame is time-warped to obtain the output image data of the Nth frame.

[0093] In addition, Fig.10 In the case shown, after the output image data of the N-1th frame is obtained, the corresponding output image data is sent to the display module, so that the display module starts to display the corresponding output image data after 0.5 / FPS from the start time of the time warping processing of the corresponding image (that is, the start time of the N+1th frame), for example, starts to scan the screen (the scanning time is the length of one frame), during which the backlight data is displayed synchronously, and the absolute delay is 3 frames.

[0094] In this case, it may be necessary to perform rendering processing on images to be rendered of two adjacent frames in the rendering thread at the same time. In this case, two or more buffers may be set to implement parallel implementation of multiple rendering processes.

[0095] The above is a description of the implementation method for rendering the image to be rendered as a whole. It can be seen that while ensuring that the rendering process and time warping process can be completed for each frame of the image to be rendered, and the output image data can be displayed (to avoid jitter caused by dropped frames), the absolute delay is different in different modes, so the most suitable mode can be selected based on the statistical rendering time. This implementation method can be applicable to the occasion of single-screen image output and the occasion of first rendering the whole and then obtaining the left eye output image data and the right eye output image data respectively through affine transformation of the rendering output result.

[0096] In some other implementations, the left eye image and the right eye image included in the image to be rendered may be rendered separately, where the left eye image and the right eye image may include the same content as the image to be rendered obtained based on the predicted user posture data at point b, that is, the left eye image and the right eye image are the same. At this time, the statistical rendering time in step S410 may further include the following steps: based on the first time of each rendering process for the left eye image and the second time of each rendering process for the right eye image in the predetermined time period before the current frame, determine the first statistical rendering time for the left eye image and the second statistical rendering time for the right eye image, wherein the first statistical rendering time is the same as the second statistical rendering time, and the statistical rendering time includes the first statistical rendering time and the second statistical rendering time. That is to say, the left eye image and the right eye image are processed separately (i.e., rendering processing and time warping processing). Optionally, in the rendering thread, the rendering processing of the left eye image and the right eye image for each frame can be parallel (e.g., based on multiple buffers) or serial (e.g., based on a single buffer), and in the asynchronous time warp thread, the time warp processing of the rendered left eye image and the rendered right eye image for each frame can be serial.

[0097] At this time, in the mode selection process in step S420, it is necessary to compare the first statistical rendering time or the second statistical rendering time with 1 / 2 of the first threshold (for example, 0.25 / FPS) and 1 / 2 of the second threshold (for example, 0.5 / FPS) to determine the selected image processing mode.

[0098] Fig.11 More details of step 430 are shown when the image to be rendered is divided into a left-eye image and a right-eye image and the current processing mode is the first mode (low-time consumption mode).

[0099] like Fig.11 As shown, in step S1110, at the first time point of the current frame, the left eye image of the current frame starts to be rendered. For example, the first time point may be the beginning moment or the middle moment of the current frame.

[0100] In step S1120, at a second time point that is a first time period away from the first time point, the right eye image of the current frame is started to be rendered, and time warping is started on the rendered left eye image of the current frame that has been rendered to obtain left eye output image data of the current frame.

[0101] In step S1130, at a third time point after the first time period from the second time point, time warping is started on the rendered right eye image of the current frame that has been rendered to obtain right eye output image data of the current frame, wherein the duration of the first time period is greater than or equal to the first statistical rendering time and the second statistical rendering time.

[0102] For example, the duration of the first time period may be equal to the duration corresponding to 1 / 2 of the first threshold. Similarly, for the convenience of timing, the first time point may be the start time or the middle time of the current frame, and the first threshold may be 0.5 / FPS, and the second threshold may be 1 / FPS, so that the first time period may be 0.25 / FPS.

[0103] For example, when the first time point is the start moment of the current frame, the second time point may be the 0.25 / FPS moment of the current frame, and the third time point may be the 0.5 / FPS moment of the current frame; and when the first time point is the middle moment of the current frame, the second time point may be the 0.75 / FPS moment of the current frame, and the third time point may be the start moment of the next frame of the current frame.

[0104] Optionally, as described above, after obtaining the left eye output image data and the right eye output image data of the current frame respectively, they can also be sent to the display module (screen scanning) and the backlight data can be calculated and sent to the backlight module respectively, so that the backlight data refresh is synchronized with the output image data display of the display module. For example, in the first mode, and / or in the second mode and the third mode described below, after each time warping of the left eye image is completed, the corresponding left eye output image data is sent to the display module, so that the display module starts to display the corresponding left eye output image data after 0.25 / FPS from the start time of the time warping of the corresponding left eye image; and after each time warping of the right eye image is completed, the corresponding right eye output image data is sent to the display module, so that the display module starts to display the corresponding right eye output image data after 0.25 / FPS from the start time of the time warping of the corresponding left eye image. In the first mode, it is feasible to start displaying the corresponding output image data after 0.5 / FPS for both the left and right eyes from the start of the time warp processing, or to start displaying the corresponding output image data after 0.5 / FPS for one of the left eye or the right eye and 0.25 / FPS for the other, etc., which can also shorten various delays.

[0105] Fig. 12A Shows Fig.11 A schematic timing diagram of asynchronously performing rendering processing and time warping processing on the to-be-rendered image of the current frame. Fig. 12B Shows Fig. 12A The operation flow chart corresponding to the schematic timing diagram in FIG. Fig. 12A In the example, the first time point is taken as the middle moment of the current frame.

[0106] like Fig. 12A As shown, at the middle moment of the N-1th frame (here as the starting moment of the current frame), the left eye image of the image to be rendered of the N-1th frame starts to be rendered, and at the 0.75 / FPS moment of the N-1th frame, the time warping processing of the rendered left eye image of the N-1th frame starts (the output image data of the left eye image, i.e., the left eye output image data can be obtained), and the right eye image of the image to be rendered of the N-1th frame starts to be rendered, and then at the starting moment of the Nth frame, the time warping processing of the rendered right eye image of the Nth frame starts (the output image data of the right eye image, i.e., the right eye output image data can be obtained), and then the current frame is updated to the Nth frame, and the above process is repeated, for example, at the middle moment of the Nth frame, the left eye image of the Nth frame starts to be rendered, and so on.

[0107] Here, when the time warping process is performed on the rendered left-eye image and the rendered right-eye image, the user posture data is also similarly involved.

[0108] For example, when performing time warping on the rendered left eye image of the current frame, it is also possible to predict the second user posture data of the expected display completion time of the left eye image of the current frame (for example, time point a shown in the figure) and the third user posture data of the expected display completion time of the right eye image of the current frame (for example, time point c shown in the figure) in a manner similar to the prediction of the first user posture data, and perform time warping on the rendered left eye image of the current frame that has been rendered based on the second user posture data; and perform time warping on the rendered right eye image of the current frame that has been rendered based on the third user posture data.

[0109] In addition, Fig. 12A In the case shown, after the left eye output image data / right eye output image data of the N-1th frame are obtained respectively, the corresponding left eye output image data / right eye output image data are sent to the display module respectively, so that the display module starts to display the corresponding left eye output image data / right eye output image data after 0.25 / FPS (or 0.5 / FPS) from the start of the time warping process of the corresponding rendered left eye image / rendered right eye image. For example, the left eye screen scan / right eye screen scan is started respectively (the screen scan duration is the duration of one frame), during which the backlight data is displayed synchronously, and at this time, the absolute delay from the start of rendering the left eye image of the N-1th frame to the moment when the corresponding left eye and right eye output image data are all displayed (point c) is 1.75 frames (or 2 frames), and as Figure 3 T1 = 1.625 frames (or 1.75 frames), T2 = 1.25 frames. These delays are relative to Figure 3 The reduction of the scheme in makes the posture data of each user obtained based on time prediction more accurate.

[0110] In addition, if Fig. 12B As shown, in response to obtaining the vertical synchronization signal of the N-1th frame, it is determined in process 1 whether there is a rendered left eye image of the N-2th frame (i.e., whether the N-1th frame is the 1st frame). If not, enter process 2, and if yes, enter process 3.

[0111] In process 2 (for the case where the N-1th frame is the first frame), determine whether the middle moment of the N-1th frame (expressed as VsyncN-1*0.5) has been reached. If so, proceed to process 4, otherwise wait. In process 4, start rendering the left eye image of the N-1th frame at the middle moment of the N-1th frame; in process 5, determine whether the 0.75 / FPS moment of the N-1th frame (expressed as Vsync(N-1)*0.75) has been reached. If so, proceed to process 6, otherwise wait. In process 6, start rendering the right eye image of the N-1th frame at the 0.75 / FPS moment of the N-1th frame, and start time-warping processing on the rendered left eye image of the N-1th frame at the same time, and calculate the left eye backlight data based on the obtained left eye output image data and send it to the backlight module. In process 7, determine whether the vertical synchronization signal of the Nth frame (expressed as Vsync(N)) is obtained. If so, enter process 8, otherwise wait. In process 8, at the moment (starting moment) when the vertical synchronization signal of the Nth frame is obtained, the time warping process of the rendered right eye image of the N-1th frame is started, and at the same time, the left eye output image data of the N-1th frame is sent to the display module for left eye screen scanning (left eye backlight data and left eye output image data are displayed synchronously), and the right eye backlight data is calculated and sent to the backlight module. In process 9, it is determined whether the middle moment of the Nth frame is reached (expressed as VsyncN*0.5). If so, proceed to process 10, otherwise wait. In process 10, at the middle moment of the Nth frame, the right eye output image data of the N-1th frame is sent to the display module for right eye screen scanning (right eye backlight data and right eye output image data are displayed synchronously), and at the same time, the left eye image of the Nth frame is rendered, and then N is replaced by N-1, and processes 5-10 are repeated.

[0112] In process 3 (for the case where the N-1th frame is not the first frame), determine whether the middle moment of the N-1th frame (expressed as VsyncN-1*0.5) has been reached. If so, proceed to process 11, otherwise wait. In process 11, start rendering the left eye image of the N-1th frame at the middle moment of the N-1th frame, and start sending the right eye output image data of the N-2th frame to the display module for right eye screen scanning; in process 12, determine whether the 0.75 / FPS moment of the N-1th frame (expressed as VsyncN-1*0.75) has been reached. If so, proceed to process 13, otherwise wait. In process 13, start rendering the right eye image of the N-1th frame at the 0.75 / FPS moment of the N-1th frame, and start time-warping processing on the rendered left eye image of the N-1th frame at the same time, and calculate the left eye backlight data based on the obtained left eye output image data and send it to the backlight module. In process 14, it is determined whether the vertical synchronization signal (expressed as Vsync(N)) of the Nth frame is obtained. If so, enter process 15, otherwise wait. In process 15, at the moment (starting moment) when the vertical synchronization signal of the Nth frame is obtained, the time warping process of the rendered right eye image of the N-1th frame is started, and at the same time, the left eye output image data of the N-1th frame is sent to the display module for left eye scanning, and the right eye backlight data is calculated and sent to the backlight module. Then replace N with N-1, and repeat process 3 and process 11-15.

[0113] Fig.13 More details of step 430 are shown when the image to be rendered is divided into a left-eye image and a right-eye image and the selected image processing mode is the second mode (intermediate mode).

[0114] like Fig.13 As shown, in step S1310, at the first time point of the current frame, the left eye image of the current frame is rendered. For example, the first time point may be the beginning or middle of the current frame.

[0115] In step S1320, at a second time point after a second time period from the first time point, the right eye image of the current frame starts to be rendered, and the rendered left eye image of the current frame is time-warped.

[0116] In step S1330, at a third time point after a second time period from the second time point, start time-warping the rendered right eye image of the current frame, and start rendering the left eye image of the next frame of the current frame, wherein the duration of the second time period is greater than or equal to the first statistical rendering time and the second statistical rendering time.

[0117] For example, the duration of the second time period may be equal to the duration corresponding to 1 / 2 of the second threshold value. Similarly, for the convenience of timing, the first time point may be the start time or the middle time of the current frame, and the first threshold value may be 0.5 / FPS, and the second threshold value may be 1 / FPS, so that the second time period may be 0.5 / FPS.

[0118] For example, when the first time point is the start moment of the current frame, the second time point may be the middle moment of the current frame, and the third time point may be the start moment of the next frame of the current frame; and when the first time point is the middle moment of the current frame, the second time point may be the start moment of the next frame of the current frame, and the third time point may be the middle moment of the next frame of the current frame.

[0119] Fig.14 Shows Fig.13 A schematic timing diagram of asynchronously rendering and time warping the image to be rendered of the current frame. Fig.14 In the example, the first time point is taken as the middle moment of the current frame.

[0120] like Fig.14 As shown, at the middle moment of the N-1th frame (current frame), the rendering process of the left eye image of the image to be rendered of the N-1th frame begins, at the beginning moment of the Nth frame, the rendering process of the right eye image of the image to be rendered of the N-1th frame begins, and the time warping process of the rendered left eye image of the N-1th frame begins; then at the middle moment of the Nth frame, the rendering process of the left eye image of the image to be rendered of the Nth frame begins, and the time warping process of the rendered right eye image of the Nth frame begins. At the beginning moment of the Nth frame, the current frame is updated to the Nth frame, and the above process is repeated, for example, at the middle moment of the Nth frame, the rendering process of the left eye image of the image to be rendered of the Nth frame begins.

[0121] In addition, Fig.14 In the case shown, after obtaining the left eye output image data / right eye output image data of the N-1th frame respectively, the corresponding left eye output image data / right eye output image data are sent to the display module respectively, so that the display module starts to display the corresponding left eye output image data / right eye output image data after 0.25 / FPS (or 0.5 / FPS) from the start of the time warping process of the corresponding rendered left eye image / rendered right eye image. For example, the left eye screen scan / right eye screen scan is started respectively (the screen scan duration is the duration of one frame), during which the backlight data is displayed synchronously, and the absolute delay is 2.25 frames at this time, and as Figure 3 T1 = 1.75 frames, T2 = 1.25 frames. These delays are relative to Figure 3The reduction of the scheme in makes the posture data of each user obtained based on time prediction more accurate.

[0122] Used for Fig.14 Operation flow chart and Fig. 12B The contents of the operation flow chart shown are similar, so they will not be described again here.

[0123] In some cases, such as when the scene of a VR application is particularly complex, the rendering time may be longer than the duration of one frame, that is, the rendering time of the left eye image and the rendering time of the right eye image are respectively greater than 0.5 / FPS, which may cause a delay of 4 frames. Considering that the rendering time may be particularly long, for example, even more than two frames, it may cause a greater delay.

[0124] Specifically, Fig.15 As shown, based on the existing timing, the image to be rendered starts to be rendered at the middle moment of the N-1th frame. If the rendered image (rendered left eye image and rendered right eye image) of the N-1th frame cannot be obtained at the middle moment of the Nth frame, the rendered left eye image of the N-1th frame will be time-warped at the middle moment of the N+1th frame, and the rendered right eye image of the N-2th frame will be time-warped at the beginning moment of the N+2th frame, and the rendered left eye image after time warping will be displayed (left eye screen scanning), and the rendered right eye image of the N-2th frame after time warping will be displayed at the middle moment of the N+2th frame, so that the display is completed at the middle moment of the N+3th frame. In this way, for the image to be rendered of the N-1th frame, there is an absolute delay of 4 frames from acquisition to final display on the screen, and T1=3.75 frames and T2=1.5 frames. Alternatively, even if the start time of the first ATW process in the figure is shifted to the left by 0.5 frames, there is still an absolute delay of 3.5 frames. In addition, in the case where the rendering takes longer, this timing may also cause frame drops, which will cause a more serious jitter problem. Therefore, in this application, a third mode is proposed for the case where the rendering takes longer.

[0125] Fig.16 More details of step 430 are shown when the image to be rendered is divided into a left-eye image and a right-eye image and the current processing mode is the third mode (high time consumption mode).

[0126] like Fig.16 As shown, in step S1610, at the first time point of the current frame, the left eye image and the right eye image of the current frame are rendered in parallel.

[0127] In step S1620, at a second time point after the rendering process of the left eye image and the right eye image of the current frame is completed, the time warping process of the rendered left eye image of the current frame is started. For example, the approximate rendering process completion time can be determined based on the first statistical rendering time for the left eye image and the second statistical rendering time for the right eye image. Optionally, for the convenience of timing, the second time point can correspond to the moment after and closest to the rendering process completion time of the current frame among P equidistant moments in the frame where the rendering process of the current frame is completed.

[0128] In step S1630, at a third time point one frame after the first time point, start time-warping the rendered right-eye image of the current frame, and start rendering the left-eye image and the right-eye image of the next frame of the current frame in parallel.

[0129] Similarly, the first threshold can be 0.5 / FPS, and the second threshold can be 1 / FPS. In this way, when the first and second statistical rendering times are both less than 0.75 / FPS, when the first time point is the start time of the current frame, the second time point can be the 0.75 / FPS time of the current frame, and the third time point can be the start time of the next frame of the current frame, when the first time point is the middle time of the current frame, the second time point can be the 0.25 / FPS time of the next frame of the current frame, and the third time point can be the middle time of the next frame of the current frame. Fig.17 Shows Fig.16 A schematic timing diagram of asynchronously rendering and time warping the image to be rendered of the current frame. Fig.17 In the example, the rendering time of the left eye image and the right eye image is 0.7 frames respectively, the first time point is the middle moment of the current frame, P=4, the second time point is 0.25 / FPS moment of the next frame of the current frame, and the third time point is the middle moment of the second frame.

[0130] like Fig.17 As shown, at the middle moment of the N-1th frame, the left eye image and the right eye image of the image to be rendered of the N-1th frame are rendered in parallel; at the 0.25 / FPS moment of the Nth frame, the time warping process of the rendered left eye image of the N-1th frame is started, and at the middle moment of the Nth frame, the time warping process of the rendered right eye image of the N-1th frame is started, and the rendering process of the left eye image and the right eye image of the Nth frame is started at the same time. At the beginning moment of the Nth frame, the current frame is updated to the Nth frame, and the above process is repeated.

[0131] In addition, Fig.17In the case shown, after obtaining the left eye output image data / right eye output image data of the N-1th frame respectively, the corresponding left eye output image data / right eye output image data are sent to the display module respectively, so that the display module starts to display the corresponding left eye output image data / right eye output image data after 0.25 / FPS (or 0.5 / FPS) from the start of the time warping process of the corresponding rendered left eye image / rendered right eye image. For example, the left eye screen scan / right eye screen scan is started respectively (the screen scan duration is the duration of one frame), during which the backlight data is displayed synchronously, and the absolute delay is 2.25 frames at this time, and as Figure 3 T1 = 1.75 frames, T2 = 1.25 frames. These delays are relative to Fig.15 The reduction of the scheme in makes the posture data of each user obtained based on time prediction more accurate.

[0132] Used for Fig.16 Operation flow chart and Fig. 12B The contents of the operation flow chart shown are similar, so they will not be described again here.

[0133] The above is a description of an implementation method for separately rendering and time-warping the left-eye image and the right-eye image of the rendered image. This implementation method can be applicable to occasions of dual-screen image output (for example, virtual reality glasses), and the left-eye output image data and the right-eye output image data after image processing can be output to the left and right display areas (for example, two lenses) respectively.

[0134] Fig.18 is a block diagram showing an image processing apparatus according to an embodiment of the present disclosure.

[0135] refer to Fig.18 According to some embodiments of the present disclosure, the image processing device 1800 includes a statistical module 1810, a mode selection module 1820 and a processing module 1830.

[0136] The statistics module 1810 is used to determine the statistical rendering time consumption based on the time consumption of each rendering process within a predetermined time period before the current frame.

[0137] The mode selection module 1820 is used to select a selected image processing mode for the image to be rendered of the current frame from a first mode, a second mode and a third mode based on the statistical rendering time consumption, wherein the first mode, the second mode and the third mode are respectively associated with different timings between the rendering processing in the rendering thread and the time warp processing in the asynchronous time warp thread.

[0138] The processing module 1830 is used to perform rendering processing and time warping processing on the image to be rendered of the current frame according to the selected image processing mode to obtain output image data of the current frame.

[0139] According to different functional division methods, the image processing device 1800 can divide modules in different ways. For example, each module can be further divided into sub-modules, or can also include other modules.

[0140] More details on the operations implemented by the above modules can be found in Figure 4-17 The content described is therefore not repeated here.

[0141] Each module in the above-mentioned image processing device 1800 can be implemented by software, hardware and firmware or any combination thereof. The method steps of other embodiments of the present disclosure can also be implemented by software, hardware and firmware or any combination thereof.

[0142] In addition, the image processing device 1800 may be included in a virtual reality display device, for example as at least a part of its processor, and, for example, combined with a display module and a backlight module, etc., to provide users with various scene images, thereby providing an immersive virtual world environment.

[0143] In addition, according to some embodiments of the present disclosure, an image processing device is provided, which includes a processor and a memory, and a computer program instruction is stored in the memory. When the computer executable instruction is executed by a computer, the image processing method according to the embodiment of the present disclosure can be implemented.

[0144] Therefore, another aspect of the present application further provides a display device, comprising a display and the image processing device as described above.

[0145] The display device provided in the present disclosure may include but is not limited to a smart phone, a tablet computer, a media player, etc. It should be noted that for the sake of clarity, the entire structure of the device is not given. To achieve the necessary functions of the device, those skilled in the art may set other structures not shown according to specific application scenarios, and the present invention does not limit this.

[0146] The computer-readable storage medium may be any combination of one or more computer-readable storage media, for example, one computer-readable storage medium contains a computer-readable program code for randomly generating a sequence of action instructions, and another computer-readable storage medium contains a computer-readable program code for performing facial activity recognition.

[0147] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0148] The processor in the embodiments of the present disclosure may be an integrated circuit chip having the ability to process signals. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc., and may be an X86 architecture or an ARM architecture.

[0149] The computer-readable storage medium in the disclosed embodiments may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the system and method described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0150] The image processing method and device of the disclosed embodiment can dynamically adjust the system timing according to the real-time duration of the rendering process, and automatically switch to the optimal timing of the rendering process and the time warping process after real-time statistics of the time spent on the rendering process, which can not only improve the use efficiency of the processor, but also improve the delay problem and jitter problem caused by the excessive time spent on the rendering process. .

[0151] The exemplary embodiments of the present invention described in detail above are merely illustrative, rather than restrictive. It should be appreciated by those skilled in the art that various modifications, combinations or sub-combinations may be made to these embodiments without departing from the principles and spirit of the present invention, and such modifications should fall within the scope of the present invention.

Claims

1. An image processing method, comprising: Determine the statistical rendering time based on the time consumed by each rendering process within a predetermined time period before the current frame; Based on the statistical rendering time, select a selected image processing mode for the to-be-rendered image of the current frame from a first mode, a second mode, and a third mode, wherein the first mode, the second mode, and the third mode are respectively associated with different timings between a rendering process in a rendering thread and a time warp process in an asynchronous time warp thread; as well as According to the selected image processing mode, the image to be rendered of the current frame is asynchronously rendered and time-warped to obtain output image data of the current frame.

2. The method according to claim 1, wherein: Based on the statistical rendering time consumption, selecting a selected image processing mode for the image to be rendered of the current frame from the first mode, the second mode, and the third mode, including: When the statistical rendering time is less than or equal to a first threshold, using the first mode as the selected image processing mode; and / or In a case where the statistical rendering time is greater than a second threshold, using the third mode as the selected image processing mode; and / or When the statistical rendering time is greater than the first threshold and less than or equal to the second threshold, the second mode is used as the selected image processing mode.

3. The method according to claim 2, wherein: When the selected image processing mode is the first mode, according to the selected image processing mode, asynchronously performing rendering processing and time warping processing on the image to be rendered of the current frame, including: At a first time point in the current frame, start rendering the image to be rendered of the current frame; and At a second time point after a first time period from the first time point, time warping processing is started on the rendered image of the current frame, wherein the duration of the first time period is greater than or equal to the counted rendering time.

4. The method according to claim 2, in, When the selected image processing mode is the second mode, according to the selected image processing mode, asynchronously performing rendering processing and time warping processing on the image to be rendered of the current frame, including: At a first time point in the current frame, start rendering the image to be rendered of the current frame; and At a second time point after a second time period from the first time point, rendering processing is started on the image to be rendered of the next frame of the current frame, and time warping is performed on the rendered image of the current frame, wherein the duration of the second time period is greater than or equal to the statistical rendering time.

5. The method according to claim 2, wherein: When the selected image processing mode is the third mode, according to the selected image processing mode, asynchronously performing rendering processing and time warping processing on the image to be rendered of the current frame, including: At a first time point in the current frame, start rendering the image to be rendered of the current frame; At a second time point after one frame of time from the first time point, starting to render the image to be rendered of the next frame of the current frame; and At a third time point after the rendering process of the to-be-rendered image of the current frame is completed, the time warping process of the rendered image of the current frame begins.

6. The method according to claim 2, wherein: The image to be rendered for each frame includes a left-eye image and a right-eye image. The statistical rendering time consumption is determined based on the time consumption of each rendering process in a predetermined time period before the current frame, including: Determine a first statistical rendering time for the left-eye image and a second statistical rendering time for the right-eye image based on a first time consumption for each rendering process for the left-eye image and a second time consumption for each rendering process for the right-eye image in the predetermined time period before the current frame, respectively; The first statistical rendering time is the same as the second statistical rendering time, and the statistical rendering time includes the first statistical rendering time and the second statistical rendering time.

7. The method according to claim 6, wherein: When the first and second statistical rendering times are less than or equal to 1 / 2 of the first threshold, so that the selected image processing mode is the first mode, rendering processing and time warping processing are performed on the image to be rendered of the current frame according to the selected image processing mode, including: At the first time point of the current frame, start rendering the left eye image of the current frame; At a second time point after a first time period from the first time point, starting to render the right eye image of the current frame, and starting to time warp the rendered left eye image of the current frame; At a third time point from the second time point through the first time period, start time warping the rendered right eye image of the current frame, wherein the duration of the first time period is greater than or equal to the first statistical rendering time and the second statistical rendering time.

8. The method according to claim 6, wherein: When the first and second statistical rendering times are greater than 1 / 2 of the first threshold and less than or equal to 1 / 2 of the second threshold, so that the selected image processing mode is the second mode, according to the selected image processing mode, asynchronously performing rendering processing and time warping processing on the to-be-rendered image of the current frame, including: At a first time point of the current frame, start rendering the left eye image of the current frame; and At a second time point after a second time period from the first time point, starting to render the right eye image of the current frame, and performing a time warping process on the rendered left eye image of the current frame that has been rendered; and At a third time point that is after the second time point and the second time period, start to perform time warping on the rendered right eye image of the current frame that has been rendered, and perform rendering on the left eye image of the next frame of the current frame, wherein the duration of the second time period is greater than or equal to the first statistical rendering time and the second statistical rendering time.

9. The method according to claim 6, wherein: When the first and second statistical rendering times are greater than 1 / 2 of the second threshold, so that the selected image processing mode is the third mode, according to the selected image processing mode, asynchronously performing rendering processing and time warping processing on the to-be-rendered image of the current frame, including: At a first time point of the current frame, starting to render the left eye image and the right eye image of the current frame in parallel; At a second time point after the rendering process of the left-eye image and the right-eye image of the current frame is completed, starting to perform time warping process on the rendered left-eye image of the current frame; At a third time point one frame later than the first time point, time warping processing is started on the rendered right eye image of the current frame, and rendering processing is started on the left eye image and the right eye image of the next frame in parallel.

10. The method according to claim 6, further comprising: Sending the output image data of the current frame to a display module; Backlight data is generated based on the output image data of the current frame and sent to a backlight module, so that the backlight data of the backlight module is displayed synchronously with the output image data of the display module.

11. The method according to claim 10, wherein: Sending the output image data of the current frame to the display module includes: Sending corresponding left-eye output image data to the display module after each time warping process of the left-eye image is completed, so that the display module starts to display the corresponding left-eye output image data after 0.25 / FPS from the start time of the time warping process of the corresponding left-eye image; and After each time warping of the right eye image is completed, the corresponding right eye output image data is sent to the display module, so that the display module starts to display the corresponding right eye output image data 0.25 / FPS after the time warping of the corresponding left eye image starts.

12. The method according to any one of claims 3 to 5, further comprising: predicting first user gesture data after a first predetermined time has passed since the start of rendering processing of the to-be-rendered image of the current frame, and Determine the image to be rendered of the current frame based on the first user posture data, The end time of the first predetermined time is the expected display completion time of the to-be-rendered image of the current frame.

13. The method according to any one of claims 7 to 9, further comprising: predicting first user gesture data after a first predetermined time has passed since the start of rendering processing of the to-be-rendered image of the current frame, and Determine the image to be rendered of the current frame based on the first user posture data, The end time of the first predetermined time is any time between the expected display completion time of the left eye image of the current frame and the expected display completion time of the right eye image of the current frame.

14. The method according to claim 13, wherein: According to the selected image processing mode, rendering processing and time warping processing are performed on the image to be rendered of the current frame, and further comprising: Predicting the second user gesture data of the expected display completion time of the left eye image of the current frame, and predicting the third user gesture data of the expected display completion time of the right eye image of the current frame, Performing a time warping process on the rendered left-eye image of the current frame based on the second user gesture data; and A time warping process is performed on the rendered right-eye image of the current frame based on the third user gesture data.

15. An image processing device, comprising: A statistics module, used to determine the statistical rendering time based on the time consumed by each rendering process within a predetermined time period before the current frame; a mode selection module, configured to select, based on the statistical rendering time, a selected image processing mode for the to-be-rendered image of the current frame from a first mode, a second mode, and a third mode, wherein the first mode, the second mode, and the third mode are respectively associated with different timings between a rendering process in a rendering thread and a time warp process in an asynchronous time warp thread; as well as The processing module is used to perform rendering processing and time warping processing on the image to be rendered of the current frame according to the selected image processing mode for the image to be rendered of the current frame, so as to obtain output image data of the current frame.

16. An image processing device, comprising: A processor and a memory, wherein the memory stores computer executable instructions, and when the computer executable instructions are executed by the processor, the image processing method according to any one of claims 1 to 14 is performed.

17. A display device, comprising: monitor; as well as The image processing apparatus according to claim 16.

18. A non-volatile storage medium storing computer executable instructions, wherein: When the computer executable instructions are executed by a processor, the image processing method according to any one of claims 1 to 14 is performed.

Images