Virtual reality display apparatus, method of controlling same, and computer storage medium

Abstract

The invention discloses a virtual reality display device, a control method thereof and a computer storage medium, and belongs to the technical field of display. The virtual reality display device comprises a control assembly and a display unit. The control component can obtain rendering time consumption for rendering a frame of picture of a display unit, and execute a plurality of data processing tasks of the display unit in different modes through at least two threads based on the difference of the rendering time consumption, according to the embodiment of the invention, the sequences of executing the multiple data processing tasks of the display unit by the at least two threads are different, so that the display flexibility is improved, the problem of poor display flexibility of a virtual reality display device in related technologies can be solved, and the effect of improving the display flexibility of the virtual reality display device is achieved.

Description

Technical Field

[0001] The present application relates to the field of display technology, and in particular to a virtual reality display device and a control method thereof, and a computer storage medium. Background Art

[0002] A virtual reality display device is a device that can realize display functions.

[0003] A current virtual reality display device is a virtual reality display device, which may include a left-eye screen, a right-eye screen, and a processor. The processor can render image data of the two screens and control the display of the two screens based on the image data.

[0004] However, the above virtual reality display device has poor flexibility in display. Summary of the invention

[0005] The embodiments of the present application provide a virtual reality display device, a control method for the virtual reality display device, and a computer storage medium. The technical solution is as follows:

[0006] According to a first aspect of the present application, a virtual reality display device is provided, the virtual reality display device comprising a control component and a display unit, the control component being used to:

[0007] Obtaining the rendering time consumed for rendering a frame of the display unit;

[0008] In response to the rendering time being within a first duration range, executing multiple data processing tasks of the display unit in sequence based on a first mode through at least two threads, the multiple data processing tasks including a rendering task of image data of the screen and a correction task of the image data, and the first mode is a mode corresponding to the first duration range;

[0009] In response to the rendering time being within a second duration range, the data processing tasks of the display unit are executed sequentially based on a second mode through at least two threads, the second mode is a mode corresponding to the second duration range, the minimum value of the second duration range is greater than the maximum value of the first duration range, and the execution order of the multiple data processing tasks of the display unit in the second mode is different from the execution order of the multiple data processing tasks of the display unit in the first mode.

[0010] Optionally, the at least two threads include a first thread and a second thread, and the display unit includes a first screen and a second screen;

[0011] The first duration ranges from 0 to 0.5 / f, and the rendering time of one frame of the first screen and the rendering time of one frame of the second screen are both less than or equal to 0.25 / f, where f is the refresh rate of the display unit.

[0012] Optionally, the control component is used to:

[0013] At a time after the start time of the nth frame and at an interval of 0.5 / f, the image data of the nth frame of the first screen is rendered by the first thread, and at a time after the start time of the nth frame and at an interval of 0.75 / f, the image data of the nth frame of the second screen is rendered by the first thread, and at the start time of the n+1 frame, the image data of the nth frame of the second screen is corrected by the first thread to obtain corrected image data of the nth frame of the second screen;

[0014] After the starting moment of the nth frame and at an interval of 0.75 / f, the image data of the nth frame of the first screen is corrected by the second thread to obtain the corrected image data of the nth frame of the first screen; at the starting moment of the n+1th frame, the backlight adjustment processing is performed on the corrected image data of the nth frame of the first screen by the second thread to obtain the backlight data of the nth frame of the first screen; after the starting moment of the n+1th frame and at an interval of 0.25 / f, the backlight adjustment processing is performed on the corrected image data of the nth frame of the second screen by the second thread to obtain the backlight data of the nth frame of the second screen.

[0015] Optionally, the control component is used to:

[0016] At a time after the start time of the nth frame and at an interval of 0.5 / f, obtaining the predicted first user posture data of the first time, and rendering the image data of the nth frame of the first screen based on the first user posture data through the first thread, the first time being the time after the start time of the n+1th frame;

[0017] After the start time of the nth frame and at an interval of 0.75 / f, the predicted second user posture data of the first moment is obtained, and the image data of the nth frame of the first screen is corrected based on the second user posture data through the second thread to obtain the corrected image data of the nth frame of the first screen.

[0018] Optionally, the control component is used to:

[0019] At a time after the start time of the nth frame and at an interval of 0.75 / f, obtaining third user posture data of a predicted second time, and rendering image data of the nth frame of the second screen based on the third user posture data through the first thread, the second time being a time after the first time;

[0020] At the starting moment of the n+1 frame, the predicted fourth user posture data of the second moment is obtained, and the image data of the nth frame of the second screen is corrected based on the fourth user posture data by the first thread to obtain the corrected image data of the nth frame of the second screen.

[0021] Optionally, the at least two threads include a first thread and a second thread, and the display unit includes a first screen and a second screen;

[0022] The second duration ranges from 0.5 / f to 1 / f, and the rendering time of one frame of the first screen and the rendering time of one frame of the second screen are both less than or equal to 0.5 / f, where f is the refresh rate of the display unit.

[0023] Optionally, the control component is used to:

[0024] In response to the nth frame being an odd frame, at the start time of the nth frame, the image data of the nth frame of the second screen is rendered by the first thread, and the image data of the nth frame of the second screen is corrected to obtain corrected image data of the nth frame of the second screen, and backlight adjustment processing is performed on the corrected image data of the nth frame of the second screen to obtain backlight data of the nth frame of the second screen;

[0025] At the start time of the nth frame, the image data of the nth frame of the first screen acquired in the n-1th frame is corrected by the second thread to obtain the corrected image data of the nth frame of the first screen, and the backlight adjustment processing is performed on the corrected image data of the nth frame of the first screen to obtain the backlight data of the nth frame of the first screen, and then the image data of the n+1th frame of the first screen is rendered;

[0026] At the start time of the n+1th frame, the image data of the nth frame of the first screen is corrected by the first thread to obtain the corrected image data of the nth frame of the first screen, and the backlight adjustment processing is performed on the corrected image data of the nth frame of the first screen by the first thread to obtain the backlight data of the nth frame of the first screen, and then the image data of the n+2th frame of the first screen is rendered;

[0027] At the starting moment of the n+1th frame, the image data of the n+1th frame of the second screen is rendered by the second thread, and the image data of the n+1th frame of the second screen is corrected to obtain the corrected image data of the n+1th frame of the second screen, and then the corrected image data of the n+1th frame of the second screen is backlight adjusted to obtain the backlight data of the n+1th frame of the second screen.

[0028] Optionally, the control component is used to:

[0029] At the start time of the nth frame, fifth user posture data of the predicted third time is obtained, and image data of the nth frame of the first screen is corrected based on the fifth user posture data by the second thread to obtain corrected image data of the nth frame of the first screen, wherein the third time is a time after the start time of the nth frame;

[0030] At a time after the start time of the nth frame and at an interval of 0.5 / f, obtaining the predicted sixth user gesture data at a fourth time, and rendering the image data of the n+1th frame of the first screen based on the fifth user gesture data through the second thread, the fourth time being the time after the start time of the n+1th frame;

[0031] At the starting moment of the nth frame, the predicted seventh user posture data of the fifth moment is obtained, and the image data of the nth frame of the second screen is rendered based on the seventh user posture data by the first thread, and then the predicted eighth user posture data of the fifth moment is obtained, and the image data of the nth frame of the second screen is corrected based on the eighth user posture data by the first thread to obtain the corrected image data of the nth frame of the second screen, and the fifth moment is the moment after the starting moment of the nth frame.

[0032] Optionally, the control component is used to:

[0033] At the starting moment of the n+1th frame, the predicted ninth user gesture data of the fourth moment is obtained, and the image data of the n+1th frame of the first screen is corrected by the first thread based on the ninth user gesture data to obtain the corrected image data of the n+1th frame of the first screen, and after the starting moment of the n+1th frame and at a time interval of 0.5 / f, the predicted tenth user gesture data of the sixth moment is obtained, and then the image data of the n+1th frame of the first screen is rendered by the first thread based on the tenth user gesture data, the sixth moment being a moment after the starting moment of the n+2th frame and at a time interval of 0.5 / f;

[0034] At the starting moment of the n+1th frame, the eleventh user posture data of the predicted seventh moment is obtained, and the image data of the n+1th frame of the second screen is rendered by the second thread based on the eleventh user posture data, and then the twelfth user posture data of the predicted seventh moment is obtained, and the image data of the n+1th frame of the second screen is corrected by the second thread based on the twelfth user posture data to obtain the corrected image data of the n+1th frame of the second screen, and the seventh moment is the moment after the starting moment of the n+2th frame.

[0035] Optionally, the at least two threads include a first thread and a second thread, and the display unit includes a first screen and a second screen;

[0036] The control component is also used for:

[0037] In response to the rendering time being within a third duration range, the data processing tasks of the display unit are sequentially executed based on a third mode by the at least two threads, the third mode being a mode corresponding to the third duration range, and the execution order of the multiple data processing tasks of the display unit in the third mode is different from the execution order of the multiple data processing tasks of the display unit in the first mode and the second mode;

[0038] The third duration ranges from 1 frame to 2 frames, and the rendering time of one frame of the first screen and the rendering time of one frame of the second screen are both less than or equal to 1 frame.

[0039] Optionally, the control component is used to:

[0040] After the start time of the n-1th frame and at an interval of 0.5 / f, the image data of the nth frame of the first screen is rendered by the first thread, the image of the nth frame of the first screen is corrected by the first thread to obtain corrected image data of the nth frame of the first screen, and the backlight adjustment processing is performed on the corrected image data of the nth frame of the first screen by the first thread to obtain backlight data of the nth frame of the first screen;

[0041] At the starting moment of the nth frame, the image data of the nth frame of the second screen is rendered by the second thread, the image of the nth frame of the second screen is corrected by the second thread to obtain the corrected image data of the nth frame of the second screen, and then the corrected image data of the nth frame of the second screen is backlight adjusted by the second thread to obtain the backlight data of the nth frame of the second screen.

[0042] Optionally, the control component is used to:

[0043] At a time after the start time of the n-1th frame and at an interval of 0.5 / f, obtaining the thirteenth user gesture data of the predicted eighth time, and rendering the image data of the nth frame of the first screen based on the thirteenth user gesture data through the first thread, the eighth time being the time after the start time of the nth frame;

[0044] After acquiring the image data of the nth frame of the first screen, the predicted fourteenth user posture data of the eighth moment is acquired, and the image of the nth frame of the first screen is corrected based on the tenth user posture data through the first thread to obtain the corrected image data of the nth frame of the first screen.

[0045] Optionally, the control component is used to:

[0046] At the start time of the nth frame, obtaining the predicted fifteenth user gesture data at the ninth time, and rendering the image data of the nth frame of the second screen based on the fifteenth user gesture data by the first thread, the ninth time being a time after the eighth time;

[0047] After acquiring the image data of the nth frame of the second screen, the predicted sixteenth user posture data of the ninth moment is acquired, and the image data of the nth frame of the second screen is corrected based on the sixteenth user posture data to obtain the corrected image data of the nth frame of the second screen.

[0048] According to another aspect of an embodiment of the present application, a method for controlling a virtual reality display device is provided, the method comprising:

[0049] Obtaining the rendering time consumed for rendering a frame of the display unit;

[0050] In response to the rendering time being within a first duration range, executing multiple data processing tasks of the display unit in sequence based on a first mode through at least two threads, the multiple data processing tasks including a rendering task of image data of the screen and a correction task of the image data, and the first mode is a mode corresponding to the first duration range;

[0051] In response to the rendering time being within a second duration range, the data processing tasks of the display unit are executed sequentially based on a second mode through at least two threads, the second mode is a mode corresponding to the second duration range, the minimum value of the second duration range is greater than the maximum value of the first duration range, and the execution order of the multiple data processing tasks of the display unit in the second mode is different from the execution order of the multiple data processing tasks of the display unit in the first mode.

[0052] According to another aspect of an embodiment of the present application, a computer storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the computer storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the above-mentioned control method of the virtual reality display device.

[0053] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0054] Provided is a virtual reality display device including a control component and a display unit, wherein the control component can obtain the rendering time of a frame of the rendering display unit, and based on the difference in the rendering time, execute multiple data processing tasks of the display unit in different modes through at least two threads, wherein in different modes, the at least two threads execute the multiple data processing tasks of the display unit in different orders, thereby improving the flexibility of display, solving the problem of poor flexibility of the virtual reality display device during display in the related art, and achieving the effect of improving the flexibility of the virtual reality display device during display.

[0055] In addition, in the virtual reality display device provided in the embodiment of the present application, since the control component can select different modes based on the different rendering times to implement multiple data processing tasks of the display unit, this can facilitate the control component to arrange the various data processing tasks more reasonably to achieve the effect of reducing screen delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 It is a schematic diagram of a virtual reality device when displaying;

[0058] Figure 2 is a structural schematic diagram of a virtual reality display device provided in an embodiment of the present application;

[0059] Figure 3 yes Figure 2 A control method of a virtual reality display device of the virtual reality display device shown;

[0060] Figure 4 is a structural block diagram of another virtual reality device provided in an embodiment of the present application;

[0061] Figure 5is a flow chart of a control method of a virtual reality device provided in an embodiment of the present application;

[0062] Figure 6 is a flow chart of a first mode of a virtual reality device provided in an embodiment of the present application;

[0063] Figure 7 yes Figure 6 A method flow chart of the first mode shown;

[0064] Figure 8 is a flow chart of a second mode of the virtual reality display device provided in an embodiment of the present application;

[0065] Fig. 9 It is a schematic diagram of a flow chart corresponding to the second mode of the virtual reality display device provided in an embodiment of the present application;

[0066] Fig.10 is a flow chart of a third mode of the virtual reality display device provided in an embodiment of the present application;

[0067] Fig.11 It is a flow chart of a virtual reality display device in the related art;

[0068] Fig.12 It is a flow chart corresponding to the third mode of the virtual reality display device provided in the embodiment of the present application.

[0069] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0070] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0071] Some virtual reality systems may include head-mounted display devices, such as virtual reality helmets and virtual reality glasses, which can generate an immersive virtual world environment based on the user's actions, perceptions, and other information. However, when using a head-mounted display device that applies virtual reality, if the user moves too fast, it may cause a delay in scene rendering. For example, when the user turns his head quickly, although the user's head has turned past the predetermined position and arrived at the new position, the image corresponding to the new position has not been rendered, or the rendered image is still the image of the previous frame, so the image seen by the user is inconsistent with the image of the new position he expects, resulting in a delay in image data and the user's dizziness.

[0072] In order to reduce the delay in rendering images in the display scene, some virtual reality devices use time warp (TW) technology. Time warp technology is a technology for correcting image frames. This technology can solve the problem of scene rendering delay by distorting (or correcting) the rendered image data based on the changes in user actions after rendering. Since the time when the time warp processing is performed is closer to the display time, the new image data obtained after the time warp processing is closer to the image that the user wants to see. At the same time, since the time warp technology only processes two-dimensional images, it is similar to the affine transformation in image processing, so the time warp processing will not bring too much system workload overhead.

[0073] On this basis, the Asynchronous Timewarp (ATW) technology can further optimize the above-mentioned time warp technology. The Asynchronous Timewarp technology can arrange rendering and time warping in two different threads (one of the two threads can be a rendering thread, and the other thread can be an asynchronous time warp thread), so that the rendering step and the time warp step can be executed asynchronously, thereby reducing the overall running time of the rendering and time warp processing. For example, when the virtual reality device is difficult to maintain a sufficient frame rate, the asynchronous time warp thread can reprocess the previously rendered image data according to the current user posture to generate a frame picture that conforms to the current user posture, thereby reducing the jitter of the picture and reducing the delay.

[0074] In the virtual reality system of the related art, an asynchronous time warp thread (called an ATW thread) can be used to perform processing in the asynchronous time warp process, and this thread and the rendering thread (called an MRT thread) run synchronously. Figure 1 As shown, Figure 1This is a schematic diagram of a virtual reality device displaying, wherein at time s after the start time of the n-1th frame and at an interval of 0.5 / f, the MRT thread renders the image to be displayed, and the rendered image at this time may depend on the user posture data at a predicted time a after a period of time (such as 1.5 / f. f is the refresh rate), and the ATW thread corrects the image frames of the left and right eyes based on the posture prediction at time a after the image rendered by the MRT thread. When ATW obtains the image data transmitted by the MRT thread, it can interpolate and determine the posture data for ATW processing of the left eye image based on the predicted user posture at time b. At the moment when the Vsync signal is obtained (each time the Vsync signal is obtained, it can be considered as the start time of a frame), the posture data for ATW processing of the right eye image is determined based on the predicted user posture. Under the refresh display mechanism of the virtual reality system, the time T1 from the moment s1 when the user posture is obtained to start rendering the image to the moment c1 when the image is displayed on the screen is 2.5 / f.

[0075] Local dimming (LD) technology is a local backlight adjustment technology, through which the brightness of some areas of the screen can be independently controlled to achieve the effect of improving the contrast of the screen. In addition, local dimming technology can also be combined with some other technologies (such as foveated rendering technology in the field of virtual reality, etc.), which will not be described in detail in the embodiments of this application.

[0076] It should be noted that the virtual reality device involved in the embodiments of the present application does not refer to a device that only applies virtual reality (VR) technology. The device can also apply augmented reality (AR) technology and mixed reality (MR) technology.

[0077] Figure 2 2 is a structural diagram of a virtual reality display device provided in an embodiment of the present application. The virtual reality display device includes a control component 21 and a display unit 22. The control component 21 is used to:

[0078] Gets the rendering time of one frame of the rendering display unit.

[0079] In response to the rendering time being within a first duration range, multiple data processing tasks of the display unit are executed sequentially based on a first mode through at least two threads. The multiple data processing tasks include a rendering task of the image data of the display unit and a correction task of the image data. The first mode is a mode corresponding to the first duration range.

[0080] In response to the rendering time being within a second duration range, data processing tasks of the display unit are executed sequentially based on a second mode through at least two threads, the second mode is a mode corresponding to the second duration range, the minimum value of the second duration range is greater than the maximum value of the first duration range, and the execution order of multiple data processing tasks of the display unit in the second mode is different from the execution order of multiple data processing tasks of the display unit in the first mode.

[0081] In the virtual reality display device provided in the embodiment of the present application, the display unit 22 may include multiple independent screens or different parts of a screen. For example, a screen may include two parts ( Figure 2 This is the case), a part of which corresponds to the left eye and is used to display the corresponding image to the left eye. This part can be the first screen a1 in the display unit 22, and the other part corresponds to the right eye and is used to display the corresponding image to the right eye. This part can be the second screen a2 in the display unit 22.

[0082] In summary, an embodiment of the present application provides a virtual reality display device including a control component and a display unit, wherein the control component can obtain the rendering time of a frame of the display unit, and based on the difference in the rendering time, execute multiple data processing tasks of the display unit in different modes through at least two threads, wherein in different modes, the order in which at least two threads execute the multiple data processing tasks of the display unit is different, thereby improving the flexibility of display, and can solve the problem of poor flexibility of the virtual reality display device during display in the related art, thereby achieving the effect of improving the flexibility of the virtual reality display device during display.

[0083] In addition, in the virtual reality display device provided in the embodiment of the present application, since the control component can select different modes based on the different rendering times to implement multiple data processing tasks of the display unit, this can facilitate the control component to arrange the various data processing tasks more reasonably to achieve the effect of reducing screen delay.

[0084] Figure 3 yes Figure 2 A control method for a virtual reality display device of the virtual reality display device shown in the figure can be applied to Figure 2 The control component in the virtual reality display device shown in the figure, the method comprises the following steps:

[0085] Step 301: Obtain the rendering time of a frame of a rendering display unit.

[0086] Step 302: In response to the rendering time being within a first duration range, execute a plurality of data processing tasks of the display unit in sequence based on a first mode by at least two threads.

[0087] The multiple data processing tasks include a rendering task of the image data of the display unit and a correction task of the image data. For example, the multiple data processing tasks may include a rendering task of the image data for each screen and a correction task of the image data for each screen, etc. The first mode is a mode corresponding to the first duration range.

[0088] Step 303: In response to the rendering time being within the second time range, the data processing tasks of the display unit are sequentially executed based on the second mode by at least two threads.

[0089] The second mode is a mode corresponding to a second time range, the minimum value of the second time range is greater than the maximum value of the first time range, and the execution order of multiple data processing tasks of the display unit in the second mode is different from the execution order of multiple data processing tasks of the display unit in the first mode.

[0090] In summary, the embodiment of the present application provides a control method for a virtual reality display device, which can obtain the rendering time of a frame of a rendering display unit, and based on the difference in the rendering time, execute multiple data processing tasks of the display unit in different modes through at least two threads, wherein in different modes, at least two threads execute the multiple data processing tasks of the display unit in different orders, thereby improving the flexibility of display, and can solve the problem of poor flexibility of the display device during display in the related technology, thereby achieving the effect of improving the flexibility of the display device during display.

[0091] In addition, in the control method of the virtual reality display device provided in the embodiment of the present application, since different modes can be selected based on different rendering times to implement multiple data processing tasks of the display unit, this can facilitate the control component to arrange the various data processing tasks more reasonably to achieve the effect of reducing screen delay.

[0092] Several modes involved in the virtual reality display device provided in the embodiments of the present application are described below.

[0093] In an exemplary embodiment, please refer to Figure 4 , Figure 4 2 is a block diagram of another virtual reality device provided in an embodiment of the present application, wherein the virtual reality display device includes a control component 21, a first screen a1 and a second screen a2, that is, the display unit includes the first screen a1 and the second screen a2. In addition, the at least two threads may include a first thread and a second thread.

[0094] Among them, one of the first thread and the second thread is a multiple render target (Multiple RenderTarget, MRT) thread, and the other thread is an asynchronous time warp (ATW) thread. For example, the first thread is an MRT thread and the second thread is an ATW thread, or the first thread is an ATW thread and the second thread is an MRT thread. One of the first screen and the second screen is a screen corresponding to the left eye, and the other screen is a screen corresponding to the right eye. For example, the first screen a1 is a screen corresponding to the left eye, and the second screen is a screen corresponding to the right eye. Of course, the first screen a1 can also be a screen corresponding to the right eye, and the second screen can be a screen corresponding to the left eye.

[0095] In addition, the at least two threads may also include a third thread, a fourth thread, etc., which may execute multiple data processing tasks of the display unit together with the first thread and the second thread. The multiple planes may also include a third screen, a fourth screen, etc., which is not limited in the present embodiment.

[0096] In addition, the virtual reality device may further include a graphics processing unit (GPU) 23, a sensor 24, and a backlight module 25. The graphics processing unit 23 may be used to render image data for screen display, and the sensor 24 may be used to sense user posture data, so that the control component 21 can process some data based on the user posture data sensed by the sensor 24. The backlight module 25 may be used to receive backlight data and control the brightness of the backlight based on the backlight data. In an exemplary embodiment, the backlight module 25 may include a first backlight module 251 corresponding to the first screen, and a second backlight module 252 corresponding to the second screen.

[0097] Please refer to Figure 5 , Figure 5 : is a flow chart of a control method of a virtual reality device provided in an embodiment of the present application, and the method may include the following steps:

[0098] Step 401: Initialize the first thread and the second thread.

[0099] The control component can initialize the first thread and the second thread after the virtual reality device is started. In addition, the control component can also set a counter to record the time consumed for rendering each frame.

[0100] In the virtual reality device provided in the embodiment of the present application, the control component may include a processor, such as an application processor (AP) processor, etc., and the AP processor may be applied to various mobile devices. Among them, the control component may record the time taken by the graphics processor to render a frame.

[0101] Step 402: Determine the average rendering time of each frame within a preset duration.

[0102] The control component can count the rendering time of each frame within the preset duration based on the data provided by the counter set in the above steps, and determine the average rendering time of each frame within the preset duration based on these data. The preset duration can be a pre-set duration, which can be one second, 0.75 seconds, 0.5 seconds, 0.25 seconds, and 0.1 seconds, etc., and this application does not limit this.

[0103] In addition, illustratively, the average rendering time may be an arithmetic mean, a geometric mean, a square mean, a weighted mean, etc. of the rendering times of multiple frames within a preset duration.

[0104] Step 403: When the average rendering time falls within the first duration range, execute the first mode.

[0105] The control component may determine to execute a corresponding mode based on the statistical average rendering time. When the average rendering time falls within a first duration range, the control component may execute the first mode.

[0106] Step 404: When the average rendering time falls within the second duration range, execute the second mode.

[0107] The minimum value of the second duration range is greater than the maximum value of the first duration range, that is, the second duration range is a range that is larger than the first duration range as a whole. When the average rendering time belongs to the second duration range, the control component can execute the second mode.

[0108] Step 405: When the average rendering time falls within the third duration range, execute the third mode.

[0109] The minimum value of the third duration range is greater than the maximum value of the second duration range, that is, the third duration range is the largest range among the three duration ranges. When the average rendering time belongs to the third duration range, the control component can execute the second mode.

[0110] It should be noted that when the virtual reality device is running, the average rendering time may be different depending on the scene to be displayed. For example, when the scene to be displayed is relatively simple, the average rendering time may be lower. At this time, the first mode can be used to perform various tasks. When the complexity of the scene to be displayed is at an average level, the average rendering time may be of medium length. At this time, the second mode can be used to perform various tasks. When the scene to be displayed is more complex, the average rendering time may be higher. At this time, the third mode can be used to perform various tasks. In this way, different modes can be used in different situations to improve the display effect of the virtual reality device provided in the embodiment of the present application in various situations and improve the user experience.

[0111] Of course, the control method of the virtual reality device provided in the embodiment of the present application can also have more modes, and the embodiment of the present application is not limited to this.

[0112] The following describes these three modes respectively. Please refer to Figure 6 , Figure 6 This is a flowchart of a first mode of a virtual reality device provided in an embodiment of the present application, wherein the first duration range can be 0 to 0.5 / f, f is the refresh rate of the display unit, and 1 / f is the display duration of one frame. The rendering time of the first screen can be considered to be the same as the rendering time of the second screen, and then the rendering time of one frame of the first screen and the rendering time of one frame of the second screen are both less than or equal to 0.25 / f. In this case, the display duration of one frame can be divided into 4 equal parts, each equal part is 0.25 / f. For example, when f=60, the display duration of one frame is 1 / 60 seconds, and when f=120, the display duration of one frame is 1 / 120 seconds.

[0113] Please refer to Figure 6 :

[0114] After the start time of the nth frame and at an interval of 0.5 / f (the interval 0.5 / f here may refer to an interval of 0.5 / f from the start time), the control component may render the image data of the nth frame of the first screen through the first thread L1 (this task is in Figure 6 For example, if f=60 times / second, then 0.5 / f=1 / 120 second.

[0115] After the start time of the nth frame and at an interval of 0.75 / f, the image data of the nth frame of the second screen is rendered by the first thread L1 (this task is in Figure 6 Marked as Rendering 2).

[0116] At the start time of the n+1 frame (i.e., the time when the sync signal is received), the image data of the nth frame of the second screen is corrected by the first thread L1 (this task is in Figure 6 The correction may include asynchronous time warping (ATW) and the like.

[0117] After the start time of the nth frame and at an interval of 0.75 / f, the control component can correct the image data of the nth frame of the first screen through the second thread L2 (this task is in Figure 6 The corrected image data of the nth frame of the first screen is obtained.

[0118] At the start time of the n+1th frame, the second thread L2 performs backlight adjustment processing on the corrected image data of the nth frame of the first screen (this task is in Figure 6 Marked as b1), the backlight data of the nth frame of the first screen is obtained. At this time, the corrected image data of the nth frame of the first screen and the backlight data of the nth frame of the first screen are obtained, and then the control component can send the corrected image data of the nth frame of the first screen to the first screen, and send the backlight data of the nth frame of the first screen to the backlight module, for example, the backlight data of the nth frame of the first screen can be sent to the first backlight module corresponding to the first screen through a serial peripheral interface (Serial Peripheral Interface, SPI) or a universal serial bus (Universal Serial Bus, USB) interface. The backlight adjustment process may include regional dimming (LD) processing, etc.

[0119] After the first screen and the backlight module receive the data, they can start scanning and displaying the corrected image data of the nth frame of the first screen at the next moment, that is, after the starting moment of the n+1th frame and at an interval of 0.25 / f. Since the corrected image data of the nth frame of the first screen is half of the image data, the scanning and display time is 0.5 frames, and thus the scanning and display of the corrected image data of the nth frame of the first screen can be completed after the starting moment of the n+1th frame and at an interval of 0.75 / f.

[0120] After the start time of the n+1th frame and after an interval of 0.25 / f, the second thread L2 performs backlight adjustment processing on the corrected image data of the nth frame of the second screen (this task is in Figure 6Marked as b2), the backlight data of the nth frame of the second screen is obtained. At this time, the corrected image data of the nth frame of the second screen and the backlight data of the nth frame of the second screen are obtained, and the control component can send the corrected image data of the nth frame of the second screen to the second plane, and send the backlight data of the nth frame of the second screen to the second backlight module corresponding to the second screen. After the second screen and the backlight module receive the data, the corrected image data of the nth frame of the second screen can be started after the scanning and display of the first screen is completed (that is, after the starting moment of the n+1th frame and at an interval of 0.75 / f). The corrected image data of the nth frame of the second screen can be displayed in a duration of 0.5 / f, that is, the scanning and display can be completed after the starting moment of the n+2th frame and at an interval of 0.25 / f.

[0121] Figure 6 The task included in the dotted box n may be a task for the nth frame, and the task included in the dotted box n+1 may be a task for the n+1th frame.

[0122] Depend on Figure 6 It can be seen that the duration T2 from the moment s2 when the user's gesture is acquired and the image is rendered to the moment k2 when the image is displayed on the screen is 1.75 / f. Figure 1 The duration of 2.5 / f in the scheme shown is shortened by 0.75 / f in the scheme provided in the embodiment of the present application, thereby achieving the effect of reducing the delay.

[0123] In addition, since the control component can perform rendering tasks and correction tasks based on predicted user posture data, the method provided in the embodiment of the present application can improve the accuracy of the predicted user posture data and thus improve the display effect after shortening the above-mentioned time duration because the predicted moment is closer to the actual reality moment.

[0124] For example, please refer to the above for some content about the predicted user posture data Figure 6 , the process of acquiring the corrected image data of the nth frame of the first screen may include:

[0125] At a moment after the start time of the nth frame and at an interval of 0.5 / f, the control component can obtain the first user posture data of the predicted first moment k1, and render the image data of the nth frame of the first screen based on the first user posture data through the first thread L1. Among them, the first moment k1 is the moment after the start time of the n+1th frame. Exemplarily, the first moment k1 is the moment after the start time of the nth frame and at an interval of 0.75 / f. From the above discussion, it can be seen that the moment after the start time of the nth frame and at an interval of 0.75 / f is the moment when the scanning and display of the corrected image data of the nth frame of the first screen is completed. This moment can be considered as the moment when the user sees the corrected image data of the nth frame of the first screen. Using this moment as the moment for predicting some data of the first screen can improve the accuracy of the predicted user posture data. Of course, some other moments can also be determined as the first moment k1. For example, a moment between the moment after the start time of the nth frame and at an interval of 0.25 / f to the moment after the start time of the n+1th frame and at an interval of 0.25 / f can be determined as the first moment k1. Among them, the control component can obtain the above-mentioned predicted user posture data based on some data provided by the sensor. The user posture data may include one or more of the data such as the direction and angle that the user is facing, the direction and angle of movement, the speed, the acceleration, etc. Exemplarily, the prediction operation can calculate or predict the user posture after a given time through the current user posture and the historical data of the user posture, thereby obtaining the user posture data at the display time. The calculation or prediction method includes but is not limited to prediction using a neural network.

[0126] At a time after the start time of the nth frame and at an interval of 0.75 / f, the control component can obtain the predicted second user posture data of the first time again. Since the time after the start time of the nth frame and at an interval of 0.75 / f is closer to the first time k1, compared with the first user posture data, the predicted second user posture data of the first time obtained at this time may be more accurate (closer to the user's actual posture). The control component can correct the image data of the nth frame of the first screen based on the second user posture data through the second thread L2 to obtain the corrected image data of the nth frame of the first screen.

[0127] The process of acquiring the corrected image data of the nth frame of the second screen may include:

[0128] After the start time of the nth frame and at an interval of 0.75 / f, the control component can obtain the third user posture data of the predicted second time k2, and render the image data of the nth frame of the second screen based on the third user posture data through the first thread L1. The second time k2 is the time after the first time k1. Figure 6It is shown that the second moment k2 is a moment after the starting moment of the n+1th frame and at an interval of 0.25 / f. From the above content, it can be known that this moment is the moment when the scanning and display of the corrected image data of the nth frame of the second screen is completed, so using this moment as the second moment k2 can improve the accuracy of the predicted user posture data. Of course, the second moment k2 can also be other moments, for example, a moment between a moment after the starting moment of the nth frame and at an interval of 0.25 / f to a moment after the starting moment of the n+1th frame and at an interval of 0.25 / f can be determined as the second moment k2.

[0129] At the starting moment of the n+1 frame, the control component can obtain the predicted fourth user posture data of the second moment k2, and correct the image data of the nth frame of the second screen based on the fourth user posture data through the first thread L1 to obtain the corrected image data of the nth frame of the second screen. Similarly, since the starting moment of the n+1 frame is closer to the second moment k2 than the moment after the starting moment of the nth frame and with an interval of 0.75 / f, the predicted fourth user posture data of the second moment obtained at this time may be more accurate than the third user posture data. The control component can correct the image data of the nth frame of the second screen based on the fourth user posture data through the second thread L2 to obtain the corrected image data of the nth frame of the first screen.

[0130] It can be seen that the way of acquiring the corrected image data of the n-th frame of the second screen is similar to the way of acquiring the corrected image data of the n-th frame of the first screen.

[0131] It should be noted that in the virtual reality device provided in the embodiment of the present application, the control component can obtain the predicted user posture data each time a rendering task and a correction task are performed, and perform the rendering task and the correction task based on the user posture data. Due to the different moments of obtaining the predicted user posture data, even if the user posture data at the same moment is predicted, the obtained user posture data will be different. For example, Figure 6 In the embodiment, the control component obtains the predicted third user posture data of the second moment k2 after the start moment of the nth frame and at an interval of 0.75 / f, and obtains the predicted fourth user posture data of the second moment k2 at the start moment of the n+1 frame. Although the third user posture data and the fourth user posture data are both data for predicting the user posture of the second moment k2, due to the different prediction moments, there will be differences between the third user posture data and the fourth user posture data, and since the predicted moment of the fourth user posture data is closer to the second moment k2, the fourth user posture data may be closer to the actual posture of the user at the second moment k2 than the third user posture data.

[0132] Please refer to Figure 7 , Figure 7 yes Figure 6 A method flow chart of a first mode is shown, the method may include:

[0133] Step 701 : After the start time of the nth frame and at an interval of 0.5 / f, the control component renders the image data of the nth frame of the first screen through the first thread L1 .

[0134] Step 702: After the start time of the nth frame and at an interval of 0.75 / f, the first thread L1 renders the image data of the nth frame of the second screen. The second thread L2 corrects the image data of the nth frame of the first screen to obtain the corrected image data of the nth frame of the first screen.

[0135] Step 703, at the starting time of the n+1 frame, the image data of the nth frame of the second screen is corrected by the first thread L1 to obtain the corrected image data of the nth frame of the second screen, and the corrected image data of the nth frame of the first screen is backlight adjusted by the second thread L2.

[0136] Step 704: After the start time of the n+1th frame and after an interval of 0.25 / f, the second thread L2 performs backlight adjustment processing on the corrected image data of the nth frame of the second screen.

[0137] For details of steps 701 to 704, please refer to Figure 6 The embodiments shown in the figure will not be described in detail herein.

[0138] Please refer to Figure 8 , Figure 8 This is a flowchart of a second mode of a virtual reality display device provided in an embodiment of the present application, wherein the second duration ranges from 0.5 / f to 1 / f, and f is the refresh rate of the display unit. The rendering time of the first screen can be considered to be the same as the rendering time of the second screen, and thus the rendering time of a frame of the first screen and the rendering time of a frame of the second screen are both less than or equal to 0.5 / f. In this case, the display duration of a frame can be divided into two equal parts, and the duration of each part is 0.5 / f. In the second mode, the control component can perform different tasks in odd frames and even frames through two threads, please refer to Figure 8 , the control components are used to:

[0139] In response to the nth frame being an odd frame (correspondingly, the n-1th frame and the n+1th frame are both even frames), at the start time of the nth frame, the image data of the nth frame of the second screen is rendered through the first thread L1, and the image data of the nth frame of the second screen is corrected to obtain the corrected image data of the nth frame of the second screen, and then the corrected image data of the nth frame of the second screen is backlight adjusted to obtain the backlight data of the nth frame of the second screen. That is, at the start time of the odd frame, the control component can sequentially perform the three tasks of rendering the image data of the nth frame of the second screen, correcting the image data of the nth frame of the second screen, and backlight adjusting the corrected image data of the nth frame of the second screen through the first thread L1.

[0140] After the start time of the n-1th frame and at an interval of 0.5 / f, the image data of the nth frame of the first screen is rendered by the first thread L1, and at the start time of the nth frame, the image data of the nth frame of the second screen is rendered by the first thread L1, and the image data of the nth frame of the second screen is corrected to obtain the corrected image data of the nth frame of the second screen, and then the backlight adjustment processing is performed on the corrected image data of the nth frame of the second screen to obtain the backlight data of the nth frame of the second screen;

[0141] At the starting moment of the nth frame, the image data of the nth frame of the first screen is corrected through the second thread L2 to obtain the corrected image data of the nth frame of the first screen, and backlight adjustment processing is performed on the corrected image data of the nth frame of the first screen to obtain the backlight data of the nth frame of the first screen. After the starting moment of the nth frame and at an interval of 0.5 / f, the image data of the nth frame of the first screen is rendered through the second thread L2.

[0142] At the start time of the n+1th frame, the image data of the nth frame of the first screen is corrected by the first thread L1 to obtain the corrected image data of the nth frame of the first screen, and the backlight adjustment processing is performed on the corrected image data of the nth frame of the first screen by the first thread L1 to obtain the backlight data of the nth frame of the first screen, and then the image data of the n+2th frame of the first screen is rendered;

[0143] At the starting moment of the n+1th frame, the image data of the n+1th frame of the second screen is rendered by the second thread L2, and the image data of the n+1th frame of the second screen is corrected to obtain the corrected image data of the n+1th frame of the second screen, and then the backlight adjustment processing is performed on the corrected image data of the n+1th frame of the second screen to obtain the backlight data of the n+1th frame of the second screen.

[0144] The following is an explanation of some of the contents of user posture data:

[0145] In odd frames, the control components are used to:

[0146] At the starting moment of the nth frame, the predicted fifth user posture data of the third moment is obtained, and the image data of the nth frame of the first screen is corrected based on the fifth user posture data through the second thread to obtain the corrected image data of the nth frame of the first screen, and the third moment is the moment after the starting moment of the nth frame.

[0147] Optionally, the third moment k3 can be a moment between the moment after the starting moment of the nth frame and the moment after the starting moment of the n+1th frame and the moment after the starting moment of the n+1th frame and the moment after the starting moment of the n+1th frame. For example, the third moment k3 can be the starting moment of the n+1th frame, which is the moment when the image scanning and display of the nth frame of the first screen is completed. Determining this moment as the third moment k3 can improve the accuracy of the prediction.

[0148] At a time after the start time of the nth frame and at an interval of 0.5 / f, the sixth user posture data of the predicted fourth time k4 is obtained, and the image data of the n+1th frame of the first screen is rendered based on the fifth user posture data through the second thread, and the fourth time is the time after the start time of the n+1th frame. Optionally, the fourth time k4 can be a time from the time after the start time of the n+1th frame and at an interval of 0.5 / f to the start time of the n+2th frame. For example, the fourth time k4 can be the start time of the n+2th frame, which is the time when the image screen scan display of the n+1th frame of the first screen is completed. Determining this time as the fourth time k4 can improve the accuracy of the prediction.

[0149] At the start time of the nth frame, the seventh user posture data of the predicted fifth moment k2 is obtained, and the image data of the nth frame of the second screen is rendered based on the seventh user posture data through the first thread L1, and then the eighth user posture data of the predicted fifth moment k5 is obtained, and the image data of the nth frame of the second screen is corrected based on the eighth user posture data through the first thread L1 to obtain the corrected image data of the nth frame of the second screen, and the fifth moment k5 is the moment after the start time of the nth frame. Optionally, the fifth moment k5 is the moment after the start time of the n+1th frame and the interval is 0.5 / f, which is the moment when the image screen scanning display of the nth frame of the second screen is completed, and determining this moment as the fifth moment k5 can improve the accuracy of the prediction.

[0150] In even frames, the control components are used to:

[0151] At the starting moment of the n+1th frame, the ninth user posture data of the predicted fourth moment k4 is obtained, and the image data of the n+1th frame of the first screen is corrected based on the ninth user posture data through the first thread L1 to obtain the corrected image data of the n+1th frame of the first screen.

[0152] After the start time of the n+1th frame and at an interval of 0.5 / f, the tenth user posture data of the predicted sixth moment k6 is obtained, and then the image data of the n+2th frame of the first screen is rendered based on the tenth user posture data through the first thread L1. The sixth moment k6 is the moment after the start time of the n+2th frame and at an interval of 0.5 / f. Optionally, the sixth moment k6 can be the end time of the n+2th frame, which is the moment when the image screen scanning display of the n+2th frame of the first screen is completed. Determining this moment as the sixth moment k6 can improve the accuracy of the prediction.

[0153] At the starting moment of the n+1th frame, the eleventh user posture data of the predicted seventh moment k7 is obtained, and the image data of the n+1th frame of the second screen is rendered by the second thread L2 based on the eleventh user posture data, and then the twelfth user posture data of the predicted seventh moment is obtained, and the image data of the n+1th frame of the second screen is corrected by the second thread L2 based on the twelfth user posture data to obtain the corrected image data of the n+1th frame of the second screen, and the seventh moment k7 is the moment after the starting moment of the n+2th frame. Optionally, the seventh moment k7 is the moment after the starting moment of the n+2th frame and the interval is 0.5 / f, which is the moment when the image screen scanning display of the n+1th frame of the second screen is completed, and determining this moment as the seventh moment k7 can improve the accuracy of the prediction.

[0154] Depend on Figure 8 It can be seen that the duration T3 from the moment s3 when the user's gesture is acquired and the image is rendered to the moment k5 when the image is displayed on the screen is 2 / f. Figure 1 The duration of 2.5 / f in the scheme shown is shortened by 0.5 / f in the scheme provided in the embodiment of the present application, thereby achieving the effect of reducing the delay.

[0155] Fig. 9 This is a flow chart corresponding to the second mode of the virtual reality display device provided in the embodiment of the present application. Please refer to Fig. 9 , the second mode may include the following steps:

[0156] Step 901, in response to the nth frame being an odd frame, at the starting moment of the nth frame, the image data of the nth frame of the second screen is rendered by the first thread L1, and the image data of the nth frame of the second screen is corrected to obtain the corrected image data of the nth frame of the second screen, and then backlight adjustment processing is performed on the corrected image data of the nth frame of the second screen to obtain the backlight data of the nth frame of the second screen.

[0157] Step 902: at the starting moment of the nth frame, the image data of the nth frame of the first screen is corrected by the second thread to obtain the corrected image data of the nth frame of the first screen, and backlight adjustment processing is performed on the corrected image data of the nth frame of the first screen to obtain the backlight data of the nth frame of the first screen.

[0158] Step 903 : After the start time of the nth frame and at a time interval of 0.5 / f, render the image data of the nth frame of the first screen through the second thread L2 .

[0159] Step 904: at the starting moment of the (n+1)th frame, the image data of the nth frame of the first screen is corrected through the first thread L1 to obtain the corrected image data of the nth frame of the first screen, and the backlight adjustment processing is performed on the corrected image data of the nth frame of the first screen through the first thread L1 to obtain the backlight data of the nth frame of the first screen.

[0160] Step 905: at the starting moment of the n+1th frame, the image data of the n+1th frame of the second screen is rendered by the second thread L2, and the image data of the n+1th frame of the second screen is corrected to obtain the corrected image data of the n+1th frame of the second screen, and then the backlight adjustment processing is performed on the corrected image data of the n+1th frame of the second screen to obtain the backlight data of the n+1th frame of the second screen.

[0161] Step 906 : After the start time of the n+1th frame and at a time interval of 0.5 / f, render the image data of the n+2th frame of the first screen through the first thread L1 .

[0162] That is, for any frame, if the frame is an odd frame, the execution starts from step 901, and if the frame is an even frame, the execution starts from step 904. The above steps can be referred to Figure 8 The embodiments shown in the figure will not be described in detail herein.

[0163] The third mode is described below:

[0164] Please refer to Fig.10 , Fig.10 It is a flowchart of a third mode of a virtual reality display device provided in an embodiment of the present application, wherein the control component can execute the data processing tasks of the display unit in sequence based on the third mode through at least two threads when the rendering time is within a third time range; the third time range is 1 / f to 2 / f, f is the refresh rate of the display unit, on this basis, the rendering time of one frame of the first screen and the rendering time of one frame of the second screen are both less than or equal to 1 frame.

[0165] In an exemplary embodiment, the control component is used to:

[0166] After the start time of the n-1th frame and at an interval of 0.5 / f, the image data of the nth frame of the first screen is rendered by the first thread L1, the image of the nth frame of the first screen is corrected by the first thread L1 to obtain the corrected image data of the nth frame of the first screen, and then the first thread L1 performs backlight adjustment processing on the corrected image data of the nth frame of the first screen to obtain the backlight data of the nth frame of the first screen. That is, the control component can perform the three tasks of rendering the image data of the nth frame of the first screen, correcting the image of the nth frame of the first screen, and performing backlight adjustment processing on the corrected image data of the nth frame of the first screen in sequence after the start time of the n-1th frame and at an interval of 0.5 / f. Since the rendering time is less than or equal to 0.5 / f, these three tasks can be completed within the display duration of one frame.

[0167] At the start time of the nth frame, the image data of the nth frame of the second screen is rendered by the second thread L2, the image of the nth frame of the second screen is corrected by the second thread L2 to obtain the corrected image data of the nth frame of the second screen, and then the corrected image data of the nth frame of the second screen is backlight adjusted by the second thread L2 to obtain the backlight data of the nth frame of the second screen. That is, the control component can, at the start time of the nth frame, sequentially perform the three tasks of rendering the image data of the nth frame of the first screen, correcting the image of the nth frame of the first screen, and backlight adjusting the corrected image data of the nth frame of the first screen. Since the rendering time is less than or equal to 0.5 / f, these three tasks can be completed within the display duration of one frame.

[0168] The following is an explanation of some of the contents of user posture data:

[0169] In an exemplary embodiment, the control component is used to:

[0170] At a time after the start time of the n-1th frame and at an interval of 0.5 / f, the predicted thirteenth user posture data of the eighth time k8 is obtained, and the image data of the nth frame of the first screen is rendered through the first thread L1 based on the thirteenth user posture data. The eighth time k8 is the time after the start time of the nth frame. Optionally, the eighth time k8 can be a time between the time after the start time of the nth frame and at an interval of 0.5 / f to the time after the start time of the n+1th frame and at an interval of 0.5 / f. For example, the eighth time k8 can be the start time of the n+1th frame, which is the time when the image screen scan display of the nth frame of the first screen is completed. Determining this time as the eighth time k8 can improve the accuracy of the prediction.

[0171] After acquiring the image data of the nth frame of the first screen, the fourteenth user posture data of the predicted eighth moment k8 is acquired, and the image of the nth frame of the first screen is corrected based on the tenth user posture data through the first thread L1 to obtain the corrected image data of the n-1th frame of the first screen.

[0172] At the start time of the nth frame, the fifteenth user posture data of the predicted ninth time k9 is obtained, and the image data of the nth frame of the second screen is rendered based on the fifteenth user posture data through the first thread L1, and the ninth time is the time after the eighth time. The ninth time k9 can be a time between the time after the start time of the nth frame and the time after the start time of the n+1th frame and the time after the start time of the n+1th frame and the time after the start time of the n+1th frame and the time after the start time of the n+1th frame and the time after the image scanning display of the nth frame of the second screen is completed. Determining this time as the ninth time k9 can improve the accuracy of the prediction.

[0173] After acquiring the image data of the nth frame of the second screen, the predicted sixteenth user posture data of the ninth moment is acquired, and the image data of the nth frame of the second screen is corrected based on the sixteenth user posture data to obtain the corrected image data of the nth frame of the second screen.

[0174] like Fig.11 As shown, Fig.11 This is a flowchart of a virtual reality display device in the related art, wherein the delay Ta1 reaches 3 / f at this time. And the duration Ta2 between the moment when the user's posture is obtained to start correcting the image and the moment when the left eye screen scan is completed is 1 frame. In addition, when the left eye image is ATW processed at the start time of the n+1th frame, the left eye image used is rendered at the start time of the n-1th frame. If the delay is too high, the problem of screen jitter may occur.

[0175] But by Fig.10 It can be seen that in the embodiment of the present application, the time T5 from the moment s4 when the user's gesture is acquired and the image is rendered to the moment k9 when the image is displayed on the screen is 2 / f, and there will be no problem of screen jitter. Fig.11 The duration of 3 / f in the scheme shown is shortened by 1 / f in the scheme provided in the embodiment of the present application, so that the effect of reducing the delay can be achieved. In addition, the duration T4 from the moment a when the user's posture is acquired to the moment k8 when the image of the first screen is scanned and displayed on the screen is less than 1 / f, which can improve the accuracy of the prediction and thus improve the display effect of the virtual reality display device.

[0176] Fig.12This is a flow chart corresponding to the third mode of the virtual reality display device provided in the embodiment of the present application. Please refer to Fig.12 , the third mode may include the following steps:

[0177] Step 1201, after the start time of the n-1th frame and at an interval of 0.5 / f, render the image data of the nth frame of the first screen through the first thread L1, correct the image of the nth frame of the first screen through the first thread L1 to obtain the corrected image data of the nth frame of the first screen, and then perform backlight adjustment processing on the corrected image data of the nth frame of the first screen through the first thread L1 to obtain the backlight data of the nth frame of the first screen.

[0178] Step 1202, at the starting moment of the nth frame, render the image data of the nth frame of the second screen through the second thread L2, correct the image of the nth frame of the second screen through the second thread L2 to obtain the corrected image data of the nth frame of the second screen, and then perform backlight adjustment processing on the corrected image data of the nth frame of the second screen through the second thread L2 to obtain the backlight data of the nth frame of the second screen.

[0179] about Fig.12 The steps in the above can also refer to Fig.10 The embodiments shown in the figure will not be described in detail herein.

[0180] In summary, an embodiment of the present application provides a virtual reality display device including a control component and a display unit, wherein the control component can obtain the rendering time of a frame of the display unit, and based on the difference in the rendering time, execute multiple data processing tasks of the display unit in different modes through at least two threads, wherein in different modes, the order in which at least two threads execute the multiple data processing tasks of the display unit is different, thereby improving the flexibility of display, and can solve the problem of poor flexibility of the virtual reality display device during display in the related art, thereby achieving the effect of improving the flexibility of the virtual reality display device during display.

[0181] In addition, in the virtual reality display device provided in the embodiment of the present application, since the control component can select different modes based on the different rendering times to implement multiple data processing tasks of the display unit, this can facilitate the control component to arrange the various data processing tasks more reasonably to achieve the effect of reducing screen delay.

[0182] In addition, an embodiment of the present application also provides a computer storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the above-mentioned control method of the virtual reality display device.

[0183] In this application, the terms "first", "second", "third", "fourth", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "plurality" refers to two or more, unless otherwise clearly defined.

[0184] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0185] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0187] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A virtual reality display device, characterized in that: The virtual reality display device comprises a control component and a display unit, wherein the control component is used for: Obtaining the rendering time consumed for rendering a frame of the display unit; In response to the rendering time being within a first duration range, executing multiple data processing tasks of the display unit in sequence based on a first mode through at least two threads, the multiple data processing tasks including a rendering task of image data of the display unit and a correction task of the image data, and the first mode is a mode corresponding to the first duration range; In response to the rendering time being within a second duration range, the data processing tasks of the display unit are executed sequentially based on a second mode through the at least two threads, the second mode is a mode corresponding to the second duration range, the minimum value of the second duration range is greater than the maximum value of the first duration range, and the execution order of the multiple data processing tasks of the display unit in the second mode is different from the execution order of the multiple data processing tasks of the display unit in the first mode.

2. The virtual reality display device according to claim 1, characterized in that: The at least two threads include a first thread and a second thread, and the display unit includes a first screen and a second screen; The first duration ranges from 0 to 0.5 / f, and the rendering time of one frame of the first screen and the rendering time of one frame of the second screen are both less than or equal to 0.25 / f, where f is the refresh rate of the display unit.

3. The virtual reality display device according to claim 2, characterized in that: The control component is used to: At a time after the start time of the nth frame and at an interval of 0.5 / f, the image data of the nth frame of the first screen is rendered by the first thread, and at a time after the start time of the nth frame and at an interval of 0.75 / f, the image data of the nth frame of the second screen is rendered by the first thread, and at the start time of the n+1 frame, the image data of the nth frame of the second screen is corrected by the first thread to obtain corrected image data of the nth frame of the second screen; After the starting moment of the nth frame and at an interval of 0.75 / f, the image data of the nth frame of the first screen is corrected by the second thread to obtain the corrected image data of the nth frame of the first screen; at the starting moment of the n+1th frame, the backlight adjustment processing is performed on the corrected image data of the nth frame of the first screen by the second thread to obtain the backlight data of the nth frame of the first screen; after the starting moment of the n+1th frame and at an interval of 0.25 / f, the backlight adjustment processing is performed on the corrected image data of the nth frame of the second screen by the second thread to obtain the backlight data of the nth frame of the second screen.

4. The virtual reality display device according to claim 3, characterized in that: The control component is used to: At a time after the start time of the nth frame and at an interval of 0.5 / f, obtaining the predicted first user posture data of the first time, and rendering the image data of the nth frame of the first screen based on the first user posture data through the first thread, the first time being the time after the start time of the n+1th frame; After the start time of the nth frame and at an interval of 0.75 / f, the predicted second user posture data of the first moment is obtained, and the image data of the nth frame of the first screen is corrected based on the second user posture data through the second thread to obtain the corrected image data of the nth frame of the first screen.

5. The virtual reality display device according to claim 3, characterized in that: The control component is used to: At a time after the start time of the nth frame and at an interval of 0.75 / f, obtaining third user posture data of a predicted second time, and rendering image data of the nth frame of the second screen based on the third user posture data through the first thread, the second time being a time after the first time; At the starting moment of the n+1 frame, the predicted fourth user posture data of the second moment is obtained, and the image data of the nth frame of the second screen is corrected based on the fourth user posture data by the first thread to obtain the corrected image data of the nth frame of the second screen.

6. The virtual reality display device according to claim 1, characterized in that: The at least two threads include a first thread and a second thread, and the display unit includes a first screen and a second screen; The second duration ranges from 0.5 / f to 1 / f, and the rendering time of one frame of the first screen and the rendering time of one frame of the second screen are both less than or equal to 0.5 / f, where f is the refresh rate of the display unit.

7. The virtual reality display device according to claim 6, characterized in that: The control component is used to: In response to the nth frame being an odd frame, at the start time of the nth frame, the image data of the nth frame of the second screen is rendered by the first thread, and the image data of the nth frame of the second screen is corrected to obtain corrected image data of the nth frame of the second screen, and backlight adjustment processing is performed on the corrected image data of the nth frame of the second screen to obtain backlight data of the nth frame of the second screen; At the start time of the nth frame, the image data of the nth frame of the first screen acquired in the n-1th frame is corrected by the second thread to obtain the corrected image data of the nth frame of the first screen, and the backlight adjustment processing is performed on the corrected image data of the nth frame of the first screen to obtain the backlight data of the nth frame of the first screen, and then the image data of the n+1th frame of the first screen is rendered; At the start time of the n+1th frame, the image data of the nth frame of the first screen is corrected by the first thread to obtain the corrected image data of the nth frame of the first screen, and the backlight adjustment processing is performed on the corrected image data of the nth frame of the first screen by the first thread to obtain the backlight data of the nth frame of the first screen, and then the image data of the n+2th frame of the first screen is rendered; At the starting moment of the n+1th frame, the image data of the n+1th frame of the second screen is rendered by the second thread, and the image data of the n+1th frame of the second screen is corrected to obtain the corrected image data of the n+1th frame of the second screen, and then the corrected image data of the n+1th frame of the second screen is backlight adjusted to obtain the backlight data of the n+1th frame of the second screen.

8. The virtual reality display device according to claim 7, characterized in that: The control component is used to: At the start time of the nth frame, fifth user posture data of the predicted third time is obtained, and image data of the nth frame of the first screen is corrected based on the fifth user posture data by the second thread to obtain corrected image data of the nth frame of the first screen, wherein the third time is a time after the start time of the nth frame; At a time after the start time of the nth frame and at an interval of 0.5 / f, obtaining the predicted sixth user gesture data at a fourth time, and rendering the image data of the n+1th frame of the first screen based on the fifth user gesture data through the second thread, the fourth time being the time after the start time of the n+1th frame; At the starting moment of the nth frame, the predicted seventh user posture data of the fifth moment is obtained, and the image data of the nth frame of the second screen is rendered based on the seventh user posture data by the first thread, and then the predicted eighth user posture data of the fifth moment is obtained, and the image data of the nth frame of the second screen is corrected based on the eighth user posture data by the first thread to obtain the corrected image data of the nth frame of the second screen, and the fifth moment is the moment after the starting moment of the nth frame.

9. The virtual reality display device according to claim 7, characterized in that: The control component is used to: At the starting moment of the n+1th frame, the predicted ninth user gesture data of the fourth moment is obtained, and the image data of the n+1th frame of the first screen is corrected by the first thread based on the ninth user gesture data to obtain the corrected image data of the n+1th frame of the first screen, and after the starting moment of the n+1th frame and at a time interval of 0.5 / f, the predicted tenth user gesture data of the sixth moment is obtained, and then the image data of the n+1th frame of the first screen is rendered by the first thread based on the tenth user gesture data, the sixth moment being a time interval of 0.5 / f after the starting moment of the n+2th frame; At the starting moment of the n+1th frame, the eleventh user posture data of the predicted seventh moment is obtained, and the image data of the n+1th frame of the second screen is rendered by the second thread based on the eleventh user posture data, and then the twelfth user posture data of the predicted seventh moment is obtained, and the image data of the n+1th frame of the second screen is corrected by the second thread based on the twelfth user posture data to obtain the corrected image data of the n+1th frame of the second screen, and the seventh moment is the moment after the starting moment of the n+2th frame.

10. The virtual reality display device according to claim 1, characterized in that: The at least two threads include a first thread and a second thread, and the display unit includes a first screen and a second screen; The control component is also used for: In response to the rendering time being within a third duration range, the data processing tasks of the display unit are sequentially executed based on a third mode by the at least two threads, the third mode being a mode corresponding to the third duration range, and the execution order of the multiple data processing tasks of the display unit in the third mode is different from the execution order of the multiple data processing tasks of the display unit in the first mode and the second mode; The third duration range is 1 / f to 2 / f, and the rendering time of one frame of the first screen and the rendering time of one frame of the second screen are both less than or equal to 1 / f, where f is the refresh frequency of the display unit.

11. The virtual reality display device according to claim 10, characterized in that: The control component is used to: After the start time of the n-1th frame and at an interval of 0.5 / f, the image data of the nth frame of the first screen is rendered by the first thread, and the image data of the nth frame of the first screen is corrected by the first thread to obtain corrected image data of the nth frame of the first screen, and the backlight adjustment processing is performed on the corrected image data of the nth frame of the first screen by the first thread to obtain backlight data of the nth frame of the first screen; At the starting moment of the nth frame, the image data of the nth frame of the second screen is rendered by the second thread, the image of the nth frame of the second screen is corrected by the second thread to obtain the corrected image data of the nth frame of the second screen, and then the corrected image data of the nth frame of the second screen is backlight adjusted by the second thread to obtain the backlight data of the nth frame of the second screen.

12. The virtual reality display device according to claim 11, characterized in that: The control component is used to: At a time after the start time of the n-1th frame and at an interval of 0.5 / f, obtaining the thirteenth user gesture data of the predicted eighth time, and rendering the image data of the nth frame of the first screen based on the thirteenth user gesture data through the first thread, the eighth time being the time after the start time of the nth frame; After acquiring the image data of the nth frame of the first screen, the predicted fourteenth user posture data of the eighth moment is acquired, and the image of the nth frame of the first screen is corrected based on the tenth user posture data through the first thread to obtain the corrected image data of the nth frame of the first screen.

13. The virtual reality display device according to claim 11, characterized in that: The control component is used to: At the start time of the nth frame, obtaining the predicted fifteenth user gesture data at the ninth time, and rendering the image data of the nth frame of the second screen based on the fifteenth user gesture data by the first thread, the ninth time being a time after the eighth time; After acquiring the image data of the nth frame of the second screen, the predicted sixteenth user posture data of the ninth moment is acquired, and the image data of the nth frame of the second screen is corrected based on the sixteenth user posture data to obtain the corrected image data of the nth frame of the second screen.

14. A method for controlling a virtual reality display device, characterized in that: The method comprises: Obtaining the rendering time consumed for rendering a frame of the display unit; In response to the rendering time being within a first duration range, executing multiple data processing tasks of the display unit in sequence based on a first mode through at least two threads, the multiple data processing tasks including a rendering task of image data of the screen and a correction task of the image data, and the first mode is a mode corresponding to the first duration range; In response to the rendering time being within a second duration range, the data processing tasks of the display unit are executed sequentially based on a second mode through at least two threads, the second mode is a mode corresponding to the second duration range, the minimum value of the second duration range is greater than the maximum value of the first duration range, and the execution order of the multiple data processing tasks of the display unit in the second mode is different from the execution order of the multiple data processing tasks of the display unit in the first mode.

15. A non-volatile computer storage medium, characterized in that: The computer storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the control method of the virtual reality display device as described in claim 14.

Images