VR image display method and virtual reality equipment

Abstract

The invention relates to the technical field of VR, provides a VR image display method and virtual reality equipment, and is used for improving the display speed of VR images. According to the method, when a display screen is lightened, a first process is immediately started to load a transition animation, binocular screen display is carried out through OpenGL, in order to ensure that the transition animation is globally controllable, when the display screen is lightened, corresponding threads are also started to execute image acquisition, VR mode starting and SLAM positioning tasks, and the method is simple and convenient to operate. And when the image frame is successfully acquired, the VR mode is started and the SLAM positioning result is stable, ending the display of the global transition animation, and immediately displaying the acquired image frame. The gray screen phenomenon is reduced by displaying the transition animation, and the transition animation can prompt a user that the VR application is being started, so that the screen fixing problem caused by frequent screen turn-on and turn-off operations when the VR image cannot be seen is avoided, the starting speed of the application is improved, the VR image is quickly displayed, and the user obtains better VR experience.

Description

Technical Field

[0001] The present application relates to the field of virtual reality (VR) technology and provides a VR image display method and a virtual reality device. Background Art

[0002] When using a VR device, the user will take off the VR device or use the power button to actively turn the screen on and off to calibrate the screen. Ideally, when the user puts on the VR device, a stable and accurate VR image should be displayed immediately. However, in reality, there is a delay of about 800ms-1200ms in the display of the VR image. Therefore, the user will see a gray screen for a period of time before seeing the normal display of the VR image. In this way, some users will press the power button again when the screen is gray to display the VR image again, thereby increasing the startup time of the VR device and providing a poor user experience. Summary of the invention

[0003] The embodiments of the present application provide a VR image display method and a virtual reality device, which are used to increase the display speed of VR images.

[0004] On the one hand, an embodiment of the present application provides a VR image display method, which is applied to a VR device, including:

[0005] In response to the display screen lighting operation, the first process is started to load the transition animation and perform binocular display, wherein the transition animation is used to prompt that the VR application is starting and the screen should not be turned on and off frequently;

[0006] During the transition animation display, the rendering thread of the VR application is started to obtain image frames, and the second thread of the VR mode is started to synchronize the multi-eye camera, and the positioning thread is started to obtain the posture data of the virtual reality device;

[0007] When the image frame is acquired successfully, the synchronization of the multi-camera is completed, and the posture data is stable, setting the loading flag of the transition animation to an off state;

[0008] The transition animation is stopped from being displayed according to the closed state of the loading mark, and the image frame is displayed.

[0009] Optionally, starting the second thread of the VR mode to synchronize the multi-eye cameras includes:

[0010] Setting the power function mode of the screen through the SurfaceFlinger thread, creating a second thread of the VR mode in the power function mode, wherein the initial state of the second thread is a disabled state to indicate that data of the VR mode is unavailable;

[0011] Calling the second thread to start the multi-eye camera on the virtual reality device and synchronize the multi-eye camera;

[0012] When the multi-camera synchronization is completed, the target state of the second thread is set to an enabled state to notify the positioning thread that positioning data preparation is complete.

[0013] Optionally, when the target state of the second thread is the start state, starting the positioning thread to obtain the position and posture data of the virtual reality device includes:

[0014] Calling the positioning thread to obtain the multi-eye images synchronously acquired by the multi-eye camera, and acquiring the motion data acquired by the IMU of the virtual reality device itself;

[0015] According to the multi-eye image, an initial 6-DOF posture of the virtual reality device is obtained by using a PnP algorithm;

[0016] According to the motion data, the initial 6-DOF posture is globally optimized to obtain the target 6-DOF posture.

[0017] Optionally, displaying the image frame includes:

[0018] When the perspective mode of the virtual reality device is turned off, calling the rendering thread to render the image frames of the VR application and perform binocular screen display;

[0019] When the perspective mode of the virtual reality device is turned on, the rendering thread is called to render the image frames of the VR application on the first layer, and to render the image frames of the multi-eye camera on the second layer, and the two image frames are superimposed and displayed on the binocular screen through the first layer and the second layer.

[0020] Optionally, the rendering thread, the second thread and the positioning thread are executed concurrently, and the execution time of the rendering thread is respectively shorter than the execution time of the second thread and the positioning thread.

[0021] On the other hand, an embodiment of the present application provides a virtual reality device, including a processor, a memory, a display screen, and a multi-eye camera, wherein the multi-eye camera, the display screen, the memory, and the processor are connected via a bus;

[0022] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0023] In response to the lighting operation of the display screen, starting the first process to load the transition animation and perform binocular display, wherein the transition animation is used to prompt that the virtual reality device is starting and the user should not frequently perform the screen lighting and off operations;

[0024] During the transition animation display, a rendering thread of the VR application is started to obtain image frames, and a second thread of the VR mode is started to synchronize the multi-eye camera, and a positioning thread is started to obtain the posture data of the virtual reality device;

[0025] When the image frame is acquired successfully, the synchronization of the multi-camera is completed, and the posture data is stable, setting the loading flag of the transition animation to an off state;

[0026] The transition animation is stopped from being displayed according to the closed state of the loading mark, and the image frame is displayed through the display screen.

[0027] Optionally, the processor starts a second thread in VR mode to synchronize the multi-eye cameras, and the specific operation is:

[0028] Setting the power function mode of the screen through the SurfaceFlinger thread, creating a second thread of the VR mode in the power function mode, wherein the initial state of the second thread is a disabled state to indicate that data of the VR mode is unavailable;

[0029] Calling the second thread to start the multi-eye camera on the virtual reality device and synchronize the multi-eye camera;

[0030] When the multi-camera synchronization is completed, the target state of the second thread is set to an enabled state to notify the positioning thread that positioning data preparation is complete.

[0031] Optionally, when the target state of the second thread is the start state, the processor starts the positioning thread to obtain the position and posture data of the virtual reality device, and the specific operation is:

[0032] Calling the positioning thread to obtain the multi-eye images synchronously acquired by the multi-eye camera, and acquiring the motion data acquired by the IMU of the virtual reality device itself;

[0033] According to the multi-eye image, an initial 6-DOF posture of the virtual reality device is obtained by using a PnP algorithm;

[0034] According to the motion data, the initial 6-DOF posture is globally optimized to obtain the target 6-DOF posture.

[0035] Optionally, the processor displays the image frame through the display screen, and the specific operation is:

[0036] When the perspective mode of the virtual reality device is turned off, calling the rendering thread to render the image frames of the VR application and perform binocular screen display;

[0037] When the perspective mode of the virtual reality device is turned on, the rendering thread is called to render the image frames of the VR application on the first layer, and to render the image frames of the multi-eye camera on the second layer, and the two image frames are superimposed and displayed on the binocular screen through the first layer and the second layer.

[0038] Optionally, the rendering thread, the second thread and the positioning thread are executed concurrently, and the execution time of the rendering thread is respectively shorter than the execution time of the second thread and the positioning thread.

[0039] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of a VR image display method provided in an embodiment of the present application.

[0040] The beneficial effects of a method for displaying a VR image provided by an embodiment of the present application are as follows:

[0041] When the display screen is turned on, the transition animation is loaded immediately and binocular display is performed, thereby prompting the user that the VR application is starting, avoiding frequent screen on and off operations, and improving the efficiency of application startup. At the same time, during the display of the transition animation, the display screen lighting operation will trigger the start of the rendering thread for acquiring image frames, the second thread of the VR mode, and the positioning thread for acquiring posture data. The execution results of these three threads are used as the conditions for the end of the transition animation, thereby globally controlling the display time of the transition animation and ensuring the normal display of the image frames.

[0042] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 A schematic diagram of a VR device provided in an embodiment of the present application;

[0045] Figure 2 A flow chart of a VR image display method provided in an embodiment of the present application;

[0046] Figure 3 A transition animation display effect diagram provided in an embodiment of the present application;

[0047] Figure 4 A flow chart of a method for starting VR mode provided in an embodiment of the present application;

[0048] Figure 5 A flow chart of the SLAM positioning method provided in the embodiment of the present application;

[0049] Figure 6 SLAM algorithm architecture diagram provided for the embodiment of the present application;

[0050] Figure 7 A flow chart of a VR image display method provided in an embodiment of the present application;

[0051] Fig. 8A This is a display effect diagram of a VR image when the perspective mode provided in an embodiment of the present application is turned off;

[0052] Figure 8B This is a display effect diagram of a VR image when the perspective mode provided in an embodiment of the present application is turned on;

[0053] Fig. 9 A flow chart of a method for switching from a transition animation to a VR image provided in an embodiment of the present application;

[0054] Fig.10 A schematic diagram of a process of switching a transition animation to a VR image provided in an embodiment of the present application;

[0055] Fig.11 A structural diagram of a virtual reality device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0057] VR devices (such as all-in-one VR devices) are a new type of mobile display device that can provide an immersive experience and are now widely used in various fields. At present, VR devices include 3-degree-of-freedom (Dof) devices and 6Dof devices. Since 3Dof only contains translation information, while 6Dof contains translation information and rotation information, the control accuracy is higher. Therefore, 6Dof devices are more commonly used in the market.

[0058] Usually, VR devices are equipped with multiple cameras. Figure 1 The area circled in the middle has an inertial measurement unit (IMU) installed inside. Therefore, the simultaneous localization and mapping (SLAM) technology can be used to provide accurate pose data (i.e. translation and rotation information) for the 6Dof device.

[0059] At present, there is a time delay when VR devices display VR images. Therefore, users will see a gray screen for a period of time before they can see the normal VR image. In this way, some users will press the power button again to display the VR image again when the screen is gray. The frequent on and off operations of the screen will cause problems such as unstable SLAM positioning, abnormal VR mode, and increased VR application startup time, thereby increasing the probability of gray screen or fixed screen problems, affecting the user's immersive experience.

[0060] In view of this, an embodiment of the present application provides a VR image display method. When the display screen of the VR device is turned on, the method immediately loads a prompt that the VR device is starting a transition animation and performing binocular display, thereby avoiding the user from entering the gray screen stage when using the VR device and being unable to see the frequent screen on and off operations generated by the VR image in real time, effectively improving the startup efficiency of the VR device. Among them, the display of the transition animation cannot be simply achieved using the life cycle of screen lighting. This method globally controls the end of the transition animation from the three processes of SLAM posture data preparation, VR mode startup and image rendering, and immediately displays the image frame of the VR application after the transition animation ends, ensuring the normal display of the image frame, while also reducing the probability of gray screen problems and improving the user's immersive experience.

[0061] See also Figure 2 , is a flow chart of a VR image display method provided in an embodiment of the present application, which is executed by a VR device and mainly includes the following steps:

[0062] S201: In response to a display screen lighting operation, a first process is started to load a transition animation and perform binocular display, wherein the transition animation is used to prompt that the VR device is starting and that the screen should not be turned on and off frequently.

[0063] In actual applications, after the user puts on the VR device or presses the power button, the display screen of the VR device lights up. Considering that there is a delay in the display of VR images, in order to prevent users from frequently turning the screen on and off after they cannot see the VR images, in one example, after the display screen lights up, the first process is immediately started to load the transition animation, and binocular display is performed through OpenGL to remind the user that the VR device is starting up and not to frequently turn the screen on and off. Among them, the effect of the transition animation binocular display is as follows: Figure 3 shown.

[0064] S202: During the transition animation display, start the rendering thread of the VR application to obtain image frames, start the second thread of the VR mode to synchronize the multi-eye cameras, and start the positioning thread to obtain the posture data of the virtual reality device.

[0065] Generally, ordinary animations can be displayed globally during the life cycle of the screen, but transition animations are only used as prompts for temporary display. After the VR application is started, the image of the VR application still needs to be displayed normally. Therefore, it cannot be simply controlled using the life cycle of the screen. In addition, the transition animation and the image display of the VR application involve cross-process communication, and the display of the transition animation needs to be globally controllable.

[0066] In one example, during the startup process of a VR device, that is, while displaying a transition animation, the background will execute tasks such as VR image rendering thread recovery, VR mode data preparation, and SLAM positioning. The execution time of these tasks varies, so it is necessary to accurately determine the timing for the transition animation to end display in order to avoid increasing useless event waiting time, thereby enhancing the user's VR experience.

[0067] In actual applications, data preparation for VR mode consumes a certain amount of time. Ideally, VR mode takes about 500ms-80ms from starting (stating) to starting (started), which is one of the main time-consuming tasks for starting VR applications. In the case of abnormal startup caused by users frequently turning the screen on and off, the time consumed for starting VR mode cannot be accurately calculated.

[0068] like Figure 4 As shown in the figure, the startup process of preparing data for VR mode mainly includes the following steps:

[0069] S202_1: Determine whether the display screen is on, if so, execute S202_2, if not, end.

[0070] When the display of the VR device is turned on, the screen driver triggers the SurfaceFlinger service and the VR mode starts.

[0071] S202_2: Set the power function mode of the screen through the SurfaceFlinger thread, and create a second thread of the VR mode in the power function mode.

[0072] After the VR mode starts, the power function mode of the screen is set through the SurfaceFlinger thread, the power function mode of the screen is set from the off state (HWC_POWER_MODE_OFF) to the on state (HWC_POWER_MODE_NORMAL), and the second thread of the VR mode is created in the power function mode.

[0073] Among them, the initial state of the second thread is a stopped state, which means that the data of the VR mode is not prepared, thereby indicating that the data is unavailable, and the second thread is required to prepare the data of the VR mode to complete the startup of the VR mode.

[0074] S202_3: Call the second thread to start the multi-eye camera on the VR device and synchronize the multi-eye camera.

[0075] The VR device is equipped with multi-eye cameras (usually four or two) to provide a data source for SLAM positioning. To ensure the accuracy and stability of SLAM positioning, the second thread needs to synchronize the multi-eye cameras after turning on them.

[0076] S202_4: Determine whether the synchronization is completed, if so, execute S202_5, if not, end.

[0077] S202_5: Set the target state of the second thread to the enabled state to notify the positioning thread that positioning data preparation is complete.

[0078] When the multi-camera synchronization is completed, it means that the VR mode is started. At this time, the target state of the second thread can be set to the enabled state to notify the positioning thread that the positioning data is ready, SLAM positioning can be performed, and the second thread task is completed.

[0079] During the startup of VR mode, the startup and synchronization of multiple cameras takes a lot of time. Therefore, the use of efficient synchronization methods has an important impact on improving the startup of VR applications.

[0080] It should be noted that the embodiments of the present application do not impose any restrictive requirements on the synchronization method of the multi-camera. For example, a feature point matching synchronization method or a timestamp synchronization method may be used.

[0081] In order to ensure the SLAM positioning effect of VR devices, VR devices are usually calibrated before leaving the factory and the calibrated parameters are maintained. When the VR application is started, SLAM positioning takes a certain amount of time, and the positioning posture data is in an unstable state when it is just started, which is also an important factor in the display delay of VR applications. Therefore, before SLAM obtains stable posture data, the transition animation can continue to be displayed.

[0082] like Figure 5 As shown in the figure, the startup process of the pose data obtained by SLAM mainly includes the following steps:

[0083] S202_6: Determine whether the positioning thread is started, if so, execute S202_7, if not, end.

[0084] In actual applications, when the VR device is turned on, it creates a positioning thread to start the SLAM service, and the screen on and off operation is used to control the status of the SLAM service. When the display screen is on, the SLAM service is in the resume state (resume), and when the display screen is off, the SLAM service is in the pause state (pause).

[0085] S202_7: Call the positioning thread to obtain the multi-eye images synchronously collected by the multi-eye cameras, and obtain the motion data collected by the IMU of the VR device itself.

[0086] When the display screen is lit, that is, during the transition animation, after the VR mode is started, the multi-eye images collected by the synchronized multi-eye cameras can provide a good visual data source for the SLAM service, combined with the motion data collected by its own IMU, thereby realizing the joint positioning of vision and inertial navigation.

[0087] like Figure 6 As shown in the figure, it is a schematic diagram of the SLAM algorithm framework, which mainly includes data acquisition, front-end, back-end, loop detection and mapping. Among them, the front-end extracts features from the collected multi-eye images and uses the PnP algorithm to calculate the relative posture of the camera; loop detection mainly eliminates the accumulated error through the similarity between images; the back-end globally optimizes the posture according to the posture calculated by the front-end, the motion data collected by the IMU and the loop detection information to eliminate the influence of noise. This optimization process is mathematically modeled as a maximum a posteriori probability estimation problem to solve it. The mainstream methods are filter methods represented by Kalman filtering and nonlinear optimization methods; mapping establishes motion trajectories according to the estimated posture to form a map.

[0088] S202_8: Determine the initial 6DoF posture of the VR device using the PnP algorithm based on the multi-eye images.

[0089] In the specific implementation, feature points are extracted from multi-view images, and the PnP algorithm is used to determine the initial 6DoF pose of the VR device in combination with the pre-calibrated camera internal parameters.

[0090] S202_9: Determine whether the 6Dof posture is stable. If so, end. If not, execute S202_10.

[0091] Among them, the initial 6DoF pose is affected by the calibration results and feature points, which may lead to unstable calculation results.

[0092] S202_10: Perform global optimization on the initial 6DoF pose according to the motion data to obtain the target 6DoF pose.

[0093] In one example, the Kalman filter algorithm can be used to filter out the unstable initial 6Dof pose at the beginning of the SLAM algorithm, so as to obtain a stable target 6Dof pose. Specifically, the motion equation is established based on the motion data collected by the IMU, and the predicted values ​​of the mean and variance are obtained. Then, the Kalman gain is calculated by combining the observation equation established based on the visual positioning result and the predicted variance, and the Kalman gain is used to update the mean and variance to obtain the globally optimized target 6Dof pose.

[0094] In actual applications, when a VR application is started, that is, during the display of the transition animation, the rendering thread in the software development kit (SDK) of the VR application is started. After the rendering thread is started, it starts to wait for the image frame of the VR application. If the image frame is obtained, the offline cache (buffer) is updated immediately. The rendering thread starts quickly, and its execution time is less than the execution time of the second thread and the positioning thread respectively. However, the rendering thread's display of the VR application's image frame depends on the startup state of the VR mode. Therefore, when displaying the transition animation, the time consumption of the rendering thread also needs to be considered.

[0095] S203: When the image frame is acquired successfully, the synchronization of the multi-camera is completed, and the pose data is stable, the loading flag of the transition animation is set to the off state.

[0096] After the transition animation is displayed, the first process starts to wait. When the rendering thread obtains the image frame of the VR application, the VR mode is successfully started (that is, the multi-camera synchronization is completed and the data preparation is successful), and the SLAM posture data is accurate and stable, the VR application is started and the transition animation can be ended. At this time, the loading flag of the transition animation is set to the off state.

[0097] S204: Stop displaying the transition animation according to the closed state of the loading mark, and display the image frame.

[0098] When the loading flag of the transition animation is set to the off state, the first process immediately stops loading the transition animation and displays the image frame of the VR application normally.

[0099] See also Figure 7 , is the display process of the image frame, which mainly includes the following steps:

[0100] S2041: Determine whether the perspective mode of the VR device is turned on, if not, execute S2042, if so, execute S2043.

[0101] When the perspective mode is turned on, the user can watch the virtual image while also seeing the real image around him through the multi-eye camera on the VR device; when the perspective mode is turned off, the user can only see the virtual image.

[0102] S2042: Calling a rendering thread to render image frames of the VR application and display them on a binocular screen.

[0103] When the perspective mode is turned off, the rendering thread draws the acquired image frames through OpenGL and displays them to the user through the binocular screen to obtain an immersive experience.

[0104] like Fig. 8A , which is a schematic diagram of the display effect of the image frame of the VR application when the perspective mode is turned off.

[0105] S2043: Calling a rendering thread to render an image frame of a VR application on a first layer, and an image frame of a multi-eye camera on a second layer, and superimposing the two image frames on a binocular screen through the first layer and the second layer.

[0106] When the perspective mode is turned on, since it is necessary to display the image frames of the VR application and the image frames of the multi-eye camera at the same time, two superimposed layers can be created. The image frames of the VR application are rendered on the first layer, and the image frames of the multi-eye camera are rendered on the second layer. The image frames of the two layers are superimposed and displayed on the binocular screen, thereby achieving a display that combines virtual and real.

[0107] like Figure 8B , which is a schematic diagram of the display effect of the image frames of the VR application and the image frames of the multi-eye camera when the perspective mode is turned on.

[0108] In the process of starting a VR application in the embodiment of the present application, a rendering thread for acquiring image frames, a second thread for starting the VR mode, and a positioning thread for determining the 6Dof posture are executed concurrently, thereby improving the startup speed of the VR application.

[0109] See also Fig. 9 , which is a complete display process of image frames from transition animation to VR application provided in the embodiment of the present application, mainly includes the following steps:

[0110] S901: When the VR device is picked up or the power button is pressed, the display screen of the VR device lights up.

[0111] S902: In response to the display screen lighting operation, start the first process to load the transition animation and perform binocular display. At the same time, start the rendering thread in the SDK of the VR application, start the SurfaceFlinger thread to set the power function mode of the screen, and start the positioning thread of the SLAM algorithm.

[0112] S903: The first process reads the state of the loading flag of the transition animation.

[0113] S904: Create a second thread in the power function mode to start the VR mode.

[0114] S905: The second thread starts the multi-eye camera and synchronizes the multi-eye camera.

[0115] S906: Determine whether the multi-camera synchronization is successful. If so, execute S907; if not, return to S905.

[0116] S907: VR mode startup is complete, and the second thread notifies the positioning thread that positioning data preparation is complete and SLAM positioning can be performed.

[0117] S908: The positioning thread calculates the initial 6Dof pose of the VR device using the PnP algorithm based on the multi-eye images collected by the multi-eye camera.

[0118] S909: The positioning thread combines the motion data collected by the IMU and uses the Kalman filter method to globally optimize the initial 6Dof pose to obtain an accurate and stable target 6Dof pose.

[0119] S910: The rendering thread obtains an image frame of the VR application.

[0120] S911: Determine whether the image frame is acquired successfully, whether the VR mode is started, and whether the SLAM positioning result is stable. If all three conditions are met, execute S912, otherwise, return to S902.

[0121] S912: Set the loading flag of the transition animation to a closed state.

[0122] S913: After the first process reads that the loading flag is in the closed state, the loading transition animation is immediately stopped.

[0123] S914: The rendering thread renders the image frames of the VR application and displays them on a binocular screen.

[0124] In the embodiment of the present application, when the user puts on the VR device or presses the power button, the display screen of the VR device is lit, and the first process is immediately started to load the transition animation and perform binocular screen display through OpenGL. At this time, the background of the VR device starts image acquisition, VR mode startup and SLAM positioning tasks. When the image frame is successfully acquired, the VR mode startup is completed, and the SLAM positioning result is stable, the display of the global transition animation is terminated, and the acquired image frame is immediately displayed, such as Fig.10 As shown. By displaying the transition animation immediately after the display screen is turned on, the gray screen phenomenon is reduced, and the transition animation can prompt the user that the VR application is starting, thereby avoiding the problem of fixed screen caused by frequent screen turning on and off when the VR screen cannot be seen, and improving the startup speed of the application. On the other hand, the display of the transition animation is globally controllable. When the three conditions of successful image frame acquisition, VR mode startup, and stable SLAM positioning results are met, the display of the transition screen ends, thereby ensuring that the application is successfully started and the VR screen can be displayed normally.

[0125] Based on the same technical concept, an embodiment of the present application provides a virtual reality device, which can implement the steps of the above-mentioned VR image display method and achieve the same technical effect.

[0126] See also Fig.11 The virtual reality device includes a processor 1101, a memory 1102, a display screen 1103 and a multi-eye camera 1104, wherein the multi-eye camera 1104, the display screen 1103, the memory 1102 and the processor 1101 are connected via a bus 1105;

[0127] The memory 1102 stores a computer program, and the processor 1101 performs the following operations according to the computer program:

[0128] In response to the lighting operation of the display screen 1103, the first process is started to load the transition animation and perform binocular display, wherein the transition animation is used to prompt that the virtual reality device is starting and the screen should not be turned on and off frequently;

[0129] During the transition animation display, the rendering thread of the VR application is started to obtain image frames, and the second thread of the VR mode is started to synchronize the multi-eye camera 1104, and the positioning thread is started to obtain the posture data of the virtual reality device;

[0130] When the image frame is acquired successfully, the synchronization of the multi-camera is completed, and the posture data is stable, setting the loading flag of the transition animation to an off state;

[0131] The transition animation is stopped from being displayed according to the closed state of the loading mark, and the image frame is displayed through the display screen 1103 .

[0132] Optionally, the processor 1101 starts the second thread of the VR mode to synchronize the multi-eye camera 1104, and the specific operation is:

[0133] Setting the power function mode of the screen through the SurfaceFlinger thread, creating a second thread of the VR mode in the power function mode, wherein the initial state of the second thread is a disabled state to indicate that data of the VR mode is unavailable;

[0134] Calling the second thread to start the multi-eye camera on the virtual reality device and synchronize the multi-eye camera 1104;

[0135] When the synchronization of the multi-camera 1104 is completed, the target state of the second thread is set to the enabled state to notify the positioning thread that the positioning data is ready.

[0136] Optionally, the virtual reality device further includes an IMU 1106. When the target state of the second thread is the start state, the processor 1101 starts the positioning thread to obtain the position and posture data of the virtual reality device. The specific operation is:

[0137] Call the positioning thread to obtain the multi-eye images synchronously collected by the multi-eye camera 1104, and obtain the motion data collected by the IMU 1106;

[0138] According to the multi-eye image, an initial 6-DOF posture of the virtual reality device is obtained by using a PnP algorithm;

[0139] According to the motion data, the initial 6-DOF posture is globally optimized to obtain the target 6-DOF posture.

[0140] Optionally, the processor 1101 displays the image frame through the display screen, and the specific operation is:

[0141] When the perspective mode of the virtual reality device is turned off, calling the rendering thread to render the image frames of the VR application and perform binocular screen display;

[0142] When the perspective mode of the virtual reality device is turned on, the rendering thread is called to render the image frames of the VR application on the first layer, and to render the image frames of the multi-eye camera on the second layer, and the two image frames are superimposed and displayed on the binocular screen through the first layer and the second layer.

[0143] Optionally, the rendering thread, the second thread and the positioning thread are executed concurrently, and the execution time of the rendering thread is respectively shorter than the execution time of the second thread and the positioning thread.

[0144] in, Fig.11 The memory 1102 in the embodiment may be a volatile memory, such as a random-access memory (RAM); the memory 1102 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1102 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be a combination of the above memories; the processor 1101 may include one or more central processing units (CPU) or a digital processing unit, etc.

[0145] It should be noted that Fig.11 This is just an example, and provides the necessary hardware for a virtual reality device to execute the steps of a VR image display method provided in an embodiment of the present application. If not shown, the virtual reality device may also include hardware of a conventional VR device such as a speaker, a pickup, function buttons, a power supply, and a communication interface.

[0146] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete a VR image display method in the aforementioned embodiment.

[0147] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute a VR image display method in the aforementioned embodiment.

[0148] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0149] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0152] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A VR image display method, characterized in that: Applied to virtual reality devices, including: In response to the display screen lighting operation, the first process is started to load the transition animation and perform binocular display, wherein the transition animation is used to prompt that the VR application is starting and the screen should not be turned on and off frequently; During the transition animation display, the rendering thread of the VR application is started to obtain image frames, and the second thread of the VR mode is started to synchronize the multi-eye camera, and the positioning thread is started to obtain the posture data of the virtual reality device; When the image frame is acquired successfully, the synchronization of the multi-camera is completed, and the posture data is stable, setting the loading flag of the transition animation to an off state; The transition animation is stopped from being displayed according to the closed state of the loading mark, and the image frame is displayed.

2. The method according to claim 1, characterized in that The second thread of starting the VR mode to synchronize the multi-eye camera includes: Setting the power function mode of the screen through the SurfaceFlinger thread, creating a second thread of the VR mode in the power function mode, wherein the initial state of the second thread is a disabled state to indicate that data of the VR mode is unavailable; Calling the second thread to start the multi-eye camera on the virtual reality device and synchronize the multi-eye camera; When the multi-camera synchronization is completed, the target state of the second thread is set to an enabled state to notify the positioning thread that positioning data preparation is complete.

3. The method according to claim 2, characterized in that When the target state of the second thread is the start state, the starting positioning thread to obtain the position and posture data of the virtual reality device includes: Calling the positioning thread to obtain the multi-eye images synchronously acquired by the multi-eye camera, and acquiring the motion data acquired by the IMU of the virtual reality device itself; According to the multi-eye image, an initial 6-DOF posture of the virtual reality device is obtained by using a PnP algorithm; According to the motion data, the initial 6-DOF posture is globally optimized to obtain the target 6-DOF posture.

4. The method according to claim 1, characterized in that The displaying of the image frame comprises: When the perspective mode of the virtual reality device is turned off, calling the rendering thread to render the image frames of the VR application and perform binocular screen display; When the perspective mode of the virtual reality device is turned on, the rendering thread is called to render the image frames of the VR application on the first layer, and to render the image frames of the multi-eye camera on the second layer, and the two image frames are superimposed and displayed on the binocular screen through the first layer and the second layer.

5. The method according to any one of claims 1 to 4, characterized in that The rendering thread, the second thread and the positioning thread are executed concurrently, and the execution time of the rendering thread is shorter than the execution time of the second thread and the positioning thread respectively.

6. A virtual reality device, characterized in that: It includes a processor, a memory, a display screen and a multi-eye camera, wherein the multi-eye camera, the display screen, the memory and the processor are connected via a bus; The memory stores a computer program, and the processor performs the following operations according to the computer program: In response to the lighting operation of the display screen, starting the first process to load the transition animation and perform binocular display, wherein the transition animation is used to prompt that the virtual reality device is starting and the user should not frequently perform the screen lighting and off operations; During the transition animation display, a rendering thread of the VR application is started to obtain image frames, and a second thread of the VR mode is started to synchronize the multi-eye camera, and a positioning thread is started to obtain the posture data of the virtual reality device; When the image frame is acquired successfully, the synchronization of the multi-camera is completed, and the posture data is stable, setting the loading flag of the transition animation to an off state; The transition animation is stopped from being displayed according to the closed state of the loading mark, and the image frame is displayed through the display screen.

7. The virtual reality device according to claim 6, characterized in that: The processor starts the second thread of the VR mode to synchronize the multi-eye cameras, and the specific operation is: The power function mode of the screen is set through the SurfaceFlinger thread, and a second thread of the VR mode is created in the power function mode, wherein the initial state of the second thread is a disabled state to indicate that data of the VR mode is unavailable; the second thread is called to start the multi-eye camera on the virtual reality device, and synchronize the multi-eye camera; When the multi-camera synchronization is completed, the target state of the second thread is set to an enabled state to notify the positioning thread that positioning data preparation is complete.

8. The virtual reality device according to claim 6, characterized in that: When the target state of the second thread is the start state, the processor starts the positioning thread to obtain the position and posture data of the virtual reality device, and the specific operation is: Calling the positioning thread to obtain the multi-eye images synchronously acquired by the multi-eye camera, and acquiring the motion data acquired by the IMU of the virtual reality device itself; According to the multi-eye image, an initial 6-DOF posture of the virtual reality device is obtained by using a PnP algorithm; According to the motion data, the initial 6-DOF posture is globally optimized to obtain the target 6-DOF posture.

9. The virtual reality device according to claim 6, characterized in that: The processor displays the image frame through the display screen, and the specific operation is: When the perspective mode of the virtual reality device is turned off, calling the rendering thread to render the image frames of the VR application and perform binocular screen display; When the perspective mode of the virtual reality device is turned on, the rendering thread is called to render the image frames of the VR application on the first layer, and to render the image frames of the multi-eye camera on the second layer, and the two image frames are superimposed and displayed on the binocular screen through the first layer and the second layer.

10. The virtual reality device according to any one of claims 6 to 9, characterized in that: The rendering thread, the second thread and the positioning thread are executed concurrently, and the execution time of the rendering thread is shorter than the execution time of the second thread and the positioning thread respectively.

Images