VR image display method and virtual reality device

Abstract

The application relates to the VR technical field and provides a VR image display method and a virtual reality device, which are used for improving the display speed of a VR image. When a display screen is lighted, a first process is started to load a transition animation and perform binocular screen display through OpenGL. In order to ensure that the transition animation is globally controllable, corresponding threads are also started to perform image acquisition, VR mode starting and SLAM positioning tasks when the display screen is lighted. When the image frame acquisition is successful, the VR mode starting is completed and the SLAM positioning result is stable, the display of the global transition animation is ended, and the acquired image frame is immediately displayed. The display of the transition animation reduces the gray screen phenomenon, and the transition animation can prompt the user that the VR application is being started, thereby avoiding the screen fixation problem caused by the frequent on-off screen operation when the VR picture cannot be seen, improving the starting speed of the application, quickly displaying the VR image, and enabling the user to obtain better VR experience.

Description

Technical Field

[0001] This application relates to the field of virtual reality (VR) technology, and provides a VR image display method and a virtual reality device. Background Technology

[0002] When using VR devices, users either remove the VR device or manually turn the screen on and off using the power button to calibrate the screen. Ideally, a stable and accurate VR image should be displayed immediately upon putting on the VR device. However, in reality, there is a latency of approximately 800ms-1200ms in VR image display. Therefore, users may see a gray screen for a period of time before seeing the normally displayed VR image. Consequently, some users may press the power button again when the screen is grayed out to re-enable the VR image, thus increasing the VR device's startup time and resulting in a poor user experience. Summary of the Invention

[0003] This application provides a VR image display method and a virtual reality device to improve the display speed of VR images.

[0004] On one hand, embodiments of this application provide a VR image display method, applied to a VR device, including:

[0005] In response to the screen being turned on, the first process is started to load the transition animation and perform binocular display. The transition animation is used to prompt that the VR application is starting up and that the screen should not be turned on and off frequently.

[0006] During the transition animation display, the rendering thread of the VR application is started to obtain image frames, and a second thread of VR mode is started to synchronize the multi-view camera, and a positioning thread is started to obtain the pose data of the virtual reality device;

[0007] When the image frame is successfully acquired, the multi-view camera synchronization is completed, and the pose data is stable, the loading flag of the transition animation is set to the off state.

[0008] The transition animation is stopped and the image frame is displayed based on the off state of the loading flag.

[0009] Optionally, the second thread for activating VR mode to synchronize multiple cameras includes:

[0010] The power function mode of the screen is set through the SurfaceFlinger thread, and a second thread for VR mode is created in the power function mode, wherein the initial state of the second thread is disabled to indicate that data for VR mode is unavailable.

[0011] The second thread is invoked to activate the multi-view camera on the virtual reality device and synchronize the multi-view camera.

[0012] When the multi-camera synchronization is completed, the target state of the second thread is set to the enabled state to notify the positioning thread that the positioning data preparation is complete.

[0013] Optionally, when the target state of the second thread is the start state, starting the positioning thread to obtain the pose data of the virtual reality device includes:

[0014] The positioning thread is invoked to obtain multi-view images synchronously acquired by the multi-view camera, and motion data acquired by the IMU of the virtual reality device itself is also acquired.

[0015] Based on the multi-view image, the PnP algorithm is used to obtain the initial 6-DOF pose of the virtual reality device;

[0016] Based on the motion data, the initial 6-DOF pose is globally optimized to obtain the target 6-DOF pose.

[0017] Optionally, displaying the image frame includes:

[0018] When the perspective mode of the virtual reality device is turned off, the rendering thread is invoked to render the image frames of the VR application and display them on the binocular screen.

[0019] When the perspective mode of the virtual reality device is enabled, the rendering thread is invoked to render the image frames of the VR application on the first layer, and the image frames of the multi-view camera are rendered on the second layer. The two image frames are then superimposed and displayed on the binocular screen through the first layer and the second layer.

[0020] Optionally, the rendering thread, the second thread, and the positioning thread are executed concurrently, and the execution time of the rendering thread is less than the execution time of the second thread and the positioning thread, respectively.

[0021] On the other hand, embodiments of this application provide a virtual reality device, including a processor, a memory, a display screen, and a multi-view camera, wherein the multi-view camera, the display screen, the memory, and the processor are connected via a bus;

[0022] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0023] In response to the screen being turned on, a first process is initiated to load a transition animation and perform binocular display. The transition animation serves as a reminder that the virtual reality device is starting up and that frequent screen on / off operations should be avoided.

[0024] During the transition animation display, the rendering thread of the VR application is started to obtain image frames, and the second thread of the VR mode is started to synchronize the multi-view camera, and the positioning thread is started to obtain the pose data of the virtual reality device.

[0025] When the image frame is successfully acquired, the multi-view camera synchronization is completed, and the pose data is stable, the loading flag of the transition animation is set to the off state.

[0026] The transition animation is stopped based on the off state of the loading flag, and the image frame is displayed on the screen.

[0027] Optionally, the processor initiates a second thread in VR mode to synchronize multiple cameras, specifically as follows:

[0028] The power function mode of the screen is set through the SurfaceFlinger thread, and a second thread for VR mode is created in the power function mode, wherein the initial state of the second thread is disabled to indicate that data for VR mode is unavailable.

[0029] The second thread is invoked to activate the multi-view camera on the virtual reality device and synchronize the multi-view camera.

[0030] When the multi-camera synchronization is completed, the target state of the second thread is set to the enabled state to notify the positioning thread that the positioning data preparation is complete.

[0031] Optionally, when the target state of the second thread is the start state, the processor starts the positioning thread to obtain the pose data of the virtual reality device, specifically as follows:

[0032] The positioning thread is invoked to obtain multi-view images synchronously acquired by the multi-view camera, and motion data acquired by the IMU of the virtual reality device itself is also acquired.

[0033] Based on the multi-view image, the PnP algorithm is used to obtain the initial 6-DOF pose of the virtual reality device;

[0034] Based on the motion data, the initial 6-DOF pose is globally optimized to obtain the target 6-DOF pose.

[0035] Optionally, the processor displays the image frame via the display screen, specifically by:

[0036] When the perspective mode of the virtual reality device is turned off, the rendering thread is invoked to render the image frames of the VR application and display them on the binocular screen.

[0037] When the perspective mode of the virtual reality device is enabled, the rendering thread is invoked to render the image frames of the VR application on the first layer, and the image frames of the multi-view camera are rendered on the second layer. The two image frames are then superimposed and displayed on the binocular screen through the first layer and the second layer.

[0038] Optionally, the rendering thread, the second thread, and the positioning thread are executed concurrently, and the execution time of the rendering thread is less than the execution time of the second thread and the positioning thread, respectively.

[0039] On the other hand, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the steps of a VR image display method provided in embodiments of this application.

[0040] The beneficial effects of the VR screen display method provided in this application embodiment are as follows:

[0041] When the display is turned on, a transition animation is immediately loaded and displayed in both eyes, thus notifying the user that the VR application is starting up. This avoids frequent screen on / off operations and improves application startup efficiency. At the same time, during the display of the transition animation, the screen turning on will trigger the start of the rendering thread for acquiring image frames, the second thread for VR mode, and the positioning thread for acquiring pose data. The execution results of these three threads serve as the conditions for the end of the transition animation, thereby globally controlling the display time of the transition animation and ensuring the normal display of image frames.

[0042] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A schematic diagram of a VR device provided in an embodiment of this application;

[0045] Figure 2 A flowchart of a VR image display method provided in an embodiment of this application;

[0046] Figure 3 This is a diagram illustrating the transition animation effect provided in an embodiment of this application.

[0047] Figure 4 A flowchart illustrating the method for activating VR mode provided in this application embodiment;

[0048] Figure 5 A flowchart of the SLAM localization method provided in the embodiments of this application;

[0049] Figure 6 This is a diagram of the SLAM algorithm architecture provided in the embodiments of this application;

[0050] Figure 7 A flowchart of a VR image display method provided in an embodiment of this application;

[0051] Figure 8A The image shows the display effect of the VR image when the perspective mode is turned off, as provided in the embodiments of this application.

[0052] Figure 8B A diagram illustrating the display effect of a VR image when perspective mode is enabled, as provided in an embodiment of this application.

[0053] Figure 9 A flowchart illustrating a method for switching from a transition animation to a VR image, as provided in an embodiment of this application.

[0054] Figure 10 This is a schematic diagram illustrating the process of switching from a transition animation to a VR image, as provided in an embodiment of this application.

[0055] Figure 11 A structural diagram of a virtual reality device provided in an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0057] VR devices (such as standalone VR headsets) are a new type of mobile display device that can provide an immersive experience and are now widely used in various fields. Currently, VR devices include 3-DOF (Degree of Freedom, DoF) devices and 6DoF devices. Since 3DoF only contains translation information, while 6DoF contains both translation and rotation information, it offers higher control precision. Therefore, 6DoF devices are more commonly used in the market.

[0058] Typically, VR devices have multiple cameras mounted externally. Figure 1 The area is circled in the middle and contains an Inertial Measurement Unit (IMU). Therefore, Simultaneous Localization and Mapping (SLAM) technology can be used to provide accurate pose data (i.e., translation and rotation information) for the 6DoF device.

[0059] Currently, VR devices experience latency when displaying VR images. As a result, users may see a gray screen for a period of time before the VR image is displayed normally. Some users may press the power button again when the screen is gray to try to re-enable the VR image. However, frequent screen on / off operations can cause problems such as unstable SLAM positioning, abnormal VR modes, and increased VR application startup time, thereby increasing the probability of gray screen or screen freeze issues and affecting the user's immersive experience.

[0060] In view of this, this application provides a VR image display method. When the VR device's display screen is turned on, this method immediately loads a prompt indicating that the VR device is starting a transition animation and performing binocular display. This avoids the frequent screen-on / off operations that prevent users from seeing the VR image in real time due to a gray screen phase, effectively improving the startup efficiency of the VR device. The display of the transition animation cannot be simply implemented using the screen's lifecycle. This method globally controls the end of the transition animation from three processes: SLAM pose data preparation, VR mode startup, and image rendering. The VR application's image frames are displayed immediately after the transition animation ends, ensuring normal image frame display and reducing the probability of gray screen problems, thus enhancing the user's immersive experience.

[0061] See Figure 2 Here is a flowchart of a VR image display method provided in this application embodiment. This process is executed by a VR device and mainly includes the following steps:

[0062] S201: In response to the display being turned on, the first process is started to load the transition animation and perform binocular display. The transition animation is used to remind the VR device that it is starting up and that the screen should not be turned on and off frequently.

[0063] In practical applications, the VR device's display lights up after the user puts on the VR headset or presses the power button. Considering the delay in VR image display, to avoid users frequently turning the screen on and off after no longer seeing the VR image, in one example, immediately after the display lights up, a transition animation is loaded and displayed in binoculars using OpenGL. This serves as a notification to the user that the VR device is starting up and should not be frequently turned on and off. The effect of the transition animation in binocular display is as follows: Figure 3 As shown.

[0064] S202: During the transition animation display, start the rendering thread of the VR application to obtain image frames, start the second thread of VR mode to synchronize the multi-view camera, and start the positioning thread to obtain the pose data of the virtual reality device.

[0065] Generally, regular animations can be displayed globally during the screen's on-time lifecycle. However, transition animations are only used as prompts for temporary display. Once the VR application is launched, the VR application's images still need to be displayed normally. Therefore, they cannot be simply controlled using the screen's on-time lifecycle. Furthermore, the display of transition animations and VR application images involves cross-process communication, and the display of transition animations needs to be globally controllable.

[0066] In one example, during the startup process of a VR device, i.e. the display of a transition animation, tasks such as restoring the VR image rendering thread, preparing VR mode data, and SLAM positioning are performed in the background. The execution time of these tasks varies, so it is necessary to accurately determine the timing of the end of the transition animation to avoid increasing useless event waiting time, thereby improving the user's VR experience.

[0067] In practical applications, data preparation for VR mode takes a certain amount of time. Ideally, VR mode takes about 500ms-80ms from starting to starting, which is one of the main time-consuming tasks in starting VR applications. In cases of abnormal startup caused by frequent screen-on and off operations by users, the startup time of VR mode cannot be accurately calculated.

[0068] like Figure 4 As shown, the startup process for preparing data for VR mode mainly includes the following steps:

[0069] S202_1: Determine if the display screen is lit. If yes, execute S202_2; otherwise, end.

[0070] When the VR device's display is turned on, the screen driver triggers the SurfaceFlinger service, and VR mode is started.

[0071] S202_2: Sets the screen's power function mode through the SurfaceFlinger thread, and creates a second thread for VR mode within the power function mode.

[0072] After VR mode is started, the power mode of the screen is set through the SurfaceFlinger thread, changing the power mode of the screen from the off state (HWC_POWER_MODE_OFF) to the on state (HWC_POWER_MODE_NORMAL), and a second thread for VR mode is created in the power mode.

[0073] The second thread is initially in a stopped state, indicating that the VR mode data is not ready and therefore unavailable. The second thread needs to prepare the VR mode data to start the VR mode.

[0074] S202_3: Call the second thread to start the multi-view camera on the VR device and synchronize the multi-view camera.

[0075] VR devices are equipped with multi-view cameras (usually four-view or two-view) to provide data sources for SLAM positioning. To ensure the accuracy and stability of SLAM positioning, the second thread also needs to synchronize the multi-view cameras after they are turned on.

[0076] S202_4: Determine if synchronization is complete. If yes, execute S202_5; otherwise, end.

[0077] S202_5: Set the target state of the second thread to enabled to notify the positioning thread that the positioning data preparation is complete.

[0078] When the multi-camera synchronization is complete, it indicates that the VR mode has been successfully started. At this time, the target status of the second thread can be set to the enabled state to notify the positioning thread that the positioning data is ready and SLAM positioning can be performed. The second thread task is then complete.

[0079] During the VR mode startup process, the startup and synchronization of multiple cameras consume a significant amount of time. Therefore, adopting an efficient synchronization method has a significant impact on improving the startup speed of VR applications.

[0080] It should be noted that the embodiments of this application do not impose restrictive requirements on the synchronization method of multi-view cameras. For example, a feature point matching synchronization method or a timestamp synchronization method can be used.

[0081] To ensure the effectiveness of SLAM positioning in VR devices, VR devices are typically calibrated at the factory and retain the calibrated parameters. However, SLAM positioning takes time when a VR application starts up, and the initial pose data is unstable. This is a significant factor contributing to display latency in VR applications. Therefore, transition animations can continue to be displayed until SLAM obtains stable pose data.

[0082] like Figure 5 The diagram shows the startup process for SLAM-acquired pose data, which mainly includes the following steps:

[0083] S202_6: Determine if the positioning thread has started. If yes, execute S202_7; otherwise, terminate.

[0084] In practical applications, when a VR device is powered on, a positioning thread is created to start the SLAM service. The screen on / off operation is used to control the state of the SLAM service. When the screen is on, the SLAM service is in a resumed state, and when the screen is off, the SLAM service is in a paused state.

[0085] S202_7: Call the positioning thread to obtain multi-view images synchronously acquired by the multi-view camera, and obtain motion data acquired by the VR device's own IMU.

[0086] When the display is lit up, during the transition animation, after the VR mode is started, the multi-view images captured by the synchronized multi-view cameras can provide a good visual data source for SLAM services. Combined with the motion data collected by its own IMU, it can achieve joint positioning of vision and inertial navigation.

[0087] like Figure 6 The diagram shows the SLAM algorithm framework, which mainly includes data acquisition, front-end, back-end, loop closure detection, and mapping. The front-end extracts features from the acquired multi-view images and uses the PnP algorithm to calculate the relative pose of the cameras. Loop closure detection primarily eliminates accumulated errors through image similarity. The back-end performs global pose optimization based on the pose calculated by the front-end, motion data acquired by the IMU, and loop closure detection information to eliminate noise. This optimization process is mathematically modeled as a maximum a posteriori probability estimation problem, solved using filter methods such as Kalman filtering and nonlinear optimization methods. Mapping establishes motion trajectories based on the estimated pose, forming a map.

[0088] S202_8: Based on the multi-view image, the PnP algorithm is used to determine the initial 6DoF pose of the VR device.

[0089] In practice, feature points are extracted from the multi-view image, and the PnP algorithm is used to determine the initial 6DoF pose of the VR device, combined with the pre-calibrated camera intra-parameters.

[0090] S202_9: Determine if the 6DOF pose is stable. If yes, end; otherwise, execute S202_10.

[0091] The initial 6DoF pose is affected by the calibration results and feature points, which may lead to unstable calculation results.

[0092] S202_10: Based on the motion data, perform global optimization on the initial 6DoF pose to obtain the target 6DoF pose.

[0093] In one example, a Kalman filter algorithm can be used to filter out the unstable initial 6DoF pose of the SLAM algorithm at the beginning, thereby obtaining a stable target 6DoF pose. Specifically, motion equations are established based on motion data acquired by the IMU to obtain predicted values ​​of the mean and variance. Then, the Kalman gain is calculated by combining the observation equation established based on the visual localization results and the predicted variance. The Kalman gain is then used to update the mean and variance to obtain the globally optimized target 6DoF pose.

[0094] In practical applications, when a VR application starts up, during the display of transition animations, the rendering thread in the VR application's Software Development Kit (SDK) is launched. After starting, the rendering thread waits for image frames from the VR application. If an image frame is acquired, the offline buffer is updated immediately. The rendering thread starts up relatively quickly, and its execution time is shorter than that of the second thread and the positioning thread. However, the rendering thread's display of VR application image frames depends on the VR mode's startup state; therefore, the rendering thread's time consumption must also be considered when displaying transition animations.

[0095] S203: When the image frame is successfully acquired, the multi-camera synchronization is completed, and the pose data is stable, set the loading flag of the transition animation to the off state.

[0096] After the transition animation is displayed, the first process begins to wait. When the rendering thread acquires the image frames of the VR application, the VR mode is successfully started (i.e., multi-camera synchronization is completed and data preparation is successful), and the SLAM pose data is accurate and stable, it indicates that the VR application has started successfully and the display of the transition animation can be ended. At this time, the loading flag of the transition animation is set to the off state.

[0097] S204: Stop displaying the transition animation based on the off state of the load flag, and display the image frame.

[0098] When the transition animation loading flag is set to off, the first process immediately stops loading the transition animation and displays the image frames of the VR application normally.

[0099] See Figure 7 The process of displaying an image frame mainly includes the following steps:

[0100] S2041: Determine whether the VR device's perspective mode is enabled. If not, proceed to S2042; if yes, proceed to S2043.

[0101] When perspective mode is enabled, users can see the real surroundings through the multi-view camera on the VR device while viewing the virtual scene; when perspective mode is disabled, users can only see the virtual scene.

[0102] S2042: Call the rendering thread to render image frames of the VR application and display them on the binocular screen.

[0103] When perspective mode is off, the rendering thread draws the image frames obtained through OpenGL and displays them to the user through the binocular screen to achieve an immersive experience.

[0104] like Figure 8A The image shown is a schematic diagram illustrating the display effect of image frames in a VR application when perspective mode is turned off.

[0105] S2043: Call the rendering thread to render the image frames of the VR application on the first layer, and render the image frames of the multi-view camera on the second layer, and display the two image frames on the binocular screen by overlaying them through the first and second layers.

[0106] When perspective mode is enabled, since it is necessary to display image frames from both the VR application and the multi-view camera simultaneously, two overlapping layers can be created. The VR application's image frames are rendered on the first layer, and the multi-view camera's image frames are rendered on the second layer. The image frames from both layers are then overlaid on the binocular screen, thus achieving a display that combines virtual and real elements.

[0107] like Figure 8B The image shown is a schematic diagram illustrating the display effect of image frames from VR applications and multi-view cameras when perspective mode is enabled.

[0108] In this embodiment of the application, during the startup of a VR application, the rendering thread for acquiring image frames, the second thread for starting the VR mode, and the positioning thread for determining the 6DOF pose are executed concurrently, thereby improving the startup speed of the VR application.

[0109] See Figure 9 The complete display process for switching from a transition animation to an image frame in a VR application, as provided in this embodiment, mainly includes the following steps:

[0110] S901: When the VR device is activated or the power button is pressed, the VR device's display lights up.

[0111] S902: In response to the display being turned on, the first process is started to load the transition animation and perform binocular display. At the same time, the rendering thread in the VR application's SDK is started, the SurfaceFlinger thread is started to set the power function mode of the screen, and the positioning thread of the SLAM algorithm is started.

[0112] S903: The status of the loading flag for the first process to read the transition animation.

[0113] S904: Create a second thread in the power function mode to start VR mode.

[0114] S905: The second thread starts the multi-view camera and synchronizes the multi-view camera.

[0115] S906: Determine whether the multi-camera synchronization is successful. If yes, proceed to S907; otherwise, return to S905.

[0116] S907: VR mode startup complete. The second thread notifies the positioning thread that positioning data preparation is complete and SLAM positioning can be performed.

[0117] S908: The localization thread calculates the initial 6DoF pose of the VR device using the PnP algorithm based on the multi-view images captured by the multi-view camera.

[0118] S909: The localization thread combines motion data acquired by the IMU and uses the Kalman filter method to globally optimize the initial 6DoF pose, thereby obtaining an accurate and stable target 6DoF pose.

[0119] S910: The rendering thread acquires image frames for the VR application.

[0120] S911: Determine whether the image frame was successfully acquired, whether the VR mode was successfully started, and whether the SLAM positioning result is stable. If all three conditions are met, execute S912; otherwise, return to S902.

[0121] S912: Set the transition animation loading flag to off.

[0122] S913: Once the first process reads that the loading flag is off, it immediately stops loading the transition animation.

[0123] S914: The rendering thread renders image frames for VR applications and displays them on a stereo screen.

[0124] In the embodiments of this application, when the user puts on the VR device or presses the power button, the VR device's display screen is lit up, and the first process immediately starts loading a transition animation and displays it on the binocular screen using OpenGL. At this time, the VR device's background begins image acquisition, VR mode startup, and SLAM positioning tasks. When the image frame acquisition is successful, the VR mode startup is complete, and the SLAM positioning result is stable, the display of the global transition animation ends, and the acquired image frame is immediately displayed, such as... Figure 10 As shown, by displaying a transition animation immediately after the screen lights up, gray screen issues are reduced. The transition animation also indicates to the user that the VR application is starting, thus avoiding screen freezes caused by frequent screen-on / off operations when the VR image is not visible, and improving application startup speed. Furthermore, the display of the transition animation is globally controllable. The transition animation only ends when three conditions are met: successful image frame acquisition, complete VR mode startup, and stable SLAM positioning results. This ensures successful application startup and proper display of the VR image.

[0125] Based on the same technical concept, this application provides a virtual reality device that can implement the steps of the above-described VR image display method and achieve the same technical effect.

[0126] See Figure 11 The virtual reality device includes a processor 1101, a memory 1102, a display screen 1103, and a multi-view camera 1104. The multi-view camera 1104, the display screen 1103, the memory 1102, and the processor 1101 are connected via a bus 1105.

[0127] The memory 1102 stores a computer program, and the processor 1101 performs the following operations according to the computer program:

[0128] In response to the lighting operation of the display screen 1103, the first process is started to load the transition animation and perform binocular display. The transition animation is used to prompt that the virtual reality device is starting up and that the screen should not be turned on and off frequently.

[0129] During the transition animation display, the rendering thread of the VR application is started to obtain image frames, and the second thread of the VR mode is started to synchronize the multi-view camera 1104, and the positioning thread is started to obtain the pose data of the virtual reality device.

[0130] When the image frame is successfully acquired, the multi-view camera synchronization is completed, and the pose data is stable, the loading flag of the transition animation is set to the off state.

[0131] The transition animation is stopped based on the off state of the loading flag, and the image frame is displayed on the display screen 1103.

[0132] Optionally, the processor 1101 initiates a second thread in VR mode to synchronize the multi-view camera 1104, specifically as follows:

[0133] The power function mode of the screen is set through the SurfaceFlinger thread, and a second thread for VR mode is created in the power function mode, wherein the initial state of the second thread is disabled to indicate that data for VR mode is unavailable.

[0134] The second thread is invoked to activate the multi-view camera on the virtual reality device and synchronize the multi-view camera 1104;

[0135] When the multi-view camera 1104 completes synchronization, the target state of the second thread is set to the enabled state to notify the positioning thread that the positioning data preparation is complete.

[0136] Optionally, the virtual reality device further includes an IMU 1106. When the target state of the second thread is the start state, the processor 1101 starts a positioning thread to obtain the pose data of the virtual reality device. The specific operation is as follows:

[0137] The positioning thread is invoked to obtain multi-view images synchronously acquired by the multi-view camera 1104, and motion data acquired by the IMU 1106;

[0138] Based on the multi-view image, the PnP algorithm is used to obtain the initial 6-DOF pose of the virtual reality device;

[0139] Based on the motion data, the initial 6-DOF pose is globally optimized to obtain the target 6-DOF pose.

[0140] Optionally, the processor 1101 displays the image frame through the display screen, specifically as follows:

[0141] When the perspective mode of the virtual reality device is turned off, the rendering thread is invoked to render the image frames of the VR application and display them on the binocular screen.

[0142] When the perspective mode of the virtual reality device is enabled, the rendering thread is invoked to render the image frames of the VR application on the first layer, and the image frames of the multi-view camera are rendered on the second layer. The two image frames are then superimposed and displayed on the binocular screen through the first layer and the second layer.

[0143] Optionally, the rendering thread, the second thread, and the positioning thread are executed concurrently, and the execution time of the rendering thread is less than the execution time of the second thread and the positioning thread, respectively.

[0144] in, Figure 11 The memory 1102 can be volatile memory, such as random-access memory (RAM); the memory 1102 can also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or the memory 1102 can be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory can be a combination of the above-described memories; the processor 1101 can include one or more central processing units (CPUs) or digital processing units, etc.

[0145] It should be noted that, Figure 11 This is merely an example illustrating the hardware necessary for a virtual reality device to perform the steps of a VR image display method provided in this application embodiment. Not shown, the virtual reality device may also include conventional VR device hardware such as speakers, microphones, function buttons, power supplies, and communication interfaces.

[0146] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform a VR image display method as described in the foregoing embodiments.

[0147] This application also provides a computer program product for storing a computer program that executes a VR image display method from the foregoing embodiments.

[0148] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0152] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A VR image display method, characterized in that, Applied to virtual reality devices, including: In response to the screen being turned on, the first process is started to load the transition animation and perform binocular display. The transition animation is used to prompt that the VR application is starting up and that the screen should not be turned on and off frequently. During the transition animation display, the rendering thread of the VR application is started to obtain image frames, and a second thread of VR mode is started to synchronize the multi-view camera, and a positioning thread is started to obtain the pose data of the virtual reality device; When the image frame is successfully acquired, the multi-view camera synchronization is completed, and the pose data is stable, the loading flag of the transition animation is set to the off state. The transition animation is stopped and the image frame is displayed based on the off state of the loading flag.

2. The method as described in claim 1, characterized in that, The second thread for activating VR mode to synchronize multiple cameras includes: The power function mode of the screen is set through the SurfaceFlinger thread, and a second thread for VR mode is created in the power function mode, wherein the initial state of the second thread is disabled to indicate that data for VR mode is unavailable. The second thread is invoked to activate the multi-view camera on the virtual reality device and synchronize the multi-view camera. When the multi-camera synchronization is completed, the target state of the second thread is set to the enabled state to notify the positioning thread that the positioning data preparation is complete.

3. The method as described in claim 2, characterized in that, When the target state of the second thread is the start state, starting the positioning thread to obtain the pose data of the virtual reality device includes: The positioning thread is invoked to obtain multi-view images synchronously acquired by the multi-view camera, and motion data acquired by the IMU of the virtual reality device itself is also acquired. Based on the multi-view image, the PnP algorithm is used to obtain the initial 6-DOF pose of the virtual reality device; Based on the motion data, the initial 6-DOF pose is globally optimized to obtain the target 6-DOF pose.

4. The method as described in claim 1, characterized in that, The display of the image frame includes: When the perspective mode of the virtual reality device is turned off, the rendering thread is invoked to render the image frames of the VR application and display them on the binocular screen. When the perspective mode of the virtual reality device is enabled, the rendering thread is invoked to render the image frames of the VR application on the first layer, and the image frames of the multi-view camera are rendered on the second layer. The two image frames are then superimposed and displayed on the binocular screen through the first layer and the second layer.

5. The method according to any one of claims 1-4, characterized in that, The rendering thread, the second thread, and the positioning thread are executed concurrently, and the execution time of the rendering thread is less than that of the second thread and the positioning thread, respectively.

6. A virtual reality device, characterized in that, It includes a processor, a memory, a display screen, and a multi-view camera, wherein the multi-view camera, the display screen, the memory, and the processor are connected via a bus; The memory stores a computer program, and the processor performs the following operations according to the computer program: In response to the screen being turned on, a first process is initiated to load a transition animation and perform binocular display. The transition animation serves as a reminder that the virtual reality device is starting up and that frequent screen on / off operations should be avoided. During the transition animation display, the rendering thread of the VR application is started to obtain image frames, and the second thread of the VR mode is started to synchronize the multi-view camera, and the positioning thread is started to obtain the pose data of the virtual reality device. When the image frame is successfully acquired, the multi-view camera synchronization is completed, and the pose data is stable, the loading flag of the transition animation is set to the off state. The transition animation is stopped based on the off state of the loading flag, and the image frame is displayed on the screen.

7. The virtual reality device as described in claim 6, characterized in that, The processor initiates a second thread in VR mode to synchronize multiple cameras, specifically as follows: The power function mode of the screen is set through the SurfaceFlinger thread. In the power function mode, a second thread for VR mode is created. The initial state of the second thread is disabled to indicate that data for VR mode is unavailable. The second thread is called to start the multi-view camera on the virtual reality device and synchronize the multi-view camera. When the multi-camera synchronization is completed, the target state of the second thread is set to the enabled state to notify the positioning thread that the positioning data preparation is complete.

8. The virtual reality device as described in claim 6, characterized in that, When the target state of the second thread is the start state, the processor starts the positioning thread to obtain the pose data of the virtual reality device. The specific operation is as follows: The positioning thread is invoked to obtain multi-view images synchronously acquired by the multi-view camera, and motion data acquired by the IMU of the virtual reality device itself is also acquired. Based on the multi-view image, the PnP algorithm is used to obtain the initial 6-DOF pose of the virtual reality device; Based on the motion data, the initial 6-DOF pose is globally optimized to obtain the target 6-DOF pose.

9. The virtual reality device as described in claim 6, characterized in that, The processor displays the image frame through the display screen, specifically as follows: When the perspective mode of the virtual reality device is turned off, the rendering thread is invoked to render the image frames of the VR application and display them on the binocular screen. When the perspective mode of the virtual reality device is enabled, the rendering thread is invoked to render the image frames of the VR application on the first layer, and the image frames of the multi-view camera are rendered on the second layer. The two image frames are then superimposed and displayed on the binocular screen through the first layer and the second layer.

10. The virtual reality device as described in any one of claims 6-9, characterized in that, The rendering thread, the second thread, and the positioning thread are executed concurrently, and the execution time of the rendering thread is less than that of the second thread and the positioning thread, respectively.

Images