Video perspective method and device, storage medium and program product

Abstract

The embodiment of the invention provides a video perspective method and device, a storage medium and a program product. The method comprises the steps of obtaining a first pose in a first degree-of-freedom mode, determining first depth information of a preset three-dimensional shape, and determining a first video perspective VST result according to the first depth information and the first pose. The embodiment of the invention provides the method. According to the video perspective method, the fixed depth information is acquired by constructing the preset three-dimensional shape, so that the video perspective effect is realized based on the fixed depth information, and the video perspective effect can also be realized under the condition that the depth information is not acquired completely or cannot be acquired; the image display fluency of the display equipment such as the head-mounted equipment is improved, and the user experience is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of extended reality, and in particular, to a video see-through method and device, a storage medium and a program product. BACKGROUND

[0002] Video see-through (VST) technology can provide interaction with the virtual world while maintaining perception of the real world, which is very important for safety, interaction and mixed reality (MR) applications.

[0003] In related technologies, an extended reality (XR) device cannot support VST display effect in some scenarios, and the user experience is poor. SUMMARY

[0004] Embodiments of the present disclosure provide a video see-through method, device, storage medium and program product to implement VST in more scenarios, improve VST fluency and improve user experience.

[0005] In a first aspect, embodiments of the present disclosure provide a video see-through method, comprising:

[0006] obtaining a first pose in a first degree of freedom mode;

[0007] determining first depth information of a preset three-dimensional shape;

[0008] determining a first video see-through (VST) result according to the first depth information and the first pose.

[0009] In a second aspect, embodiments of the present disclosure provide a video see-through device, comprising:

[0010] an obtaining module configured to obtain a first pose in a first degree of freedom mode;

[0011] a depth module configured to determine first depth information of a preset three-dimensional shape;

[0012] a determining module configured to determine a first video see-through (VST) result according to the first depth information and the first pose.

[0013] In a third aspect, embodiments of the present disclosure provide an electronic device, comprising a processor and a memory;

[0014] the memory stores computer-executable instructions;

[0015] the processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the video see-through method according to the first aspect and various possible designs of the first aspect.

[0016] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the video perspective method described in the first aspect and various possible designs of the first aspect.

[0017] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the video perspective method as described in the first aspect and various possible designs of the first aspect.

[0018] This embodiment provides a video perspective method, device, storage medium, and program product. The method acquires a first pose in a first degree of freedom mode, determines first depth information of a preset 3D shape, and determines a first video perspective (VST) result based on the first depth information and the first pose. The video perspective method provided in this disclosure acquires fixed depth information by constructing a preset 3D shape, and then achieves a video perspective effect based on this fixed depth information. It can achieve a video perspective effect even when depth information is incomplete or unavailable, improving the smoothness of image display on display devices such as head-mounted devices and enhancing the user experience. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram illustrating an application scenario of the video perspective method provided in the embodiments of this disclosure;

[0021] Figure 2 Flowchart of the video perspective method provided in this embodiment of the disclosure Figure 1 ;

[0022] Figure 3 Flowchart of the video perspective method provided in this embodiment of the disclosure Figure 2 ;

[0023] Figure 4 This is a structural block diagram of a video perspective device provided in an embodiment of the present disclosure;

[0024] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0025] To make the purposes, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are some but not all of the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present disclosure.

[0026] Video See-Through (VST) is a see-through technology implemented in an Extended Reality (XR) device, such as a Virtual Reality (VR) device, which allows users to see a real-time view of the real world through a display device, such as a head-mounted display device (headset). This technology usually captures the external environment through a camera on the headset and combines video images with virtual content, displayed on the screen. VST technology can provide interaction with the virtual world while maintaining perception of the real world, which is very important for safety, interaction, and Mixed Reality (MR) applications.

[0027] In related technologies, the synthesis of VST images is usually based on the 6 Degree Of Freedom (DOF) pose provided by the display device. However, the above-mentioned method has strong dependence on 6DOF. When 6DOF data is problematic, VST cannot be implemented, resulting in low VST smoothness and poor user experience.

[0028] To solve the above technical problems, the present inventors have found that in the case where the display device cannot obtain 6DOF data, for example, can only provide 3DOF data, mainly cannot obtain depth information, and then fuse with the depth information, therefore, the present inventors have found that the display device can be provided with preset depth information, such as preset stereoscopic shape depth information, and the obtained image of the environment is fused with the depth information to realize VST display, improve VST smoothness, and improve user experience. Based on this, the present disclosure provides a video see-through method.

[0029] Figure 1 The application scenario of the video see-through method provided by the present disclosure is shown in FIG. 1. Figure 1 As shown in FIG. 1, a user wears a display device 101. The display device 101 can be a VR device, an AR device, an MR device, or other XR devices, and can be a head-mounted display device.

[0030] In the specific implementation process, the display device 101 can enter a first degree of freedom mode (e.g., 3DOF mode), acquire the first pose in the first degree of freedom mode, determine the first depth information of the preset solid shape 102 (which can be a sphere with radius R), and determine the first video perspective (VST) result based on the first depth information and the first pose. The video perspective method provided in this embodiment of the present disclosure obtains fixed depth information by constructing a preset solid shape, and then realizes the video perspective effect based on the fixed depth information. It can achieve the video perspective effect even when the depth information is incomplete or unavailable, improve the smoothness of image display on display devices such as head-mounted devices, and enhance the user experience.

[0031] It should be noted that, Figure 1 The schematic diagram shown is merely an example. The video perspective method and scenario described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of the system and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0032] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0033] Figure 2 Flowchart of the video perspective method provided in this embodiment of the disclosure Figure 1 .like Figure 2 As shown, the video perspective method includes:

[0034] 201. Obtain the first pose in the first degree of freedom mode.

[0035] The execution subject of this disclosure embodiment can be a display device, for example, such as Figure 1 The display device 101 shown.

[0036] The first degree of freedom (DOF) mode can be a mode with fewer than 6 DOFs, such as 3DOF mode. The first pose can include position and orientation, or only orientation, depending on the number of DOFs. In 3DOF mode, although only orientation can be provided and not position, the video perspective method provided in this embodiment can provide first depth information of a preset stereoscopic shape, thereby enabling VST display in 3DOF mode based on the first depth information and the first pose. This broadens the application scenarios of VST, improves the smoothness of VST, and enhances the user experience.

[0037] In one embodiment of this disclosure, the first degree of freedom mode can be a mode of the display device or a mode switched to when a trigger condition is met. Specifically, obtaining the first pose in the first degree of freedom mode may include: if the quality of the pose data obtained in the second degree of freedom mode is lower than a preset standard, then switching to the first degree of freedom mode and obtaining the first pose in the first degree of freedom mode; the number of degrees of freedom obtained in the first degree of freedom mode is less than the number of degrees of freedom obtained in the second degree of freedom mode.

[0038] In one embodiment of this disclosure, the quality of the pose data acquired in the second degree of freedom mode is lower than a preset standard, including satisfying at least one of the following: Option 1: the light intensity of the current environment is less than a first intensity or greater than a second intensity; the first intensity is less than the second intensity; Option 2: the jitter value of the display device is greater than a preset jitter threshold; Option 3: the error of the sensor data of the display device is greater than a first preset error value; Option 4: the error of the image data acquired by the camera of the display device is greater than a second preset error value.

[0039] For example, taking the second degree of freedom mode as 6DOF mode as an example, the 6DOF quality can be determined to be lower than the preset standard in the following situations:

[0040] The first scenario: Invalid image acquisition – because the head-mounted device needs to acquire environmental images for 6DOF positioning, when the camera is detected to be blocked, or the acquired image is identified as invalid, the 6DOF quality may be lower than the preset standard.

[0041] The second scenario: Excessive shaking of the head-mounted device – When the head-mounted device shakes too violently, it is usually difficult to acquire high-quality images or calculate the position and orientation of the device, which may result in 6DOF quality being lower than the preset standard.

[0042] The third scenario: Insufficient environmental feature points – because head-mounted devices need to perform 6DOF positioning based on feature points in the acquired environmental images, the 6DOF quality may be lower than the preset standard when the user is facing a solid-color wall, wallpaper with a repeating pattern, or a mirror.

[0043] The fourth scenario: low quality of environmental feature points - when the brightness of the acquired image is too low, below the preset first brightness threshold, or the brightness of the acquired image is too high, above the second brightness threshold, the image quality is poor, which may result in the 6DOF quality being lower than the preset standard.

[0044] In this embodiment of the disclosure, the triggering condition for switching from the second degree of freedom mode to the first degree of freedom mode may also be that the depth sensor is unavailable or cannot obtain depth information.

[0045] 202. Determine the first depth information of the preset 3D shape.

[0046] In this embodiment of the disclosure, the preset three-dimensional shape can be a centrally symmetric three-dimensional shape, such as a sphere or a cube, or a non-centrally symmetric three-dimensional shape, such as a cuboid, an ellipse, or a cone. The preset three-dimensional shape can also be depth information predicted based on historical image data obtained by the display device or image data of the current environment obtained from other means (such as the Internet), such as the three-dimensional shape corresponding to a mesh.

[0047] In one embodiment of this disclosure, the preset three-dimensional shape is a sphere. Determining the first depth information of the preset three-dimensional shape may include: determining the center of the sphere based on the location of the display device, determining the corresponding sphere with a preset length as the radius, and determining the first depth information based on the surface of the sphere. The preset length may be greater than or equal to 4 meters, for example, 5 meters.

[0048] In this embodiment of the disclosure, the location of the display device may not be its position coordinates in a real world coordinate system, but may be a relative position determined based on sensor data, such as inertial sensor data. As long as the display device is located at the center of the sphere, this embodiment of the disclosure does not limit the method of determining the location of the display device.

[0049] 203. Determine the first video perspective (VST) result based on the first depth information and the first pose.

[0050] Specifically, the texture information of the environment in which the display device is located can be collected by a visual sensor (such as an RGB camera) on the display device, and then, based on the first pose, the texture information can be fused onto a mesh (such as a mesh determined by the surface of a sphere) based on the first depth information.

[0051] As can be seen from the above description, the video perspective method provided in this disclosure constructs a preset three-dimensional shape to obtain fixed depth information, and then realizes the video perspective effect based on the fixed depth information. It can achieve the video perspective effect even when the depth information is incomplete or unavailable, thereby improving the smoothness of image display on display devices such as head-mounted devices and enhancing the user experience.

[0052] refer to Figure 3 , Figure 3 Flowchart of the video perspective method provided in this embodiment of the disclosure Figure 2 This embodiment details the implementation process and switching strategy of video perspective in different scenarios. The video perspective method includes:

[0053] 301. Obtain the second pose in the second degree of freedom mode.

[0054] 302. Acquire sensor data and determine the second depth information based on the sensor data.

[0055] 303. Determine the second VST result based on the second pose and the second depth information.

[0056] 304. Determine whether the quality of the pose data acquired in the second degree of freedom mode is lower than the preset standard. If yes, proceed to step 305. If no, continue to maintain the second degree of freedom mode.

[0057] In one embodiment of this disclosure, the display device includes a color sensor, and the step of acquiring sensing data and determining second depth information based on the sensing data may include: acquiring first image data through the color sensor; and determining second depth information based on the first image data.

[0058] Specifically, the second depth information can be determined using only the first image data acquired by a color sensor, such as an RGB camera.

[0059] In one embodiment of this disclosure, the display device further includes a depth sensor, and the step of determining the second depth information based on the first image data may include: if the depth sensor fails, determining the second depth information based on the first image data; if the depth sensor is working normally, acquiring depth sensing data through the depth sensor; and determining the second depth information based on the depth sensing data and the first image data.

[0060] Specifically, when the display device is equipped with a depth sensor (such as a Time of Flight (TOF) sensor), if the depth sensor is functioning correctly, the second depth information can be determined by combining the depth sensing data acquired by the depth sensor with the first image data acquired by the color sensor. This method provides higher accuracy in depth information compared to the method described above, which determines the second depth information solely based on the first image data acquired by the color sensor. Therefore, this method should be preferred when improving the accuracy of the VST display is required. When the depth sensor malfunctions, the method of determining the second depth information solely based on the first image data acquired by the color sensor should be switched to.

[0061] 305. Switch to the first degree of freedom mode and obtain the first pose in the first degree of freedom mode; the number of degrees of freedom obtained in the first degree of freedom mode is less than the number of degrees of freedom obtained in the second degree of freedom mode.

[0062] In this embodiment of the disclosure, when switching from the second degree of freedom mode to the first degree of freedom mode, the depth information used is switched from the second depth information to the first depth information.

[0063] In one possible implementation, when switching from the second depth information to the first depth information, taking a mesh as an example, the origin coordinates of the mesh will switch from (0,0,0) in the world coordinate system to the location of the display device, i.e., the center of the sphere.

[0064] 306. Obtain the first pose in the first degree of freedom mode.

[0065] 307. Determine the first depth information of the preset 3D shape.

[0066] 308. Determine the first video perspective (VST) result based on the first depth information and the first pose.

[0067] Steps 306 to 308 in this embodiment are similar to steps 201 to 203 in the above embodiment, and will not be repeated here.

[0068] For example, the video perspective method provided in this disclosure can cover the following scenarios:

[0069] The first scenario: VST works normally when depth information services (such as a Time-of-Flight (TOF) sensor that is not working properly) are available.

[0070] The second scenario: VST can still be used when depth information services (such as a Time-of-Flight (TOF) sensor that is not working properly) fail.

[0071] At this point, some depth information is missing. The depth information collected through RGB is still synthesized into VST through 6DOF, which will slightly worsen the VST effect.

[0072] In the second scenario, if the 6DOF quality is lower than the preset standard, the system can switch to the third scenario. The switching process can be instantaneous or completed within a preset time, such as 10ms. The switching process can be set to instantaneous to quickly ensure a backup experience, or it can be set to remain stationary without an instantaneous switch.

[0073] The third scenario: VST can still be used in 3DOF mode. For example, when the ambient light is extremely bright / dark, 6DOF service is unavailable, but VST can be used in 3DOF mode.

[0074] In this scenario, there is a distinct transition during the switch. At this point, the VST's depth can switch to the first depth information of a preset 3D shape, such as the depth information determined by the surface of a 5m sphere. Before switching to this state, the 6DoF service state may be unstable (the 6DoF remains on the last result, thus fixing the depth).

[0075] The trigger condition for switching back to the second scenario can be: 6DOF signal recovery. The switching process can be set to have a delay of approximately 70-100ms.

[0076] As can be seen from the above description, the video perspective method provided in this disclosure adopts different depth information determination methods for different scenarios, thereby achieving VST display effects in different scenarios, enabling process switching between different scenarios, and improving user experience.

[0077] Corresponding to the video perspective method in the above embodiments, Figure 4 This is a structural block diagram of a video perspective device provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 4 The device includes: an acquisition module 401, a depth module 402, and a determination module 403.

[0078] The acquisition module 401 is used to acquire the first pose in the first degree of freedom mode.

[0079] Depth module 402 is used to determine the first depth information of the preset three-dimensional shape.

[0080] The determination module 403 is used to determine the first video perspective (VST) result based on the first depth information and the first pose.

[0081] In one embodiment of this disclosure, the preset three-dimensional shape is a sphere.

[0082] In one embodiment of this disclosure, the depth module 402 is specifically used for:

[0083] The center of the sphere is determined based on the location of the display device, and the corresponding sphere is determined with a preset length as the radius;

[0084] The first depth information is determined based on the surface of the sphere.

[0085] In one embodiment of this disclosure, the acquisition module 401 is specifically used for:

[0086] If the quality of the pose data acquired in the second degree of freedom mode is lower than the preset standard, then switch to the first degree of freedom mode to acquire the first pose in the first degree of freedom mode; the number of degrees of freedom acquired in the first degree of freedom mode is less than the number of degrees of freedom acquired in the second degree of freedom mode.

[0087] In one embodiment of this disclosure, the quality of the pose data acquired in the second degree of freedom mode is lower than a preset standard, including satisfying at least one of the following:

[0088] The light intensity of the current environment is less than the first intensity or greater than the second intensity; the first intensity is less than the second intensity.

[0089] The jitter value of the display device is greater than a preset jitter threshold;

[0090] The error in the sensor data of the display device is greater than a first preset error value;

[0091] The error in the image data acquired by the camera of the display device is greater than a second preset error value.

[0092] In one embodiment of this disclosure, the acquisition module 401 is further configured to:

[0093] Obtain the second pose in the second degree of freedom mode;

[0094] Acquire sensor data, and determine second depth information based on the sensor data;

[0095] The second VST result is determined based on the second pose and the second depth information.

[0096] In one embodiment of this disclosure, the display device includes a color sensor, and the acquisition module 401 is specifically used for:

[0097] First image data is acquired through the color sensor;

[0098] The second depth information is determined based on the first image data.

[0099] In one embodiment of this disclosure, the display device further includes a depth sensor, and the acquisition module 401 is specifically used for:

[0100] If the depth sensor fails, then the second depth information is determined based on the first image data;

[0101] If the depth sensor is working properly, depth sensing data is acquired through the depth sensor; and second depth information is determined based on the depth sensing data and the first image data.

[0102] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0103] To implement the above embodiments, this disclosure also provides an electronic device.

[0104] refer to Figure 5The diagram illustrates a structural schematic of an electronic device 900 suitable for implementing embodiments of the present disclosure. The electronic device 900 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0105] like Figure 5 As shown, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0106] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0107] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0108] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0109] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0110] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0111] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0113] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0114] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0115] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0116] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0117] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0118] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method of video see-through, characterized by, The application is applied to a display device, comprising: obtaining a first pose in a first degree of freedom mode; determining first depth information of a preset three-dimensional shape; determining a first video see-through (VST) result according to the first depth information and the first pose.

2. The method of claim 1, wherein, The preset three-dimensional shape is a sphere.

3. The method of claim 2, wherein, The determination of the first depth information of the preset three-dimensional shape comprises: determining a sphere center according to a position where the display device is located, and determining a corresponding sphere with a preset length as a radius; determining the first depth information based on a sphere surface of the sphere.

4. The method of claim 1, wherein, The obtaining of the first pose in the first degree of freedom mode comprises: if quality of pose data obtained in a second degree of freedom mode is lower than a preset standard, switching to the first degree of freedom mode to obtain the first pose in the first degree of freedom mode; the number of degrees of freedom obtained in the first degree of freedom mode is less than the number of degrees of freedom obtained in the second degree of freedom mode.

5. The method of claim 4, wherein, The quality of the pose data obtained in the second degree of freedom mode being lower than the preset standard comprises at least one of the following: light intensity of a current environment is less than a first intensity or greater than a second intensity; the first intensity is less than the second intensity; a jitter value of the display device is greater than a preset jitter threshold value; an error of sensor data of the display device is greater than a first preset error value; an error of image data obtained by a camera of the display device is greater than a second preset error value.

6. The method according to any one of claims 1 to 5, characterized in that, Before the obtaining of the first pose in the first degree of freedom mode, the method further comprises: obtaining a second pose in a second degree of freedom mode; obtaining sensor data, and determining second depth information according to the sensor data; determining a second VST result according to the second pose and the second depth information.

7. The method of claim 6, wherein, The display device comprises a color sensor, and the obtaining of the sensor data and the determination of the second depth information according to the sensor data comprise: obtaining first image data through the color sensor; determining the second depth information according to the first image data.

8. The method of claim 7, wherein, The display device further comprises a depth sensor, and the determination of the second depth information according to the first image data comprises: if the depth sensor is invalid, determining the second depth information according to the first image data; if the depth sensor is normal, obtaining depth sensor data through the depth sensor, and determining the second depth information according to the depth sensor data and the first image data.

9. A video see-through device, characterized by, The method comprises: a obtaining module, configured to obtain a first pose in a first degree of freedom mode; a depth module, configured to determine first depth information of a preset three-dimensional shape; a determining module, configured to determine a first video see-through (VST) result according to the first depth information and the first pose.

10. An electronic device, comprising: The method comprises: a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the processor executes the video see-through method according to any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the video see-through method according to any one of claims 1 to 8 is realized.

12. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the video see-through method according to any one of claims 1 to 8.

Images