Method, apparatus, storage medium and electronic device for predicting depth information

Abstract

The embodiment of the present disclosure discloses a method and device for obtaining depth information, a storage medium and an electronic device, wherein the method comprises: obtaining posture information of a head-mounted display device; determining a height from the ground of the head-mounted display device in the vertical direction based on displacement information in the vertical direction and a base value in the vertical direction; determining an observation angle of the head-mounted display device based on rotation information; and performing depth information prediction based on the height from the ground and the observation angle to obtain a predicted observation depth at a target time. The embodiment of the present disclosure can make a reasonable prediction of the depth information in the case where the depth information of a virtual object cannot be directly obtained, so as to perform distortion processing by using the predicted depth information and improve the display effect of the head-mounted display device.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular, to a method and device for obtaining depth information, a storage medium and an electronic device. BACKGROUND

[0002] In a virtual reality (VR) or augmented reality (AR) or mixed reality (MR) scene, a terminal provides an interactive immersive experience for a user by constructing a virtual environment.

[0003] When the user watches a display screen using an AR display device or a VR display device, in order to make the interaction between the virtual scene and the real world more realistic (for example, to make a static virtual object static relative to the real environment), a distortion process is usually performed based on the posture of the head-mounted display device before and after rendering the screen and the depth information of the virtual object in the rendered screen. In some scenarios such as cloud rendering, it is difficult to obtain the depth information of the virtual object, and thus it is difficult to perform the distortion process, which affects the user experience.

[0004] How to reasonably predict the depth information in the case where it is difficult to obtain the depth information of the virtual object is a problem to be solved. SUMMARY

[0005] To solve the above technical problems, the present disclosure is proposed. Embodiments of the present disclosure provide a method and device for obtaining depth information, a storage medium and an electronic device.

[0006] According to a first aspect of an embodiment of the present disclosure, a method for predicting depth information is provided, comprising:

[0007] obtaining posture information of a head-mounted display device, wherein the posture information at least includes rotation information relative to a reference coordinate system and displacement information in a vertical direction, and the vertical direction of the reference coordinate system is aligned with the direction of gravity;

[0008] determining the height from the ground of the head-mounted display device in the vertical direction based on the displacement information in the vertical direction and a basic value in the vertical direction;

[0009] determining an observation angle of the head-mounted display device based on the rotation information, wherein the observation angle is the included angle between the front direction of the head-mounted display device and the vertical direction of the reference coordinate system;

[0010] performing depth information prediction based on the height from the ground and the observation angle to obtain a predicted observation depth at a target time, wherein the predicted observation depth is the distance between the displayed screen at the target time and the head-mounted display device.

[0011] According to a second aspect of the embodiments of the present disclosure, a device for predicting depth information is provided, comprising:

[0012] an attitude information obtaining module, configured to obtain attitude information of the head-mounted display device, the attitude information comprising at least rotation information and vertical direction displacement information relative to a reference coordinate system, wherein the vertical direction of the reference coordinate system is aligned with the direction of gravity;

[0013] a height-from-ground determining module, configured to determine the height-from-ground of the head-mounted display device in the vertical direction based on the vertical direction displacement information and a basic value of the vertical direction;

[0014] an observation angle determining module, configured to determine an observation angle of the head-mounted display device based on the rotation information, wherein the observation angle is the included angle between the front direction of the head-mounted display device and the vertical direction of the reference coordinate system;

[0015] a depth predicting module, configured to predict the depth information based on the height-from-ground and the observation angle to obtain a predicted observation depth at a target time, wherein the predicted observation depth is the distance between the displayed picture at the target time and the head-mounted display device.

[0016] According to a third aspect of the embodiments of the present disclosure, a head-mounted display device is provided, comprising the device for predicting depth information according to the second aspect.

[0017] According to a fourth aspect of the embodiments of the present disclosure, an electronic device is provided, comprising:

[0018] a processor;

[0019] a memory for storing processor-executable instructions;

[0020] the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for predicting depth information according to the first aspect.

[0021] According to a fifth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores a computer program for executing the method for predicting depth information according to the first aspect.

[0022] The technical solutions of the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0023] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which: The accompanying drawings are provided to assist in understanding the embodiments of the present disclosure, and constitute a part of the specification. The drawings illustrate embodiments of the present disclosure and, together with the description, serve to explain the present disclosure. However, the present disclosure is not limited by the accompanying drawings.

[0024] Figure 1 is an exemplary system architecture diagram that can be applied to an embodiment of the method or apparatus for predicting depth information of the present disclosure;

[0025] Figure 2 is a flowchart of a method for predicting depth information in one embodiment of the present disclosure;

[0026] Figure 3 is a schematic diagram of a head-mounted display device coordinate system and a user focus point in one example of the present disclosure;

[0027] Figure 4 is a flowchart of step S8 in one embodiment of the present disclosure;

[0028] Figure 5 is a structural block diagram of an apparatus for predicting depth information in one embodiment of the present disclosure;

[0029] Figure 6 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Hereinafter, example embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, and not all embodiments of the present disclosure. It should be understood that the present disclosure is not limited by the example embodiments described herein.

[0031] It should be noted that: unless otherwise specified, the relative arrangement, numerical expression and numerical value of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0032] Those skilled in the art can understand that the terms "first", "second" and the like in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meaning, nor do they represent the inevitable logical sequence between them.

[0033] It should also be understood that in the embodiments of the present disclosure, "a plurality of" can mean two or more, and "at least one" can mean one, two or more.

[0034] It should also be understood that, whenever used in the present disclosure, a description of one or more particular embodiments of the application does not mean that the scope of the application is limited to only those embodiments. The disclosure of aspects of the application in the detailed description and drawings does not constitute an admission that all of the aspects of the application, and / or each aspect of the application, must include the described aspects and / or options thereof. And, the disclosure of aspects of the application in the detailed description and drawings does not, and should not be taken to mean that all of the aspects of the application must include the discussed aspects and / or options thereof.

[0035] In addition, the term "and / or" in the present disclosure is merely used to describe associated objects, and can represent that there can be three types of relationships, for example, A and / or B can represent that there are three cases of A alone, A and B, and B alone. In addition, the character " / " in the present disclosure generally represents an "or" relationship between the front and rear associated objects.

[0036] It should also be understood that the description of the various embodiments of the present disclosure emphasizes the differences between the various embodiments, and the same or similar parts can be referred to each other, and for the sake of brevity, will not be repeated.

[0037] The following description of at least one example embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application or uses.

[0038] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein. However, where appropriate, such techniques, methods, and devices can be considered part of the specification.

[0039] It should be noted that similar reference numerals and letters refer to similar items in the following drawings, and therefore, once an item is defined in one drawing, it need not be discussed further in subsequent drawings.

[0040] Embodiments of the present disclosure can be applied to terminal devices, computer systems, servers, and other electronic devices, which can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with terminal devices, computer systems, servers, and other electronic devices include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems, and the like.

[0041] Electronic devices such as terminal devices, computer systems, servers, and the like can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like that perform particular tasks or implement particular abstract data types. Computer systems / server can be practiced in distributed cloud computing environments with remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in local or remote computer system storage media including memory storage devices.

[0042] Figure 1 is an exemplary system architecture diagram that can be applied to embodiments of the method or apparatus for predicting depth information of the present disclosure.

[0043] As shown in Figure 1 , the system architecture can include a head-mounted display device 1, a network 2, and a server 3. The network 2 can be a medium for providing a communication link between the head-mounted display device 1 and the server 3. The network 2 can include various connection types, such as wired, wireless communication links, or fiber optic cables, and the like.

[0044] The head-mounted display device 1 described above can be an electronic device with an image display function, including but not limited to AR smart glasses, VR smart glasses, and the like. The head-mounted display device 1 can be an all-in-one machine, i.e., the head-mounted display device 1 can install various client applications. Alternatively, the head-mounted display device 1 can also be used in cooperation with a terminal device to realize a split-type device with an image display function, at which time the terminal device can undertake the computing function of the head-mounted display device. Here, for the sake of convenience, the all-in-one machine and the split-type device can be collectively referred to as the head-mounted display device.

[0045] The head-mounted display device 1 described above can provide various services, such as determining the predicted observation depth of the head-mounted display device at a target time based on the height above ground level and the observation angle of the head-mounted display device after obtaining the height above ground level and the observation angle.

[0046] It should be noted that the method for predicting depth information provided by the embodiments of the present disclosure can be executed by the head-mounted electronic device 1, and accordingly, the apparatus for predicting depth information is generally provided in the head-mounted electronic device 1.

[0047] Alternatively, the method for predicting depth information provided by the embodiments of the present disclosure can also be executed by a terminal device connected to the head-mounted electronic device 1, and accordingly, the apparatus for predicting depth information can also be provided in the terminal device.

[0048] It should be noted that although the scheme of the present disclosure can be applied to the head-mounted display device, it does not exclude that the scheme can also be applied to the server 3, which can be a background server. In the case where the scheme of the present disclosure is applied to the server 3, the server obtains the attitude information of the head-mounted display device, and based on the displacement information of the vertical direction in the attitude information and the base value of the vertical direction, the height from the ground of the head-mounted display device 1 can be determined. The server can determine the observation angle of the head-mounted display device from the rotation information of the vertical direction in the attitude information, and then the server predicts the depth information based on the height from the ground and the observation angle to obtain the predicted observation depth of the head-mounted display device at the target time. In this case, the method for predicting the depth information can be executed by the server 3, and accordingly, the device for predicting the depth information can also be provided in the server 3.

[0049] Exemplary method

[0050] Figure 2 is a flowchart of a method for predicting depth information in an embodiment of the present disclosure. As shown in Figure 2 , the method comprises the following steps:

[0051] S2: Obtain the attitude information of the head-mounted display device.

[0052] The head-mounted display device is provided with a device capable of obtaining data related to the attitude information, and the data related to the attitude information obtained by the device is used to calculate the attitude information of the head-mounted display device. For example, the head-mounted display device is provided with an inertial sensor (Inertial Measurement Unit, IMU), a camera device, etc. The attitude angle and three-axis acceleration of the head-mounted display device can be obtained by the IMU. The image within the camera range of the head-mounted display device can be obtained by the camera device. The above-mentioned data can be processed by using a Simultaneous Localization And Mapping (SLAM) algorithm to obtain the attitude information of the head-mounted display device. The attitude information at least includes rotation information relative to a reference coordinate system and displacement information in the vertical direction. The reference coordinate system is a three-dimensional coordinate system with a certain position as the coordinate origin, and the vertical direction of the reference coordinate system is aligned with the direction of gravity. It can be understood that the reference coordinate system can be fixed relative to the real world. The rotation information can be represented by a rotation matrix, a quaternion, etc. The displacement information in the vertical direction can be understood as the displacement of the head-mounted display device in the vertical direction relative to the origin of the reference coordinate system.

[0053] S4: Based on the displacement information in the vertical direction and the base value in the vertical direction, determine the height from the ground of the head-mounted display device in the vertical direction.

[0054] The vertical baseline value can be the height of the head-mounted display device from the ground as determined when the device is powered on. For example, the vertical baseline value can be a set value or a distance measurement value of the head-mounted display device before step S2. The vertical displacement value of the head-mounted display device can be summed with the vertical baseline value to obtain the vertical height of the head-mounted display device from the ground. It is understood that the term "summation" here is not limited to simple addition; it can also be based on the displacement value and the baseline value combined with other relevant information to obtain the height from the ground. In this application, "height from the ground" refers to the assessed distance of the head-mounted display device from the vertically downward baseline plane in the environment in which the head-mounted display device is located. For example, when a user wears the head-mounted display device indoors, the height from the ground can be understood as the assessed distance of the head-mounted display device from the indoor floor.

[0055] S6: Determine the observation angle of the head-mounted display device based on rotation information.

[0056] The observation angle can be the angle between the front direction of the head-mounted display device and the vertical direction of the reference coordinate system. The front direction of the observation angle can be the direction of the line connecting the origin of the head-mounted display device's coordinate system and the user's point of interest.

[0057] Figure 3 This is a schematic diagram of the coordinate system of a head-mounted display device and the user's point of focus in one example of this disclosure. For example... Figure 3 As shown, the coordinate system of the head-mounted display device can be the center point O of the head-mounted display device, and the forward direction of the observation angle can be the Z-axis direction of the head-mounted display device's coordinate system. Furthermore, as the posture of the head-mounted display device changes, the origin of the head-mounted display device's coordinate system and the directions of the three-axis coordinate system are adjusted accordingly.

[0058] The vertical direction of the reference coordinate system is aligned with the direction of gravity, which can be obtained through an IMU. For example, the observation angle of a head-mounted display can be obtained based on the pitch angle and vertical direction from the rotation information.

[0059] S8: Based on the altitude above the ground and the observation angle, depth information is predicted to obtain the predicted observation depth of the head-mounted display device at the target time.

[0060] Please refer to this again. Figure 3This method allows defining the user's focus point D along the Z-axis of the head-mounted display's coordinate system. Using the head-mounted display's height above the ground and the viewing angle as inputs for depth prediction, a preset depth prediction formula is used to obtain the predicted observation depth of the head-mounted display at a target time. The target time can be a preset time after the ground height and viewing angle have been determined. The predicted observation depth is the distance between the displayed image and the head-mounted display at the target time. After predicting the target observation depth, the displayed image at the target time can be distorted based on this predicted depth, thereby improving the display effect of the head-mounted display and enhancing the user experience.

[0061] In one example, the distance between the image displayed at the target time and the head-mounted display device can be understood as the distance between the main virtual content in the image displayed at the target time and the head-mounted display device. This virtual content can be objects, people, animals, scenes, etc., in the displayed image. In another example, the distance between the image displayed at the target time and the head-mounted display device can be understood as the straight-line distance between a point associated with the image and a point associated with the head-mounted display device, or as the distance between a point associated with the image and a plane associated with the head-mounted display device. Similarly, it can be understood as the straight-line distance between a plane associated with the image and a point associated with the head-mounted display device, or as the distance between a plane associated with the image and a plane associated with the head-mounted display device. Optionally, the point associated with the image and the plane can be points and planes of virtual content within a preset area of ​​the image. For example, a point of virtual content located at the center of the screen, a point of virtual content located at the top corner of the screen, a point of virtual content located along the edge of the screen, a plane of virtual content located within a preset range of the center of the screen, and so on. Similarly, a location point associated with a head-mounted display device can be the origin of the head-mounted display device's coordinate system, a preset location point on the head-mounted display device itself, or a point at a preset distance from the origin of the head-mounted display device's coordinate system, etc. A plane associated with a head-mounted display device can be a plane defined between two coordinate axes in the head-mounted display device's coordinate system, a preset plane on the head-mounted display device itself, or a plane at a preset distance from the origin of the head-mounted display device's coordinate system, etc.

[0062] In this embodiment, after obtaining the attitude information of the head-mounted display device, on the one hand, based on the displacement information in the vertical direction in the attitude information combined with the basic value in the vertical direction, the height of the head-mounted display device from the ground in the vertical direction can be determined. On the other hand, based on the rotation information in the attitude information, the viewing angle of the head-mounted display device can be determined. Furthermore, based on the height of the head-mounted display device from the ground and the viewing angle, the predicted viewing depth at the target moment can be reasonably predicted, so as to realize the reasonable prediction of the depth information when the depth information of the virtual object cannot be directly obtained in scenarios such as cloud rendering, etc., so as to use the predicted depth information to perform distortion processing on the display screen, improve the display effect of the head-mounted display device, and enhance the user experience.

[0063] Figure 4 It is a schematic flowchart of step S8 in an embodiment of the present disclosure. As Figure 4 shown, in an embodiment of the present disclosure, step S8 includes:

[0064] S8-2: Perform a standardized numerical range processing on the height from the ground to obtain a processed value of the height from the ground of the head-mounted display device.

[0065] Generally, when a user wears a head-mounted display device, it can be considered that the height of the head-mounted display device from the ground is within [H1, H2], which respectively correspond to the numerical intervals of the height from the ground when an average user wears the head-mounted display device while sitting and the height from the ground when the user wears the head-mounted display device while standing. Among them, H1 < H2. For example, H1 can be 1 meter and H2 can be 1.5 meters. When the height from the ground is less than H1, the processed value h of the height from the ground can be set to 0. When the height from the ground is greater than H2, the processed value h of the height from the ground can be set to 1. When the height from the ground is between [H1, H2], the processed value of the height from the ground can be set to a value between (0, 1). For example, the processed value h of the height from the ground is obtained according to the following formula:

[0066]

[0067] where H represents the height from the ground.

[0068] S8-4: Perform a standardized numerical range processing on the viewing angle to obtain a processed value θ of the viewing angle of the head-mounted display device.

[0069] Typically, when a user wears a head-mounted display, the viewing angle of the display is within the range [θ1, θ2], corresponding to the union of the typical viewing angle ranges when the user is sitting and standing. Here, θ1 < θ2; for example, θ1 can be π / 3 radians, and θ2 can be 5π / 16 radians. When the viewing angle is less than θ1, the viewing angle processing value θ can be set to 0. When the viewing angle is greater than θ2, the viewing angle processing value θ can be set to 1. When the height above the ground is between [θ1, θ2], the viewing angle processing value θ can be set to a value between (0, 1), for example, the viewing angle processing value θ can be obtained using the following formula:

[0070]

[0071] Where ω represents the observation angle.

[0072] S8-6: Determine the predicted observation depth based on the processed values ​​of the altitude above the ground and the processed values ​​of the observation angle.

[0073] By using the ground elevation value h and the observation angle value θ as inputs for depth prediction, the predicted observation depth of the head-mounted display device at the target time can be obtained through a preset depth prediction formula.

[0074] In this embodiment, it can be assumed that when a user stands up or walks while using a head-mounted display device, the objects they observe are farther away; when the user sits down or bends over, the objects they observe are closer, such as when observing a virtual car interior. When a user looks straight ahead or upwards while using the head-mounted display device, the objects they observe are generally farther away; when the observer looks downwards, the objects they observe are closer; at the same time, the downward viewing distance is also limited by the possible virtual ground, thus shortening accordingly. Based on the above assumptions, by processing the height from the ground and the observation angle separately to obtain the processed values ​​for the height from the ground and the observation angle, and then based on these processed values, depth information can be reasonably predicted, and the predicted values ​​can be within a specified numerical range.

[0075] In another embodiment of this disclosure, the predicted observation depth can be determined in step S8-6 using the following formula:

[0076] D = [a*Log2(P+1)+b]

[0077] P = h * θ

[0078] Where [a*Log2(P+1)+b] is the result of a*Log2(P+1)+b rounded down, where a and b are both given values, h is the height above the ground processed value and h∈[0,1], and θ is the observation angle processed value and θ∈[0,1].

[0079] In this embodiment, the method used to determine the predicted observation depth has good image processing effects for depths ranging from 0 to 5 meters (any value within this range) near the focal depth. For example, an integer value of 1 to 5 meters (any value within this range) can be used as the depth (e.g., 1 meter, 1.5 meters, 2.7 meters, etc.). This is because objects that need to be viewed at close range are generally considered to be 0.5 to 1 meter away from the human eye, and in most head-mounted display scenarios, the furthest point of the displayed object is generally no more than 10 meters away from the user using the head-mounted display. The logarithmic function is used due to the requirement of the rate of change. When the predicted depth is far, the position of the user's focus point changes slowly, and when the predicted depth is near, the position of the user's focus point changes rapidly, which is consistent with most observation scenarios in the application scenarios displayed by most head-mounted display devices. Since continuous image processing accompanied by depth changes may cause image distortion or floating, rounding the value can avoid the above situations and improve the display effect. Therefore, the method for determining the predicted observation depth used in this embodiment can be applied to most scenarios displayed by head-mounted display devices, and can meet the needs of ergonomics and scenarios where head-mounted display devices are used.

[0080] Understandably, considering the indoor scenes displayed by most head-mounted displays, in one example, the final result D can be expected to be an integer from 1 to 5. In this example, we can first determine the value of b, for example, it could be a value from 0.5 to 1.4, such as 0.55, 0.6, or 0.8, hoping to have a larger proportion of the P value range so that the output result D is 1. Then we determine a, the value of a can be from 4.5-b to 5.4-b, hoping that the upper limit of the output result D is 5.

[0081] In one embodiment of this disclosure, step S8-6 includes:

[0082] S8-6-A-2: Obtain the gaze point of the user wearing the head-mounted display device.

[0083] It can acquire users' binocular gaze data in real time, and use eye-tracking technology to process the binocular gaze data to obtain the gaze point of the user wearing the head-mounted display device.

[0084] S8-6-A-4: Obtain the distance between the gaze point and the head-mounted display device.

[0085] In one example of this disclosure, the coordinate position of the head-mounted display device in the reference coordinate system can be obtained based on the positioning device built into the head-mounted display device, and the coordinate position of the gaze point in the reference coordinate system can be obtained. Then, the distance between the gaze point and the head-mounted display device can be calculated based on the coordinate position of the head-mounted display device in the reference coordinate system and the coordinate position of the gaze point in the reference coordinate system.

[0086] In another example of this disclosure, a camera is set on the head-mounted display device to capture images of the user's eyes while wearing the head-mounted display device. The coordinates of the user's gaze point in the camera coordinate system are determined based on the images of the user's eyes. The distance between the gaze point and the head-mounted display device can be calculated based on the position of the camera in the head-mounted display device and the coordinates of the gaze point in the camera coordinate system.

[0087] S8-6-A-6: Determine the predicted observation depth based on the processed values ​​of ground elevation, observation angle, and the distance between the gaze point and the head-mounted display.

[0088] By taking the processed values ​​of ground height, observation angle, and the distance between the gaze point and the head-mounted display as inputs, the predicted observation depth can be determined using a preset depth calculation formula.

[0089] In this embodiment, the predicted observation depth can be reasonably calculated based on the ground height processing value, the observation angle processing value, and the distance between the gaze point and the head-mounted display device.

[0090] In another embodiment of this disclosure, step S8-6 may also include:

[0091] S8-6-B-2: Determine the basic observation depth value based on the processed values ​​of ground elevation and observation angle. For example, the basic observation depth value D' can be calculated using the following formula:

[0092] D' = [a*Log2(P+1)+b]

[0093] P = h * θ.

[0094] S8-6-B-4: If the base observation depth value is greater than the maximum preset observation depth, the base observation depth value is reduced to obtain the predicted observation depth. For example, the minimum and maximum preset observation depth values ​​can be set based on the size of the space where the head-mounted display device is located. In one example, these minimum and maximum values ​​can be set to D1 and D2, respectively. For example, when D' > D2, the value of D' is reduced to D2, and D2 is used as the predicted observation depth D. For example, when D' is 5.5 meters and D2 is 5 meters, D' is reduced to 5 meters to avoid the observation depth being too large, which could lead to a decrease in display effect when the user uses the head-mounted display device.

[0095] S8-6-B-6: If the value of the basic observation depth is less than the minimum value of the preset observation depth, perform a numerical increase process on the value of the basic observation depth to obtain the predicted observation depth. For example, when D’ < D1, increase the value of D’ to D1, and use D1 as the predicted observation depth D. For example, when D’ is 0.2 meters and D1 is 0.5 meters, increase D’ to 0.5 meters, which can avoid the reduction of the display effect caused by the predicted depth being too close when the user uses the head-mounted display device due to the observation depth being too small.

[0096] S8-6-B-8: If the value of the basic observation depth is less than the maximum value of the preset observation depth and greater than the minimum value of the preset observation depth, determine the value of the basic observation depth as the predicted observation depth. For example, when D’ ∈ [D1, D2], use D’ as the predicted observation depth D. For example, when D’ is 1.2 meters, D1 is 0.5 meters, and D2 is 5 meters, at this time, the value of D’ is within the range of the observation depth with a better display effect, so there is no need to adjust the value of D’.

[0097] It should be noted that this embodiment does not limit the execution order between step S8-6-6-4, step S8-6-6-6, and step S8-6-6-8.

[0098] In this embodiment, based on the processed value of the height from the ground and the processed value of the observation angle, the value of the basic observation depth can be determined, and by processing the value of the basic observation depth based on the maximum value and the minimum value of the preset observation depth, the predicted observation depth can be made to fall within the numerical range interval that meets the ergonomics requirements.

[0099] In an embodiment of the present disclosure, step S8-6-A-6 may include:

[0100] S8-6-A-6-2: Based on the distance between the fixation point and the head-mounted display device, determine the predicted depth adjustment coefficient. The predicted depth adjustment coefficient can be determined by the following formula:

[0101] f(G) = 1 + ([a + b] - G) * c

[0102] Among them, f(G) represents the predicted depth adjustment coefficient, G represents the distance between the fixation point and the head-mounted display device (for the explanation of the distance between the two, refer to the aforementioned understanding of "distance"), and a, b, and c are all set constants. By reasonably setting the above constants, using this formula can make the value range of f(G) around 1. For example, by reasonably setting the values of a, b, and c, the value of f(G) can be around 1, such as between 0.8 and 1.2. When the distance between the fixation point and the head-mounted display device is relatively far (for example, (a + b) < G), the predicted depth adjustment coefficient is less than 1. When the distance between the fixation point and the head-mounted display device is relatively close (for example, (a + b) > G), the predicted depth adjustment coefficient is greater than 1.

[0103] S8-6-A-6-4: Determine the predicted observation depth based on the ground distance processing value, the observation angle processing value, and the predicted depth adjustment coefficient.

[0104] The predicted observation depth can be determined by the following formula:

[0105] D” = [[a*Log2(P + 1)+b]*f(G)]

[0106] Among them, D” represents the predicted observation depth.

[0107] In this embodiment, it can be considered that the main virtual content in the to-be-displayed screen has a certain size. Directly judging the speculated depth as G will cause the entire virtual content to be displayed at a farther / nearer position when the distance G between the fixation point and the head-mounted display device is relatively far / close. Based on the above considered results, determining the predicted depth adjustment coefficient based on the distance between the fixation point and the head-mounted display device can make the value of the predicted depth adjustment coefficient around 1. When the distance between the fixation point and the origin is relatively far, the predicted observation depth can be pulled in. When the distance between the fixation point and the origin is relatively close, the speculated depth can be pushed away, so that the predicted depth can be within the numerical range interval corresponding to ergonomics.

[0108] In one embodiment of this disclosure, before step S8-6, the method further includes: obtaining the spatial length of the space where the head-mounted display device is located. For example, if the user is in a living room, the maximum length d of the living room can be used as the spatial length, and a and b can be determined based on the spatial length. Since the values ​​of a and b in the formula for predicting the observation depth affect the range of the predicted observation depth D, for example, when the spatial length is 5 meters, the reasonable range of the predicted observation depth D can be 1 meter to 3 meters, and thus the range of the values ​​of a and b can be deduced. It is understood that the maximum value of the predicted observation depth D can be 1 / 2 of the spatial length, or the spatial length minus a preset value (e.g., any value in the range of 1 to 5 meters, such as 1 meter, 1.8 meters, 3.5 meters, etc.). In this way, in most indoor scenarios, the virtual objects in the display screen can be basically located within the space.

[0109] Optionally, the integer value of a+b can fall within the range of [1~d / 2] or [1~(d-(0~5))], that is, a=[1~d / 2]-b or a=[1~(d-(0~5))]-b.

[0110] In one example of this disclosure, when the maximum length of the living room is 5 meters and the expected value of the predicted observation depth D is 1 to 3 meters, b can be set to 1 meter, and the range of a*Log2(P+1) is [0,2]. When the range of P is determined to be [0,1] based on P=h*θ, a can take the value 2.

[0111] In this embodiment, based on the spatial length of the space where the head-mounted display device is located, the values ​​of a and b in the formula for predicting the observation depth can be determined, thereby ensuring that the predicted observation depth is within a reasonable depth range that satisfies ergonomics.

[0112] In one embodiment of this disclosure, the attitude information further includes translation information in the horizontal direction of the reference coordinate system, and the method for predicting depth information further includes:

[0113] Based on spatial length and translation information, the distance Δd between the head-mounted display device and the cutting-off surface in the forward direction is determined. For example, using a SLAM algorithm, the translation information can be combined to determine the current position and orientation of the head-mounted display device in the space. Based on the current position and orientation of the head-mounted display device in the space, and combined with the spatial length, the distance between the head-mounted display device and the cutting-off surface (e.g., a wall) in the forward direction can be further determined.

[0114] a and b are determined based on the distance between the head-mounted display device and the cutoff surface in the forward direction.

[0115] Optionally, the integer value of a+b falls within the range of [1~Δd / 2] or [1~(Δd-(0~5))], that is, a=[1~Δd / 2]-b or a=[1~(Δd-(0~5))]-b.

[0116] In one example of this disclosure, the head-mounted display is located at the center of a living room with a maximum length of 10 meters and a maximum width of 5 meters. If the head-mounted display is oriented towards the direction of the maximum length of the living room based on its translation information, the furthest distance between the head-mounted display and the direction of the maximum length of the living room is 5 meters. If the expected value of the predicted observation depth D is between 1 meter and 3 meters, b can be set to 1 meter. Then, the range of a*Log2(P+1) is [0,2]. When the range of P is determined to be [0,1] based on P=h*θ, a can take the value 2.

[0117] In this embodiment, based on the spatial length and posture information of the space where the head-mounted display device is located, the distance between the head-mounted display device and the cutoff surface in the forward direction can be determined, thereby determining a reasonable depth range that satisfies ergonomics.

[0118] In one embodiment of this disclosure, the basic value in the vertical direction is one of a preset value in the vertical direction and a measured value in the vertical direction at a preset time. The preset time can be a time prior to step S2, for example, one second before executing step S2.

[0119] In this embodiment, a basic value in the vertical direction can be determined by setting or measuring so that a reasonable height of the head-mounted display device from the ground can be obtained based on the basic value in the vertical direction.

[0120] Any of the methods for predicting depth information provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the methods for predicting depth information provided in this disclosure can be executed by a processor, such as by a processor executing any of the methods for predicting depth information mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated upon below.

[0121] Exemplary apparatus

[0122] Figure 5 This is a structural block diagram of an apparatus for predicting depth information according to one embodiment of this disclosure. Figure 5 As shown, the apparatus for predicting depth information includes:

[0123] The posture information acquisition module 100 is used to acquire the posture information of the head-mounted display device. The posture information includes at least rotation information relative to the reference coordinate system and displacement information in the vertical direction, wherein the vertical direction of the reference coordinate system is aligned with the direction of gravity.

[0124] The ground clearance determination module 200 is used to determine the ground clearance of the head-mounted display device in the vertical direction based on the vertical displacement information and the vertical base value;

[0125] The observation angle determination module 300 is used to determine the observation angle of the head-mounted display device based on the rotation information, wherein the observation angle is the angle between the front direction of the head-mounted display device and the vertical direction of the reference coordinate system;

[0126] The depth prediction module 400 is used to predict depth information based on the ground height and the observation angle to obtain the predicted observation depth at the target time, wherein the predicted observation depth is the distance between the screen displayed at the target time and the head-mounted display device.

[0127] In one embodiment of this disclosure, the depth prediction module 400 is used to perform standardized numerical range processing on the ground height to obtain a ground height processing value for the head-mounted display device; the depth prediction module 400 is also used to perform standardized numerical range processing on the observation angle to obtain an observation angle processing value for the head-mounted display device; the depth prediction module 400 is also used to determine the predicted observation depth based on the ground height processing value and the observation angle processing value.

[0128] In one embodiment of this disclosure, the depth prediction module 400 is used to obtain the gaze point of the user wearing the head-mounted display device; the depth prediction module 400 is also used to obtain the distance between the gaze point and the head-mounted display device; the depth prediction module 400 is also used to determine the predicted observation depth based on the ground height processing value, the observation angle processing value, and the distance between the gaze point and the head-mounted display device.

[0129] In one embodiment of this disclosure, the depth prediction module 400 is used to determine the predicted observation depth using the following formula:

[0130] D = [a*Log2(P+1)+b]

[0131] P = h * θ

[0132] Where [a*Log2(P+1)+b] is the result of a*Log2(P+1)+b rounded down, where a and b are both given values, h is the processed value of the altitude above the ground, h∈[0,1], and θ is the processed value of the observation angle, θ∈[0,1].

[0133] In one embodiment of this disclosure, the depth prediction module 400 is further configured to obtain the spatial length of the space where the head-mounted display device is located; the depth prediction module 400 is further configured to determine a and b based on the spatial length.

[0134] In one embodiment of this disclosure, the attitude information further includes translation information in the horizontal direction of the reference coordinate system; the depth prediction module 400 is also used to determine a and b based on the distance between the head-mounted display device and the cutoff surface in the forward direction.

[0135] In one embodiment of this disclosure, the depth prediction module 400 is used to determine a basic observation depth value based on the processed value of the altitude above ground and the processed value of the observation angle; the depth prediction module 400 is further used to reduce the basic observation depth value to obtain the predicted observation depth if the basic observation depth value is greater than the maximum value of a preset observation depth; the depth prediction module 400 is further used to increase the basic observation depth value to obtain the predicted observation depth if the basic observation depth value is less than the maximum value of the preset observation depth and greater than the minimum value of the preset observation depth; the depth prediction module 400 is further used to determine the basic observation depth value as the predicted observation depth if the basic observation depth value is less than the maximum value of the preset observation depth and greater than the minimum value of the preset observation depth.

[0136] In one embodiment of this disclosure, the depth prediction module 400 is used to determine a predicted depth adjustment coefficient based on the distance between the gaze point and the head-mounted display device; the depth prediction module 400 is also used to determine the predicted observation depth based on the ground height processing value, the observation angle processing value, and the predicted depth adjustment coefficient.

[0137] In one embodiment of this disclosure, the basic value in the vertical direction is one of a preset value in the vertical direction and a measured value in the vertical direction at a preset time.

[0138] It should be noted that the specific implementation of the apparatus for predicting depth information in this disclosure is similar to the specific implementation of the method for predicting depth information in this disclosure. For details, please refer to the method for predicting depth information section. To reduce redundancy, it will not be described again.

[0139] Exemplary electronic device

[0140] Below, for referenceFigure 6 To describe an electronic device according to embodiments of this disclosure. For example... Figure 6 As shown, the electronic device includes one or more processors 10 and memory 20.

[0141] The processor 10 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0142] The memory 20 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 10 may execute the program instructions to implement the methods for predicting depth information and / or other desired functions described in the various embodiments of this disclosure above. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0143] In one example, the electronic device may also include an input device 30 and an output device 40, these components being interconnected via a bus system and / or other forms of connection mechanism (not shown). The input device 30 may be, for example, a keyboard, a mouse, etc. The output device 40 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, etc.

[0144] Of course, for the sake of simplicity, Figure 6 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0145] Exemplary computer-readable storage medium

[0146] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0147] Furthermore, this disclosure also provides a head-mounted display device, including the means for predicting depth information described in the above embodiments.

[0148] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0149] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0150] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0151] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0152] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0153] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0154] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for predicting depth information, comprising: Acquire the posture information of the head-mounted display device, wherein the posture information includes at least rotation information relative to a reference coordinate system and displacement information in the vertical direction, and the vertical direction of the reference coordinate system is aligned with the direction of gravity; Based on the vertical displacement information and the vertical base value, the height of the head-mounted display device above the ground in the vertical direction is determined; Based on the rotation information, the observation angle of the head-mounted display device is determined, wherein the observation angle is the angle between the front direction of the head-mounted display device and the vertical direction of the reference coordinate system; Depth information is predicted based on the ground height and the observation angle to obtain the predicted observation depth of the head-mounted display device at the target time, wherein the predicted observation depth is the distance between the image displayed at the target time and the head-mounted display device; The step of predicting depth information based on the ground elevation and the observation angle to obtain the predicted observation depth of the head-mounted display device at the target time includes: The ground clearance is standardized to obtain the ground clearance value of the head-mounted display device. The observation angle is standardized to obtain the processed observation angle value of the head-mounted display device; The predicted observation depth is determined based on the processed value of the altitude above the ground and the processed value of the observation angle. Determining the predicted observation depth based on the processed value of the altitude above the ground and the processed value of the observation angle includes: The predicted observation depth is determined using the following formula: D = [a*Log2(P+1)+b] P = h * θ Where [a*Log2(P+1)+b] is the result of a*Log2(P+1)+b rounded down, where a and b are both given values, h is the processed value of the altitude above the ground, h∈[0,1], and θ is the processed value of the observation angle, θ∈[0,1].

2. The method according to claim 1, wherein, Determining the predicted observation depth based on the processed value of the altitude above the ground and the processed value of the observation angle includes: Obtain the gaze point of the user wearing the head-mounted display device; Obtain the distance between the gaze point and the head-mounted display device; The predicted observation depth is determined based on the processed value of the ground elevation, the processed value of the observation angle, and the distance between the gaze point and the head-mounted display device.

3. The method according to claim 1, wherein, Before determining the predicted observation depth based on the processed value of the altitude above the ground and the processed value of the observation angle, the method further includes: Obtain the spatial length of the space where the head-mounted display device is located; Based on the spatial length, determine a and b.

4. The method according to claim 3, wherein, The attitude information also includes translation information in the horizontal direction of the reference coordinate system, and the method further includes: Based on the spatial length and the translation information, the distance between the head-mounted display device and the cutoff surface in the forward direction is determined; a and b are determined based on the distance between the head-mounted display device and the cutoff surface in the forward direction.

5. The method according to claim 1, wherein, Determining the predicted observation depth based on the processed value of the altitude above the ground and the processed value of the observation angle includes: Based on the processed value of the altitude above the ground and the processed value of the observation angle, the basic observation depth value is determined; If the base observation depth value is greater than the maximum value of the preset observation depth, the base observation depth value is reduced numerically to obtain the predicted observation depth. If the basic observation depth value is less than the minimum value of the preset observation depth, the basic observation depth value is numerically increased to obtain the predicted observation depth. If the basic observation depth value is less than the maximum value of the preset observation depth but greater than the minimum value of the preset observation depth, the basic observation depth value is determined as the predicted observation depth.

6. The method according to claim 2, wherein, Determining the predicted observation depth based on the processed value of the ground elevation, the processed value of the observation angle, and the distance between the gaze point and the head-mounted display device includes: The predicted depth adjustment coefficient is determined based on the distance between the gaze point and the head-mounted display device; The predicted observation depth is determined based on the processed value of the altitude above the ground, the processed value of the observation angle, and the predicted depth adjustment coefficient.

7. The method according to any one of claims 1-6, wherein, The basic value in the vertical direction is one of the preset value in the vertical direction and the measured value in the vertical direction at a preset time.

8. An apparatus for predicting depth information, comprising: The posture information acquisition module is used to acquire the posture information of the head-mounted display device. The posture information includes at least rotation information relative to the reference coordinate system and displacement information in the vertical direction, wherein the vertical direction of the reference coordinate system is aligned with the direction of gravity. The ground clearance determination module is used to determine the ground clearance of the head-mounted display device in the vertical direction based on the vertical displacement information and the vertical base value; An observation angle determination module is used to determine the observation angle of the head-mounted display device based on the rotation information, wherein the observation angle is the angle between the front direction of the head-mounted display device and the vertical direction of the reference coordinate system; A depth prediction module is used to predict depth information based on the ground height and the observation angle to obtain the predicted observation depth at the target time, wherein the predicted observation depth is the distance between the screen displayed at the target time and the head-mounted display device; The depth prediction module is used to standardize the ground elevation to obtain a ground elevation value for the head-mounted display device; standardize the observation angle to obtain an observation angle value for the head-mounted display device; and determine the predicted observation depth based on the ground elevation value and the observation angle value. The depth prediction module is used to determine the predicted observation depth using the following formula: D = [a*Log2(P+1)+b] P = h * θ Where [a*Log2(P+1)+b] is the result of a*Log2(P+1)+b rounded down, where a and b are both given values, h is the processed value of the altitude above the ground, h∈[0,1], and θ is the processed value of the observation angle, θ∈[0,1].

9. A head-mounted display device comprising the means for predicting depth information as described in claim 8.

10. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for predicting depth information as described in any one of claims 1-7.

11. A computer-readable storage medium storing a computer program for performing the method for predicting depth information as described in any one of claims 1-7.

Images