Method, apparatus, and electronic device for displaying a screen on a head-mounted display device

Abstract

The present application discloses a method, an apparatus, and an electronic device for displaying a screen on a head-mounted display device. As a specific implementation means, based on the angular velocity data of the current time of the head-mounted display device and the first corrected posture for processing the display screen of the head-mounted display device at the previous time, a step (110) of determining a reference posture for processing the screen to be displayed at the current time; a step (120) of processing the posture error between the reference posture and a predetermined posture for processing the display screen to obtain a corrected posture difference value; a step (130) of compensating the reference posture with the corrected posture difference value to obtain a second corrected posture for processing the screen to be displayed at the current time; and a step (140) of displaying the screen to be displayed on the head-mounted display device by rendering the screen to be displayed on the head-mounted display device based on the second corrected posture.

Description

Technical Field

[0001] This disclosure claims the priority of a Chinese patent application filed with the China National Intellectual Property Administration on May 11, 2022, with the application number CN202210510750.8 and the invention title "Method, apparatus, and electronic device for displaying a screen on a head-mounted display device", and all of its content is incorporated herein by reference.

[0002] This disclosure relates to the technical field of head-mounted display devices, and in particular, to a method, an apparatus, and an electronic device for displaying a screen on a head-mounted display device.

Background Art

[0003] Currently, the application of head-mounted display devices is becoming increasingly widespread. During the use of a head-mounted display device, pitching, shaking, etc. may occur, which will affect the display effect of the head-mounted display device. How to ensure the display effect of the head-mounted display device and improve the user experience is an urgent problem to be solved by those skilled in the art.

Summary of the Invention

Means for Solving the Problems

[0004] Embodiments of this disclosure provide a method, an apparatus, and an electronic device for displaying a screen on a head-mounted display device.

[0005] According to one aspect of the present disclosure, a method for displaying a screen on a head-mounted display device is provided. The method includes: determining a reference posture for processing a screen to be displayed at the current time based on the angular velocity data at the current time of the head-mounted display device and a first corrected posture for processing the display screen of the head-mounted display device at the previous time; processing the posture error between the reference posture and a predetermined posture for processing the display screen to obtain a corrected posture difference value; compensating the reference posture with the corrected posture difference value to obtain a second corrected posture for processing the screen to be displayed at the current time; and displaying the screen to be displayed on the head-mounted display device by rendering the screen to be displayed on the head-mounted display device based on the second corrected posture.

[0006] According to another aspect of the present disclosure, a computer-readable storage medium storing a computer program for executing the above method for displaying a screen on the head-mounted display device is provided.

[0007] According to a further aspect of the present disclosure, an electronic device is provided, including a processor and a memory for storing instructions executable by the processor. The processor reads executable instructions from the memory and is used to execute the instructions to implement the above method for displaying a screen on the head-mounted display device.

[0008] Hereinafter, with reference to the drawings and embodiments, the technical solution of the present disclosure will be described in more detail.

Brief Description of the Drawings

[0009] The above and other objects, features, and advantages of the present disclosure will become more apparent by describing embodiments of the present disclosure in more detail with reference to the drawings. The drawings are provided to further understand the embodiments of the present disclosure, constitute a part of the specification, are used together with the embodiments of the present disclosure to explain the present disclosure, and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same members or steps.

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11-1

Figure 11-2

Figure 11-3

Figure 11-4

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Modes for Carrying Out the Invention

[0011] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the drawings. It should be clearly understood that the described embodiments are only some embodiments of the present disclosure, not all embodiments of the present disclosure, and the present disclosure is not limited to the exemplary embodiments described herein.

[0012] Note that, unless specifically described otherwise, the relative arrangements, mathematical formulas, and numerical values of the members and steps described in these embodiments do not limit the scope of the present disclosure.

[0013] Those skilled in the art can understand that terms such as "first", "second", etc. in the embodiments of the present disclosure are only for distinguishing different steps, devices, or modules, etc., and do not represent a specific technical meaning, nor do they represent an inevitable logical order between them.

[0014] Also, in the embodiments of the present disclosure, it should be understood that "a plurality of" can refer to two or more, and "at least one" can refer to one, two, or two or more.

[0015] Also, for any member, data, or structure mentioned in the embodiments of the present disclosure, unless specifically limited or there is no contrary suggestion from the context, it can generally be understood as one or more.

[0016] In addition, the term "and / or" in the present disclosure is merely a relationship for describing related objects, indicating that three relationships can exist. For example, A and / or B can indicate three situations: only A exists, A and B exist simultaneously, and only B exists. Also, the symbol " / " in this application generally indicates that the related objects before and after are in an "or" relationship.

[0017] In addition, the description of each embodiment in the present disclosure focuses on highlighting the differences between the embodiments. It should be understood that the same or similar points can be referred to each other, and detailed descriptions are omitted for the sake of brevity.

[0018] The following description of at least one exemplary embodiment is actually merely illustrative and does not limit the present disclosure and its application or use in any way.

[0019] Regarding technologies, methods, and devices known to those skilled in the art, although detailed descriptions may not be provided, when appropriate, such technologies, methods, and devices should be regarded as part of the specification.

[0020] It should be noted that similar reference numerals and characters represent similar items in the following drawings. Therefore, if an item is defined in a certain drawing, there is no need to further consider it in subsequent drawings.

[0021] Embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., and can operate in conjunction with many other general-purpose or dedicated computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable household appliances, networked personal computers, mini-computer systems, mainframe computer systems and distributed cloud computing technology environments including any of these systems, etc., but are not limited thereto.

[0022] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system-executable instructions (e.g., program modules) executed by a computer system. Generally, program modules can include routines, programs, target programs, components, logic, data configurations, etc. that perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment, in which tasks are executed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can be located on the storage media of a local or remote computing system including a storage device.

[0023] Exemplary method FIG. 1 is a flowchart of a method for displaying a screen on a head-mounted display device provided by an exemplary embodiment of the present disclosure. The method shown in FIG. 1 can include step 110, step 120, step 130, and step 140, and each step will be described below.

[0024] In step 110, based on the angular velocity data of the current time of the head-mounted display device and the first corrected posture for processing the display screen of the head-mounted display device at the previous time, a reference posture for processing the screen to be displayed at the current time is determined.

[0025] Note that the head-mounted display device may also be referred to as a Head-Mounted Display (HMD) or a head display. The head-mounted display device can be used to realize cross-reality (XR) effects such as augmented reality (AR) effects, virtual reality (VR) effects, and mixed reality (MR) effects. Optionally, the head-mounted display device may be used to realize the AR effect. In this case, the head-mounted display device may be AR glasses.

[0026] Generally, the head-mounted display device can include an inertial measurement unit (IMU) and an optical device. The IMU can include an accelerometer, a gyroscope, a magnetometer, and the like. The accelerometer is used to collect acceleration data, and the gyroscope is used to collect angular velocity data. The optical device is the imaging system of the head-mounted display device, and the optical device can include a display screen and an optical element.

[0027] Generally, the user cannot directly view the display screen of the head-mounted display device, and the display screen is provided to the user by the following solution. The display screen is used to emit the light of the display screen, and the optical element can process the above light so that the light of the display screen is projected onto the user's eyes and finally the user can view the display screen. On the other hand, while the user is wearing the head-mounted display device, there is almost no relative displacement or rotation between the head-mounted display device and the user, so the head-mounted display device moves with the movement of the user. Therefore, in order to adapt to the corresponding application scenario or user needs, the display screen needs to be processed based on the position and posture of the head-mounted display device (which can actually be regarded as the user) so that the display screen is always positioned within a certain position or area, or can move according to the movement of the user.

[0028] Before step 110, the angular velocity data of the current time of the IMU, for example, the angular velocity data collected by the gyro in the IMU, can be obtained, and the first corrected posture for processing the display screen of the head-mounted display device at the previous time can be obtained. The first corrected posture is obtained in the same way as the following second corrected posture, but for the sake of clarity of the layout, the acquisition method of the first corrected posture will not be described in detail here.

[0029] In step 110, by referring to the integration method from the angular velocity of the gyro to the angular position and performing integration processing using the angular velocity data of the current time and the first corrected posture, a reference posture for processing the screen to be displayed at the current time (the screen to be displayed may be the screen to be displayed by the head-mounted display device) can be obtained. If the angular velocity data of the current time is Gyro k , the first corrected posture is Pose k-1 , and the reference posture is

Number

Number

[0030] In step 120, the posture error between the reference posture and a predetermined posture for processing the display screen is processed to obtain a corrected posture difference value.

[0031] Note that the predetermined posture may be a posture preset in the head-mounted display device that can ensure the display effect of the head-mounted display device. That is, the predetermined posture may be a posture for processing the display screen to ensure the display effect.

[0032] In an example that can be selected, the predetermined posture is a posture for displaying the display screen of the head-mounted display device at a predetermined position in the coordinate system of the head-mounted display device. It can be understood that it may also be a predetermined position in the IMU coordinate system.

[0033] The predetermined position in the coordinate system of the head-mounted display device may be a fixed position in the coordinate system of the head-mounted display device. For example, it can be understood that the display screen is displayed at a position visible to the user, such as the front, upper left, or lower right of the head-mounted display device. Note that the usage habits of many users are to display the display screen at a position centered in front. In the embodiments of the present disclosure, by setting the predetermined posture to a posture that conforms to the usage habits of many users, the display effect of the head-mounted display device can satisfy the requirements of as many users as possible, which is advantageous for improving the user experience.

[0034] In an example that can be selected, the predetermined posture is a posture for aligning the display screen of the head-mounted display device with the direction of gravity (in this posture, the yaw angle of the head-mounted display device may be 0).

[0035] That the display screen of the head-mounted display device is aligned with the gravitational direction means that when the user views the display screen through the head-mounted display device, it can be understood that the extension direction of the plane of the display screen is the same as the gravitational direction in the real world. As a result, the display screen is kept substantially perpendicular to the user's horizontal line of sight, conforming to the viewing habit. Note that the usage habit of many users is to align the display screen of the head-mounted display device with the gravitational direction. In the embodiments of the present disclosure, by setting the predetermined posture to be a posture that conforms to the usage habits of many users, the display effect of the head-mounted display device can satisfy the requirements of as many users as possible, which is advantageous for improving the user experience.

[0036] Of course, the user may set a predetermined posture in which the display screen of the head-mounted display device can be displayed at a customized position as needed.

[0037] In step 120, a difference can be made between the reference posture and the predetermined posture so as to obtain the posture error. The posture error can be in the form of a 3×1 axis angle. Note that making a difference here is not a simple subtraction but a broad function, and the "difference" in making a difference refers to a difference, and making a difference may refer to obtaining the relative rotation between two angles. If the reference posture is

Number

Number

[0038] After obtaining the posture error, by processing the posture error to realize the correction of the posture error, a corrected posture difference value can be obtained.

[0039] It can be understood that the predetermined posture may be a posture preset in the head-mounted display device. When the predetermined posture and the reference posture are postures in the same coordinate system and the predetermined posture is a zero-rotation posture, for the sake of process saving, the step of differentiating between the reference posture and the predetermined posture may be eliminated, and the reference posture may be directly used as the subsequent processing target amount. That is, in the subsequent steps, the reference posture is directly processed, and by realizing the correction of the posture error, a corrected posture difference value is obtained.

[0040] In step 130, the reference posture is compensated with the corrected posture difference value to obtain a second corrected posture for processing the screen to be displayed at the current time.

[0041] In step 130, the corrected posture difference value can be updated as a compensation value to the reference posture so as to obtain a second corrected posture for processing the screen to be displayed at the current time. Let the second calibrated posture be Pose k and it can be expressed as

Equation

[0042] Note that the correction value here may represent a relative rotation, and updating the correction value to the reference posture may be to obtain another posture (i.e., the second corrected posture) by superimposing a relative rotation based on the reference posture.

[0043] It can be understood that the corrected posture difference value is for updating the reference posture so as to obtain the second corrected posture. Whether the corrected posture difference value is obtained by processing the reference posture or by processing the posture error between the reference posture and the predetermined posture, the second corrected posture is updated based on the corrected posture difference value.

[0044] In step 140, based on the second corrected posture, the screen to be displayed on the head-mounted display device is rendered to display the screen to be displayed on the head-mounted display device.

[0045] It can be understood that both the first corrected posture and the second corrected posture are postures for processing the display screen of the head-mounted display device, and may be referred to as corrected postures. For example, when the head-mounted display device executes this method for the first time without the corrected posture at the previous time, the corrected posture at the previous time may be adopted as the true posture of the head-mounted display device, a predetermined posture for processing the display screen, or other preset postures, etc.

[0046] Note that based on the second corrected posture or the corrected posture, there are various specific implementation forms for processing the display screen of the head-mounted display device, and examples will be given and described below.

[0047] In an alternative embodiment, the rendering engine of the head-mounted display device can perform screen rendering using the second corrected posture or the corrected posture so as to obtain the corresponding display screen, and the obtained display screen is updated and displayed on the optical device of the head-mounted display device, thereby realizing the correction of the display screen of the head-mounted display device. It is easy to understand that such an implementation form realizes the processing of the display screen of the head-mounted display device by software. For example, this solution can adjust the overall parameters of the screen to be displayed based on the second corrected posture or the corrected posture, and the overall parameters can include the three-dimensional size of the display screen, etc.

[0048] In another alternative embodiment, as shown in FIG. 2, the second corrected posture (for example, Pose in FIG. 2) kOr perform screen rendering based on the corrected posture, and adjust the display position on the display screen of the light device of the head-mounted display device (for example, Display in FIG. 2), so as to realize the correction of the display screen of the head-mounted display device. By adopting such an implementation form, the processing of the display screen of the head-mounted display device can be realized. For example, the display screen of the head-mounted display device can adjust the display position up, down, left, and right. For example, this solution can adjust the two-dimensional size of the screen to be displayed based on the second corrected posture or the corrected posture.

[0049] In a further selectable implementation form, as shown in FIG. 3, while performing screen rendering based on the second corrected posture or the corrected posture, the light device in the head-mounted display device can be arranged. For example, by adjusting the overall position of the light device in the head-mounted display module with an electronic control box or adjusting the position of the display screen in the light device, the correction of the display screen of the head-mounted display device can be realized (this is equivalent to performing corresponding vibration damping adjustment with the display screen in front of the user's eyes). This implementation form realizes the processing of the display screen of the head-mounted display device in a hardware manner, and it is easy to understand that the execution logic of this implementation form may be the rotation of the second corrected posture → electronic control box → light device.

[0050] In an embodiment of the present disclosure, by combining the angular velocity data of the head-mounted display device at the current time with the first corrected posture for processing the display screen of the head-mounted display device at the previous time, a reference posture for processing the screen to be displayed at the current time can be determined. Next, the posture error between the reference posture and the predetermined posture for processing the display screen is processed, or the reference posture is processed to obtain a corrected posture difference value, and the reference posture is corrected with the corrected posture difference value to obtain a second corrected posture for processing the screen to be displayed at the current time. The second corrected posture is considered to be determined with reference to the angular velocity data of the head-mounted display device at the current time, the first corrected posture, and the predetermined posture, and is adapted to the posture of the head-mounted display device at the current time (which may actually be regarded as the user). Rendering the screen to be displayed by the head-mounted display device based on this posture helps to correct the display effect of the head-mounted display device based on the display effect in the predetermined posture. In this way, even when a small angular change occurs during the use of the head-mounted display device (such as pitching, swaying, vibration, etc.) or a large angular change occurs during the use of the head-mounted display device (for example, bending at a large angle), it is possible to ensure that the display effect is adjusted so that the head-mounted display device corresponds to the predetermined posture, thereby improving the user experience.

[0051] Optionally, each posture according to an embodiment of the present disclosure may all be in the form of a rotation matrix. Each corrected posture according to an embodiment of the present disclosure may all be a posture for screen rendering used in the rendering engine of the head-mounted display device.

[0052] In addition, when a head-mounted display device displays content, there are a plurality of modes in the current existing modes, and the first mode and the second mode can be included. For example, the first mode may be a three-degree-of-freedom (dof) mode. In the first mode, the head-mounted display device performs a process of positioning the display screen. At this time, the content on the display screen of the head-mounted display device is related to the user's posture, and the display screen of the head-mounted display device may remain rotated in the world coordinate system. The second mode is similar to the projection mode from a mobile phone to a television. In the second mode, the head-mounted display device does not perform a process related to aligning the display screen. At this time, the content on the display screen of the head-mounted display device can be related to the progress of content playback regardless of the user's posture, and the display screen of the head-mounted display device moves according to the movement of the user's head. For the convenience of explanation, the second mode may be called a zero-degree-of-freedom (dof) mode.

[0053] It can be understood that the user can use the head-mounted display device on a vehicle, a ship, an airplane, or other types of mobile platforms. In the 0dof mode, if the operation of the mobile platform is stable, the display effect of the head-mounted display device is still acceptable. When pitching or swaying occurs, for example, because the road surface is not flat, a running vehicle pitches or sways, or, for example, due to the influence of air currents, an airplane in flight pitches or sways, the user wearing the head-mounted display device sways or shakes. In this 0dof mode, since the display screen also sways or shakes accordingly, in this case, the user cannot stably and clearly observe the content on the display screen of the head-mounted display device, and the user experience deteriorates.

[0054] In the 3DOF mode, when the above pitching and swaying occur, the process of positioning the display screen is performed. Therefore, even if the head-mounted display device shakes or sways, the positioning of the display screen does not change, and the user can clearly view the content of the display screen of the head-mounted display device. However, when the orientation of the mobile platform changes (for example, when it rotates 90 degrees), since the user (head-mounted display device) also changes in the same orientation along with the mobile platform, if the positioning of the display screen does not change, the user needs to change to the posture before the orientation change to observe the required display screen. Doing so will reduce the user experience.

[0055] Therefore, according to the embodiments of the present disclosure, it is possible to realize correcting the display effect of the head-mounted display device according to the display effect in a predetermined posture of the head-mounted display device. For example, by processing the display screen of the head-mounted display device according to the correction posture, when the change is at a small angle during the use of the head-mounted display device, a display effect similar to the 3DOF mode can be obtained, and when the change is at a large angle, a display effect similar to the 0DOF mode can be obtained, thereby improving the user experience.

[0056] In an example that can be selected, the head-mounted display device is coupled to the installed mobile platform and is movably arranged with respect to the mobile platform.

[0057] Note that the head-mounted display device can be coupled to a mobile platform installed via a user, and the mobile platform may be in a moving state. Optionally, as shown in FIG. 4, the head-mounted display device can be worn on the head of a user (e.g., a passenger in a vehicle), and the relative position between the head-mounted display device and the user can be kept stationary. The user can sit or stand inside the vehicle, and when the vehicle moves, the user moves with the vehicle. The user can further move inside the vehicle (e.g., move from one position to another position, or move from one direction to another direction). Also, when the mobile platform is of another type such as a ship, the user can sit or stand on the ship's edge.

[0058] Note that when the head-mounted display device is coupled to a mobile platform on which it is installed and is movably arranged relative to the mobile platform, during movement, the mobile platform may pitch or sway, or the mobile platform may change direction. In view of this, by executing the method steps of the present disclosure in this case, the display effect of the head-mounted display device can be ensured, and the user experience can be improved.

[0059] Regarding the shake correction of the display screen of a mobile phone (or computer) on the mobile platform, currently, there is a solution where the user manually adjusts the relative position and angle between the actual display screen of the mobile phone and the user's eyes. That is, the actual display screen of the mobile phone is displaced and rotated relative to the user's eyes. In comparison, when the shake correction solution in the embodiments of the present disclosure is adopted, the head-mounted display device hardly displaces or rotates relative to the user's eyes.

[0060] Based on the embodiment shown in FIG. 1, as shown in FIG. 5, step 120 includes the following. In step 1201, a low pass filter process is performed on the attitude error corresponding to the current time to obtain a corrected attitude difference value.

[0061] Note that the attitude error obtained by taking the difference between the reference attitude determined in step 110 and the predetermined attitude is the attitude error corresponding to the current time. And when the predetermined attitude and the reference attitude are attitudes in the same coordinate system and the predetermined attitude is an attitude of zero rotation, a low pass filter process can be performed on the reference attitude to obtain a corrected attitude difference value.

[0062] Optionally, a low pass filter such as an infinite impulse response (IIR) digital filter or a finite impulse response (FIR) digital filter may perform a low pass filter process on the attitude error corresponding to the current time. Each output value of the IIR digital filter can depend only on the previous one sampling value, so the application of the process is very convenient. The low pass filter process of the FIR digital filter relates to a sliding window, and each output value of the FIR digital filter is obtained by calculating several previous sample values. If the attitude error corresponding to the current time is represented as error and the corrected attitude difference value is represented as error 1 then,

Equation

[0063] Ideally, when only data below the cut-off frequency can pass, the value of the corrected attitude difference value error 1 is not zero, that is, subsequently, it can be understood that the second corrected attitude after compensating the reference attitude with the corrected attitude difference value error 1 is different from the reference attitude. However, if data higher than the cut-off frequency cannot pass, the corrected attitude difference value error 1The value of 1 is 0, that is, subsequently, the second corrected posture after compensating the reference posture with the corrected posture difference value error 1 is the same as the reference posture. However, in the actual situation, it is possible that the value of the corrected posture difference value error

[0064] is not 0, and it may be a very small value that is not noticed by the user. In an embodiment of the present disclosure, as shown in FIG. 6, after acquiring the angular velocity data Gyro k of the current time collected by the gyro in the IMU, the angular velocity data Gyro k of the current time and the first corrected posture Pose k-1 for processing the display screen of the head-mounted display device at the previous time are integrated to obtain the reference posture

Number

Number

Number

[0065] In addition to the embodiment shown in FIG. 5, as shown in FIG. 7, step 1201 includes the following: In step 12011, low-pass filter processing is performed on the pose error corresponding to the current time by a low-pass filter.

[0066] The method further includes step 150 and step 160.

[0067] In step 150, an input operation by the user is received.

[0068] Optionally, the input operation by the user includes, but is not limited to, voice input operations, touch input operations, etc. For example, the input operation by the user may be an operation where the user says in voice "Adjust the cut-off frequency to 2 hz (it may also be 5 hz or other frequencies)". Alternatively, the head-mounted display device may be provided with a paired handle, and the input operation by the user may be an operation where the user clicks a cut-off frequency adjustment button on the handle. Alternatively, a cut-off frequency selection box or an input box may be displayed on the display screen, and the user can directly select or input the required cut-off frequency.

[0069] In step 160, in response to the input operation by the user, the cut-off frequency of the low-pass filter is adjusted.

[0070] In step 160, in response to the input operation by the user, after the user determines the required cut-off frequency, in any feasible form, the cut-off frequency of the low-pass filter can be adjusted to the cut-off frequency required by the user.

[0071] As can be seen from the above description, through the low-pass filtering process of the low-pass filter, when the moving frequency of the head-mounted display device is equal to or lower than the cut-off frequency of the low-pass filter, the display screen of the head-mounted display device changes following the movement of the head-mounted display device. When the moving frequency of the head-mounted display device is higher than the cut-off frequency of the low-pass filter, the display screen of the head-mounted display device tends not to move. This method may not be as sensitive to movements closer to the cut-off frequency. Thus, it can be understood that the higher the moving frequency of the head-mounted display device is above the cut-off frequency, the closer the actual mode of the head-mounted display device is to the 3dof mode. The lower the moving frequency of the head-mounted display device is below the cut-off frequency, the closer the actual mode of the head-mounted display device is to the 0dof mode. Therefore, the higher the cut-off frequency, the more likely the frequencies of most movements are lower than this cut-off frequency. Thus, with this solution, when compensation processing is performed to make the display screen follow the movement of the head-mounted display device for most movements of the head-mounted display device, it can be understood that the actual mode of the head-mounted display device approaches the 0dof mode. On the other hand, the lower the cut-off frequency, the more likely the frequencies of most movements are higher than this cut-off frequency. Thus, with this solution, when correction processing is performed to make the display screen not move for most movements of the head-mounted display device, the actual mode of the head-mounted display device approaches the 3dof mode.

[0072] In an embodiment of the present disclosure, by adjusting the cut-off frequency of the low-pass filter, a user can make the actual mode of the head-mounted display device approach the 0dof mode or the 3dof mode according to actual needs.

[0073] In the above description, the case where the cut-off frequency of the low-pass filter is adjustable has been explained. However, when specifically implemented, the cut-off frequency of the low-pass filter can also be set to a fixed value that cannot be adjusted.

[0074] In addition to the embodiment shown in FIG. 1, as shown in FIG. 8, step 120 includes the following: In step 1203, the attitude error corresponding to the current time is corrected with a proportional correction coefficient greater than 0 and less than 1 to obtain a corrected attitude difference value.

[0075] Optionally, the proportional correction coefficient may be 0.2, 0.5, 0.6, 0.8, or other values between 0 and 1, but they will not be listed one by one here.

[0076] It can be understood that when the predetermined attitude and the reference attitude are attitudes in the same coordinate system and the predetermined attitude is an attitude of zero rotation, the reference attitude may be corrected with a proportional correction coefficient to obtain a corrected attitude difference value.

[0077] In step 1203, the proportional correction coefficient can be multiplied by the attitude error corresponding to the current time, and the multiplication result can be used as the corrected attitude difference value.

[0078] When there is vibration or shaking, it can be understood that the user moves their head back and forth, and there are plus and minus values in the actual posture of the head-mounted display device, and there are also plus and minus values in the posture error. After compensation, the display screen of the head-mounted display device tends to stop moving, and thereby, the user can stably and clearly observe the content of the display screen of the head-mounted display device, and thereby, the user experience can be improved. Further, when the user turns their head or the mobile platform changes direction (for example, when the mobile platform changes direction, the user's head is also changed in direction accordingly), when a movement of direction change occurs in the head-mounted display device, the actual posture of the head-mounted display device is always plus or minus, and the posture error is also always plus or minus. After compensation, the display screen of the head-mounted display device follows the direction of the user's head so that the user can easily observe the required display screen, and the user experience is also improved.

[0079] It should be noted that the larger the proportional compensation coefficient is, the closer the corrected posture difference value and the posture error are. When the corrected posture difference value is used to determine the second corrected posture, the second corrected posture is closer to the predetermined posture. Accordingly, the correction speed of the display screen of the head-mounted display device becomes faster. In this way, the larger the proportional compensation coefficient is, the closer the actual mode of the head-mounted display device is to the 0dof mode, and the smaller the proportional compensation coefficient is, the closer the actual mode of the head-mounted display device is to the 3dof mode.

[0080] In an embodiment of the present disclosure, by introducing a complementary proportionality coefficient to correct the posture error, the corrected posture of the head-mounted display device can be made closer to a predetermined posture, and using the corrected posture for rendering the display screen of the head-mounted display device helps to correct the display effect of the head-mounted display device based on the display effect in the predetermined posture of the head-mounted display device. In this way, even when a small angle change occurs during the use of the head-mounted display device (such as pitching, swaying, rolling, etc.) or when a large angle change occurs during the use of the head-mounted display device (for example, bending at a large angle), it is possible to ensure that the display effect is adjusted so that the head-mounted display device corresponds to the predetermined posture, thereby improving the user experience. Also, by controlling the complementary proportionality coefficient, the user can bring the actual mode of the head-mounted display device closer to the 0dof mode or the 3dof mode according to actual needs.

[0081] In addition to the embodiment shown in FIG. 8, as shown in FIG. 9, this method further includes the following: In step 115, a complementary proportionality coefficient is determined based on compensation reference information including at least one of a posture error corresponding to the current time, an integrated value of the posture error, an environmental situation where the mobile platform on which the head-mounted display device is installed is placed, and direction conversion information of the mobile platform.

[0082] Optionally, step 115 may be executed before step 120, that is, after determining the proportional compensation coefficient based on the compensation reference information, the attitude error may be corrected with the determined proportional compensation coefficient. Alternatively, step 115 may be executed after step 120, that is, after correcting the attitude error with the proportional compensation coefficient, the proportional compensation coefficient may be determined based on the compensation reference information. Thereby, using the determination result, dynamic update of the proportional compensation coefficient is performed. In this way, the updated proportional compensation coefficient can be subsequently used for correcting the attitude error. The embodiments of the present disclosure do not limit the execution order of step 115 and step 120 in any way. Therefore, it can be understood that the proportional compensation coefficient in the embodiments of the present disclosure may be dynamically adjusted based on the above factors.

[0083] It can be understood that when the predetermined attitude and the reference attitude are attitudes in the same coordinate system and the predetermined attitude is an attitude of zero rotation, the integrated value of the attitude error corresponding to the current time and the attitude error may be understood as the integrated value of the reference attitude corresponding to the current time and the reference attitude.

[0084] The environmental situation where the mobile platform on which the head-mounted display device is installed is placed can be determined based on the information collected by the devices (such as cameras, radars, etc.) installed on the platform. When the mobile platform on which the head-mounted display device is installed is a vehicle, the environmental state where the mobile platform on which the head-mounted display device is installed is placed can include the road surface information of the road surface where the vehicle is located, and the road surface information can include the road surface flatness. When the mobile platform on which the head-mounted display device is installed is an airplane, the environmental state where the mobile platform on which the head-mounted display device is installed is placed can include the strength of the airflow at the location where the airplane is located.

[0085] The direction conversion information of the mobile platform on which the head-mounted display device is installed can include the degree of direction conversion, and the degree of direction conversion can be expressed by a direction conversion angle. When the mobile platform is a vehicle, the direction conversion information is obtained by being detected by an in-vehicle direction conversion system.

[0086] Optionally, the proportional compensation coefficient shows a positive correlation with the posture error corresponding to the current time.

[0087] The greater the difference between the reference posture and the predetermined posture, the greater the proportional compensation coefficient. The greater the proportional compensation coefficient, the more advantageous it is to more quickly correct the display screen of the head-mounted display device based on the predetermined posture, thereby being advantageous for ensuring the display effect of the head-mounted display device.

[0088] Optionally, the proportional compensation coefficient shows a positive correlation with the integrated value of the posture error. The integrated value of the posture error may be the integrated value of the posture error within a preset period or the integrated value of the continuous posture error. Compared with the above example, the proportional compensation coefficient is determined by the overall movement tendency before the screen display, and the proportional compensation coefficient can be adjusted according to the actual posture error. For example, when the mobile platform makes a large-angle turn, the display screen of the head-mounted display device can be quickly corrected based on the predetermined posture, thereby being advantageous for ensuring the display effect of the head-mounted display device.

[0089] It can be understood that the value of the posture error can be either a positive number or a negative number. For example, if the value of the posture error in the first direction is defined as a positive number, the value of the posture error in the second direction opposite to the first direction is a negative number. Both the positive posture error and the negative posture error can be processed by arithmetic operations. When evaluating the "large", "small", and "positive correlation" of the posture error, the absolute value of the posture error can be referred to for evaluation.

[0090] Optionally, the environmental conditions include road surface flatness, and the proportional correction coefficient shows a positive correlation with the road surface flatness. In this way, the greater the unevenness of the road surface, the smaller the proportional correction coefficient. With a smaller proportional correction coefficient, the speed at which the display screen of the head-mounted display device is corrected based on a predetermined posture becomes slower, which is advantageous for ensuring the stability of the display screen of the head-mounted display device.

[0091] Optionally, the direction change information includes the degree of direction change, and the proportional correction coefficient shows a positive correlation with the degree of direction change. In this way, the greater the change in the orientation of the mobile platform, the larger the proportional correction coefficient. The larger the proportional correction coefficient, the more advantageous it is to more quickly correct the display screen of the head-mounted display device based on a predetermined posture, which is advantageous for ensuring the follow-up effect of the display screen of the head-mounted display device.

[0092] In an embodiment of the present disclosure, by reasonably determining a proportional correction coefficient with reference to at least one of the posture error corresponding to the current time, the environmental conditions where the mobile platform on which the head-mounted display device is installed is placed, and the direction change information of the mobile platform, the determined proportional correction coefficient can be adapted to the actual situation.

[0093] Of course, similar to the adjustment scheme of the cut-off frequency, it can be understood that the user can also adjust the above-mentioned proportional correction coefficient according to usage needs.

[0094] In addition to the embodiment shown in FIG. 8, as shown in FIG. 10, this method further includes steps 180 and 190.

[0095] In step 180, based on a posture error series including posture errors corresponding to each of a plurality of times, a change rule of the posture error is determined.

[0096] After executing step 110 to determine the reference posture corresponding to the current time, the posture error corresponding to the current time can be determined by taking the difference between the reference posture corresponding to the current time and the predetermined posture. In a similar manner, the posture error corresponding to each time can be determined, and thus, a plurality of posture errors corresponding one-to-one to a plurality of times can be obtained. The plurality of posture errors can be arranged in ascending order of the corresponding times to form a posture error series.

[0097] It can be understood that when the predetermined posture and the reference posture are postures in the same coordinate system and the predetermined posture is a posture of zero rotation, the change rule of the posture error may be determined based on the reference posture series.

[0098] In an example that can be selected, the posture error series can include 20 arranged posture errors which are R1, R2, R3,..., R20 respectively. The time corresponding to R1 may be t1, the time corresponding to R2 may be t2, the time corresponding to R3 may be t3,..., and the time corresponding to R20 may be t20. Optionally, t20 may be the current time.

[0099] By analyzing the posture error series, the change rule of the posture error can be determined. The change rule of the posture error includes, but is not limited to, information such as whether the overall change trend of the posture error is an increasing trend or a decreasing trend, which times the maximum value and the minimum value of the posture error respectively correspond to, how many maximum values and how many minimum values there are in the posture error, which times each maximum value and each minimum value respectively correspond to, and the magnitude relationship between the posture error corresponding to each time (considered as the reference time) after a certain time and the posture error corresponding to the reference time.

[0100] In step 190, the proportional compensation coefficient is adjusted based on the change rule of the posture error.

[0101] Note that since there are various implementation forms of step 190, examples will be given below for explanation.

[0102] In an alternative embodiment, as shown in FIG. 11-1, step 190 includes the following: In step 1901, when it is determined that all the attitude errors corresponding to each time in the target period are greater than or equal to the attitude error corresponding to the target time based on the change law of the attitude error, the proportional compensation coefficient is increased.

[0103] The target time, the start time of the target period, and the end time of the target period are each one of a plurality of times. The start time of the target period is later than the target time, and the time length of the target period is equal to or greater than a predetermined time length.

[0104] Optionally, the predetermined time length may be 3 seconds, 4 seconds, 5 seconds, 8 seconds, or other time lengths, which will not be listed in detail here.

[0105] In an alternative example, the attitude error series can include 20 attitude errors arranged in sequence, which are R1, R2, R3,..., R20 respectively. The time corresponding to R1 may be t1, the time corresponding to R2 may be t2, the time corresponding to R3 may be t3,..., and the time corresponding to R20 may be t20. Optionally, t20 may be the current time.

[0106] If t13 among t1 to t20 is taken as the target time, the variation rule of the attitude error can include the magnitude relationship between each of R14 to R20 and R13. If the interval between any two adjacent times is 0.5 seconds, the predetermined time length is 3 seconds, the start time of the target period is t14, and the end time of the target period is t20, when each of R14 to R20 is not less than R13, it is considered that the attitude error continues to increase from the time of t13. That is, when correcting the attitude error based on the current proportional correction coefficient, the correction effect is not ideal. At this time, the proportional correction coefficient can be increased. For example, the proportional correction coefficient can be adjusted from 0.5 to 0.6 or 0.7. By increasing the proportional correction coefficient, the display screen of the head-mounted display device can be corrected more quickly based on the predetermined attitude, which is advantageous for ensuring the display effect of the head-mounted display device.

[0107] In another alternative embodiment, as shown in FIG. 11-2, step 190 includes the following: In step 1903, based on the variation law of the attitude error, when it is determined that the attitude error corresponding to each time in the target period is less than the attitude error corresponding to the target time, the proportional correction coefficient is decreased.

[0108] The target time, the start time of the target period, and the end time of the target period are each one of a plurality of times. The start time of the target period is later than the target time, and the time length of the target period is not less than the predetermined time length.

[0109] Optionally, the predetermined time length may be 3 seconds, 4 seconds, 5 seconds, 8 seconds, or other time lengths, which are not listed in detail here.

[0110] Continuing with the examples of the foregoing embodiments, when each of R14 to R20 is less than R13, it is considered that the attitude error continues to decrease from the time t13. That is, when correcting the attitude error based on the current proportional correction coefficient, the correction effect can meet the requirements. In this case, the proportional correction coefficient can be reduced. For example, the proportional correction coefficient can be adjusted from 0.5 to 0.3 or 0.4. By reducing the proportional correction coefficient, the speed at which the display screen of the head-mounted display device is corrected based on the predetermined attitude can be made slower, which is advantageous for ensuring the stability of the display screen of the head-mounted display device.

[0111] In a further selectable embodiment, as shown in FIG. 11-3, step 190 includes the following: In step 1905, when it is determined that the change tendency of the attitude error is an increasing tendency based on the change law of the attitude error, the proportional correction coefficient is increased.

[0112] Suppose it is determined that the change tendency of the attitude error is an increasing tendency based on the change law of the attitude error. For example, continuing with the example in the first embodiment, if R1, R2, R3,..., R20 increase sequentially, or if R1, R2, R3,..., R20 do not increase sequentially but the overall tendency gradually increases (for example, the first 10 attitude errors among the 20 attitude errors increase sequentially, the 11th attitude error is the same as the 12th attitude error, and the subsequent 8 attitude errors increase sequentially), then the proportional correction coefficient can be increased. For example, the proportional correction coefficient can be adjusted from 0.5 to 0.6 or 0.7. By increasing the proportional correction coefficient, the display screen of the head-mounted display device can be corrected more quickly based on the predetermined attitude, which is advantageous for ensuring the display effect of the head-mounted display device.

[0113] In a further selectable embodiment, as shown in FIG. 11-4, step 190 includes the following: In step 1907, when it is determined that the change tendency of the attitude error is a decreasing tendency based on the change law of the attitude error, the proportional compensation coefficient is decreased.

[0114] If it is determined that the change tendency of the attitude error is a decreasing tendency based on the change law of the attitude error, for example, continuing with the example in the first implementation form, if R1, R2, R3, …, R20 decrease sequentially, or if R1, R2, R3, …, R20 do not decrease sequentially but the overall tendency gradually decreases (for example, among the 20 attitude errors, the first 10 attitude errors decrease sequentially, the 11th attitude error is the same as the 12th attitude error, and the subsequent 8 attitude errors decrease sequentially), then the proportional compensation coefficient can be decreased. For example, the proportional compensation coefficient can be adjusted from 0.5 to 0.3 or 0.4. By decreasing the proportional compensation coefficient, the speed at which the display screen of the head-mounted display device is corrected based on a predetermined attitude can be made slower, which is advantageous for ensuring the stability of the display screen of the head-mounted display device.

[0115] As can be seen from this, in the embodiments of the present disclosure, by referring to the change law of the reference attitude error and adjusting the proportional compensation coefficient, the display screen of the head-mounted display device can be corrected at a more appropriate speed according to the actual situation, thereby ensuring the display effect and the stability of the display screen of the head-mounted display device.

[0116] In addition to the embodiment shown in FIG. 1, as shown in FIG. 12, this method further includes step 102, step 104, and step 106.

[0117] In step 102, the type of the platform on which the head-mounted display device is installed is recognized.

[0118] It should be noted that there are various implementation forms for recognizing the type of the platform on which the head-mounted display device is installed, and examples will be given below for explanation.

[0119] In an alternative embodiment, the head-mounted display device can establish a communication connection with the installed platform and obtain relevant information of the platform during the establishment of the communication connection. The relevant information of the platform includes, but is not limited to, identification information, type information, etc. By extracting the type information from the obtained relevant information and recognizing the category represented by the type information, it is possible to determine whether the type of the platform is a mobile platform.

[0120] In another alternative embodiment, the head-mounted display device may further include a camera. The head-mounted display device can collect an image of the environment where it is installed via the camera and determine whether the image matches a mobile platform such as a vehicle by detecting the target in the image. If they match, it can be determined that the type of the platform where the head-mounted display device is installed is a mobile platform; otherwise, it can be determined that the type of the platform where the head-mounted display device is installed is not a mobile platform.

[0121] In a further alternative implementation, acceleration data collected by the accelerometer in the IMU of the head-mounted display device or angular velocity data collected by the gyroscope can be obtained and compared with a preset threshold. Based on the acceleration data, if it is determined that the acceleration of the head-mounted display device is greater than the set acceleration threshold, or based on the angular velocity data, if it is determined that the angular velocity of the head-mounted display device is greater than the set angular velocity threshold, it can be determined that the type of the platform where the head-mounted display device is installed is a mobile platform. Otherwise, it can be determined that the type of the platform where the head-mounted display device is installed is not a mobile platform.

[0122] In step 104, when the type of the platform where the head-mounted display device is installed is a mobile platform, control is performed to switch the head-mounted display device to the target mode.

[0123] In an embodiment of the present disclosure, the head-mounted display device can include at least three modes, and the at least three modes can include the above 0dof mode, 3dof mode, and target mode in step 104.

[0124] In step 106, it is determined whether the head-mounted display device is in the target mode. If so, step 110 is triggered.

[0125] In an embodiment of the present disclosure, the type of the platform where the head-mounted display device is installed can be recognized. When the type of the platform is not a mobile platform, the head-mounted display device can be set to the normal 0dof mode or 3dof mode. When the type of the platform is a mobile platform, the head-mounted display device can be automatically switched to the target mode. Thereby, in the target mode, step 110 is triggered to realize the execution from step 110 to step 140 above. As described above, the target mode is a mode between the 0dof mode and the 3dof mode. In the target mode, when the mobile platform pitches or sways, by adjusting the display screen of the head-mounted display device, pitching and swaying can be corrected and the screen can be stably held. Moreover, when the mobile platform or the user's head changes direction, it is possible to keep the display screen of the head-mounted display device following and positioning the user's head, thereby improving the user experience.

[0126] Note that the above description explains that when the head-mounted display device is on a mobile platform, the head-mounted display device automatically switches to the target mode. Of course, the user can also manually select to set the head-mounted display device to any one of the 0dof mode, 3dof mode, or target mode according to actual needs.

[0127] Note that the mobile platform according to the embodiments of the present disclosure may also be referred to as a mobile carrier. In the embodiments of the present disclosure, the angular velocity data is not collected by the gyroscope in the IMU, but may be collected by other sensors capable of acquiring angular velocity data.

[0128] As described above, in the embodiments of the present disclosure, on the premise of not using the acceleration data collected by the IMU, by using the angular velocity data collected by the IMU, the display effect of the head-mounted display device can always be corrected according to the display effect in the predetermined posture of the head-mounted display device, and for the head-mounted display device, whether it is a short-time small-angle movement or a long-time large-angle movement, the display effect can be ensured.

[0129] Any method for displaying a screen on the head-mounted display device provided by the embodiments of the present disclosure may be executed by any suitable device having data processing capabilities, including but not limited to terminal devices, servers, etc. Alternatively, any method for displaying a screen on the head-mounted display device provided by the embodiments of the present disclosure may be executed by a processor. For example, the processor calls the corresponding instructions stored in the memory to execute any method for displaying a screen on the head-mounted display device mentioned in the embodiments of the present disclosure. The detailed description is omitted below.

[0130] Embodiments of the present disclosure further provide a method for displaying a screen on a head-mounted display device. The method includes: obtaining angular velocity data of the head-mounted display device; determining a reference posture for processing a screen to be displayed based on the angular velocity data and a previous first corrected posture for processing the display screen of the head-mounted display device; processing the reference posture to obtain a corrected posture difference value; compensating the reference posture with the corrected posture difference value to obtain a second corrected posture for processing the screen to be displayed; and displaying the screen to be displayed on the head-mounted display device by rendering the screen to be displayed on the head-mounted display device based on the second corrected posture.

[0131] Embodiments of the present disclosure further provide a method for displaying a screen on a head-mounted display device. The method includes: obtaining angular velocity data of the head-mounted display device; determining a reference posture for processing a screen to be displayed based on the angular velocity data and a previous first corrected posture for processing the display screen of the head-mounted display device; processing the posture error between the reference posture and a predetermined posture for processing the display screen to obtain a corrected posture difference value; compensating the reference posture with the corrected posture difference value to obtain a second corrected posture for processing the screen to be displayed; and displaying the screen to be displayed on the head-mounted display device by rendering the screen to be displayed on the head-mounted display device based on the second corrected posture.

[0132] Embodiments of the present disclosure further provide a method for displaying a screen on a head-mounted display device. The method includes determining a reference posture for processing a screen to be displayed at the current time based on the angular velocity data at the current time of the head-mounted display device and a first corrected posture for processing the display screen of the head-mounted display device at the previous time; processing the reference posture to obtain a corrected posture difference value; compensating the reference posture with the corrected posture difference value to obtain a second corrected posture for processing the screen to be displayed at the current time; and displaying the screen to be displayed on the head-mounted display device by rendering the screen to be displayed on the head-mounted display device based on the second corrected posture.

[0133] In some selectable embodiments, in response to a user's selection, the display screen mode of the head-mounted display device is switched to any one of a 0-degree-of-freedom mode, a target mode, and a 3-degree-of-freedom mode. When the head-mounted display device is in the target mode, a step of determining a reference posture for processing a screen to be displayed at the current time based on the angular velocity data at the current time of the head-mounted display device and a first corrected posture for processing the display screen of the head-mounted display device at the previous time is executed.

[0134] In some selectable embodiments, the method for displaying a screen on a head-mounted display device further includes receiving an input operation by a user and adjusting the above-mentioned proportional correction coefficient in response to the input operation by the user.

[0135] In some selectable embodiments, the method for displaying a screen on a head-mounted display device further includes a step of being able to determine a proportional correction coefficient based on compensation reference information after correcting the posture error with the proportional correction coefficient, thereby facilitating dynamic updating of the proportional correction coefficient with the determination result.

[0136] Exemplary device FIG. 13 is a schematic configuration diagram of a device for displaying a screen on a head-mounted display device provided according to an exemplary embodiment of the present disclosure. The device shown in FIG. 13 includes a first determination module 1310, a first processing module 1320, a compensation module 1330, and a second processing module 1340.

[0137] The first determination module 1310 is used to determine a reference posture for processing the screen to be displayed at the current time based on the angular velocity data of the head-mounted display device at the current time and the first corrected posture for processing the display screen of the head-mounted display device at the previous time. The first processing module 1320 is used to process the posture error between the reference posture and a predetermined posture for processing the display screen so as to obtain a corrected posture difference value. The compensation module 1330 is used to compensate the reference posture with the corrected posture difference value to obtain a second corrected posture for processing the screen to be displayed at the current time. The second processing module 1340 is used to display the screen to be displayed on the head-mounted display device by rendering the screen to be displayed on the head-mounted display device based on the second corrected posture.

[0138] In an optional example, the first processing module 1320 is used to perform a low-pass filter process on the posture error corresponding to the current time so as to obtain a corrected posture difference value.

[0139] In an optional example, the first processing module 1320 is used to perform a low-pass filter process on the posture error corresponding to the current time by a low-pass filter. As shown in FIG. 14, the device includes a reception module 1350 for receiving an input operation by a user. It further includes a first adjustment module 1360 for adjusting the cut-off frequency of the low-pass filter in response to an input operation by the user.

[0140] In an optional example, the first processing module 1320 is used to correct the attitude error corresponding to the current time with a proportional correction coefficient greater than 0 and less than 1 to obtain a corrected attitude difference value.

[0141] In an optional example, as shown in FIG. 15, the apparatus further includes a second determination module 1315 for determining a proportional correction coefficient based on compensation reference information including at least one of the attitude error corresponding to the current time, the integrated value of the attitude error, the environmental situation where the mobile platform on which the head-mounted display device is installed is placed, and the direction conversion information of the mobile platform.

[0142] In an optional example, it satisfies at least one of the following four: The proportional correction coefficient shows a positive correlation with the attitude error corresponding to the current time, The proportional correction coefficient shows a positive correlation with the integrated value of the attitude error, The environmental situation includes road surface flatness, and the proportional correction coefficient shows a positive correlation with the road surface flatness, The direction conversion information includes the degree of direction conversion, and the proportional correction coefficient shows a positive correlation with the degree of direction conversion.

[0143] In an optional example, as shown in FIG. 16, the apparatus further includes a third determination module 1380 for determining the change rule of the attitude error based on an attitude error series including the attitude errors corresponding to each of a plurality of times, and a second adjustment module 1390 for adjusting the proportional correction coefficient based on the change rule of the attitude error.

[0144] In an example that can be selected, the second adjustment module 1390 includes at least one of a first adjustment unit 13901, a second adjustment unit 13903, a third adjustment unit 13905, and a fourth adjustment unit 13907 shown in FIG. 16, When it is determined that, based on the change law of the attitude error, the attitude errors corresponding to each time in the target period are all greater than or equal to the attitude error corresponding to the target time, the first adjustment unit 13901 is used to increase the proportionality coefficient, When it is determined that, based on the change law of the attitude error, the attitude errors corresponding to each time in the target period are all less than the attitude error corresponding to the target time, the second adjustment unit 13903 is used to decrease the proportionality coefficient, When it is determined that, based on the change law of the attitude error, the change tendency of the attitude error is an increasing tendency, the third adjustment unit 13905 is used to increase the proportionality coefficient, When it is determined that, based on the change law of the attitude error, the change tendency of the attitude error is a decreasing tendency, the fourth adjustment unit 13907 is used to decrease the proportionality coefficient.

[0145] Here, the target time, the start time of the target period, and the end time of the target period are each one of a plurality of times. The start time of the target period is later than the target time, and the time length of the target period is equal to or greater than a predetermined time length.

[0146] In an example that can be selected, As shown in FIG. 17, the apparatus further includes a recognition module 1302 for recognizing the type of the platform on which the head-mounted display device is installed, a control module 1304 for controlling the head-mounted display device to switch to a target mode when the type is a mobile platform, a trigger module 1306 for triggering a first determination module 1310 when the head-mounted display device is in the target mode.

[0147] In an alternative example, the head-mounted display device is coupled to the installed mobile platform and is movably disposed with respect to the mobile platform.

[0148] In an alternative example, it satisfies one of the following two: The predetermined posture is a posture for displaying the display screen of the head-mounted display device at a predetermined position in the coordinate system of the head-mounted display device. The predetermined posture is a posture for aligning the display screen of the head-mounted display device in the direction of gravity.

[0149] In an alternative example, as shown in FIG. 18, the device further includes an arrangement module 1395 for arranging an optical device in the head-mounted display device based on a second corrected posture.

[0150] Exemplary Electronic Device Hereinafter, with reference to FIG. 19, an electronic device according to an embodiment of the present disclosure will be described. The electronic device may be either one or both of a first device and a second device, or may be a separate stand-alone device capable of communicating with the first device and the second device to receive a collected input signal.

[0151] FIG. 19 illustrates a block diagram of an electronic device 1900 according to an embodiment of the present disclosure.

[0152] As shown in FIG. 19, the electronic device 1900 includes one or more processors 1910 and a memory 1920.

[0153] The processor 1910 may be a central processing unit (CPU) or another form of processing device having data processing capabilities and / or instruction execution capabilities, and can control other components within the electronic device 1900 to perform desired functions.

[0154] ​Memory 1920 can include one or more computer program products including various forms of computer-readable storage media such as volatile memory and / or non-volatile memory. The volatile memory can include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory can include, for example, read-only memory (ROM), hard disk, flash memory, etc. The computer-readable storage media can store one or more computer program instructions executable by processor 1910 for, for example, a method of displaying a screen on the head-mounted display device of each embodiment of the present disclosure described above and / or for implementing other desired functions. The computer-readable storage media can further store various contents such as input signals, signal components, noise components, etc.

[0155] In one example, the electronic device 1900 may further include an input device 1930 and an output device 1940, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0156] For example, when the electronic device 1900 is the first device or the second device, the input device 1930 may be a microphone or a microphone array. When the electronic device 1900 is a stand-alone device, the input device 1930 may be a communication network connector for receiving input signals collected from the first device and the second device.

[0157] Furthermore, the input device 1930 may further include, for example, a keyboard, a mouse, etc.

[0158] The output device 1940 can output various information to the outside. The output device 1940 can include, for example, a display, a speaker, a printer, a communication network, and remote output devices connected thereto.

[0159] Of course, for simplicity, FIG. 19 shows only some of the components related to the present disclosure in the electronic device 1900, and omits components such as buses and input / output interfaces. Additionally, the electronic device 1900 may further include other appropriate components according to specific application scenarios.

[0160] Exemplary computer program product and computer-readable storage medium

[0161] In addition to the above methods and apparatuses, embodiments of the present disclosure may also be a computer program product including computer program instructions that cause a processor to execute steps in a method for displaying a screen on a head-mounted display device according to various embodiments of the present disclosure described in the "Exemplary Method" section herein when executed by the processor.

[0162] The computer program product may describe program code for executing the operations of the embodiments of the present disclosure in any combination of one or more programming languages including object-oriented programming languages such as Java and C++, and general-purpose procedural programming languages such as the "C" language. The program code may be executed entirely on a user computing device, partially on a user device, executed as an independent software package, partially on a user computing device and partially on a remote computing device, or alternatively, executed entirely on a remote computing device or server.

[0163] Furthermore, embodiments of the present disclosure may also be a computer-readable storage medium storing computer program instructions that cause a processor to execute steps in a method for displaying a screen on a head-mounted display device according to various embodiments of the present disclosure described in the "Exemplary Method" section herein when executed by the processor.

[0164] The computer-readable storage medium can employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or any combination thereof. More specific examples (but not an exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any combination of the above.

[0165] As described above, the basic principles of the present disclosure have been described with reference to specific embodiments. It should be noted that the advantages, benefits, effects, etc. mentioned in the present disclosure are merely examples and not for limitation, and it is considered that these advantages, benefits, effects, etc. are not essential to each embodiment of the present disclosure. The specific details disclosed above are merely for the purpose of illustration and facilitating understanding, and not for limitation, and the above details do not limit that the present disclosure must be implemented using the above specific details.

[0166] Each embodiment in this specification is recursively described. Each embodiment is described with an emphasis on the differences from other embodiments. For the same or similar parts between each embodiment, reference may be made to each other. For the embodiment of the system, since it basically corresponds to the embodiment of the method, it is described relatively simply, and for the relevant parts, reference may be made to the description of the corresponding part of the embodiment of the method.

[0167] The block diagrams of the devices, apparatuses, equipment, and systems according to the present disclosure are merely exemplary examples, and it is not intended to require or suggest that they must be connected, configured, and arranged as in the block diagrams. As will be recognized by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, configured, and arranged in any manner. For example, words such as "include," "comprise," and "have" are open-ended words, referring to "including but not limited to," and can be used interchangeably with each other. As used herein, the terms "or" and "and" mean the term "and / or," and these terms can be used interchangeably with each other unless explicitly indicated otherwise in the context. As used herein, the term "for example" means the phrase "for example, but not limited to," and can be used interchangeably with each other.

[0168] The methods and apparatuses of the present disclosure may be implemented in many ways. For example, the methods and apparatuses of the present disclosure can be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of the steps used in the method is for illustrative purposes only, and the steps of the method of the present disclosure are not limited to the specifically described order above unless otherwise specifically described in other forms. Further, in some embodiments, the present disclosure can also be implemented as a program recorded on a recording medium including machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0169] It should be noted that in the apparatuses, equipment, and methods of the present disclosure, each component or each step can be disassembled and / or recombined. These disassemblies and / or recombinations should be regarded as equivalents of the present disclosure.

[0170] The foregoing description of the disclosed embodiments is provided so that those skilled in the art can make and use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein are applicable to other embodiments without departing from the scope of the present disclosure. Accordingly, the present disclosure is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0171] The above description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present disclosure to the form disclosed herein. Although numerous exemplary embodiments and examples have been considered, some variations, corrections, changes, additions, and sub-combinations will be recognized by those skilled in the art.

Description of Reference Numerals

[0172] 1302 Recognition Module 1304 Control Module 1306 Trigger Module 1310 First Decision Module 1315 Second Decision Module 1320 First Processing Module 1330 Compensation Module 1340 Second Processing Module 1350 Receiving Module 1360 First Adjustment Module 1380 Third Decision Module 1390 Second Adjustment Module 13901 First Adjustment Unit 13903 Second Adjustment Unit 13905 Third Adjustment Unit 13907 Fourth Adjustment Unit 1395 Arrangement Module 1900 Electronic Device 1910 Processor 1920 Memory 1930 Input Device 1940 Output device

Claims

1. A method for displaying a screen on a head-mounted display device, comprising: determining a reference posture for processing a screen to be displayed at the current time based on the angular velocity data at the current time of the head-mounted display device and a first corrected posture for processing the display screen of the head-mounted display device at the previous time; processing a posture error between the reference posture and a predetermined posture for processing the display screen to obtain a corrected posture difference value; compensating the reference posture with the corrected posture difference value to obtain a second corrected posture for processing the screen to be displayed at the current time; displaying the screen to be displayed on the head-mounted display device by rendering the screen to be displayed on the head-mounted display device based on the second corrected posture; A method comprising the steps of.

2. The step of processing the posture error between the reference posture and the predetermined posture for processing the display screen to obtain the corrected posture difference value includes: The method according to claim 1, further comprising the step of correcting a posture error corresponding to the current time with a correction proportionality coefficient greater than 0 and less than 1 to obtain a corrected posture difference value.

3. The method according to claim 2, further comprising the step of determining the correction proportionality coefficient based on at least one of a posture error corresponding to the current time, an integrated value of the posture error, an environmental condition in which the mobile platform on which the head-mounted display device is installed is placed, and direction conversion information of the mobile platform.

4. Satisfying at least one of the following four: The correction proportionality coefficient shows a positive correlation with the posture error corresponding to the current time; The correction proportionality coefficient shows a positive correlation with the integrated value of the posture error; The environmental condition includes road surface flatness, and the correction proportionality coefficient shows a positive correlation with the road surface flatness; The direction conversion information includes the degree of direction conversion, and the correction proportionality coefficient shows a positive correlation with the degree of direction conversion. The method according to claim 3.

5. determining a change rule of the posture error based on a posture error series including posture errors corresponding to each of a plurality of times; The method according to claim 2, further comprising the step of adjusting the correction proportionality coefficient based on the change rule of the posture error.

6. Based on the change law of the posture error, the step of adjusting the proportional correction coefficient includes at least one of the following four: Based on the change law of the posture error, when it is determined that the posture errors corresponding to each moment in the target period are all greater than or equal to the posture error corresponding to the target moment, the step of increasing the proportional correction coefficient; Based on the change law of the posture error, when it is determined that the posture errors corresponding to each moment in the target period are all less than the posture error corresponding to the target moment, the step of decreasing the proportional correction coefficient; Based on the change law of the posture error, when it is determined that the change trend of the posture error is an increasing trend, the step of increasing the proportional correction coefficient; Based on the change law of the posture error, when it is determined that the change trend of the posture error is a decreasing trend, the step of decreasing the proportional correction coefficient, and The target moment, the start moment of the target period, and the end moment of the target period are each one of the plurality of moments, the start moment of the target period is later than the target moment, and the time length of the target period is equal to or greater than a predetermined time length. The method according to claim 5.

7. The step of receiving an input operation by a user; The method according to claim 2, further including the step of adjusting the proportional correction coefficient in response to an input operation by a user.

8. The step of processing the posture error between the reference posture and a predetermined posture for processing the display screen to obtain the corrected posture difference value includes: The method according to claim 1, including the step of performing a low-pass filter process on the posture error corresponding to the current moment so as to obtain the corrected posture difference value.

9. The step of performing a low-pass filter process on the posture error corresponding to the current moment includes: The step of performing a low-pass filter process on the posture error corresponding to the current moment by a low-pass filter, and The method includes: The step of receiving an input operation by a user; The method according to claim 8, further including the step of adjusting the cut-off frequency of the low-pass filter in response to the input operation by the user.

10. In response to a user's selection, the method further includes the step of switching the display screen mode of the head-mounted display device to any one of a zero-degree-of-freedom mode, a target mode, and a three-degree-of-freedom mode. When the head-mounted display device is in the target mode, a step of determining a reference posture for processing a screen to be displayed at the current time is executed based on the angular velocity data at the current time of the head-mounted display device and the first corrected posture for processing the display screen of the head-mounted display device at the previous time. The method according to any one of claims 1 to 9.

11. A step of recognizing the type of the platform on which the head-mounted display device is installed; When the type is a mobile platform, a step of controlling the head-mounted display device to switch to the target mode, further comprising: When the head-mounted display device is in the target mode, a step of determining a reference posture for processing a screen to be displayed at the current time is executed based on the angular velocity data at the current time of the head-mounted display device and the first corrected posture for processing the display screen of the head-mounted display device at the previous time. The method according to any one of claims 1 to 9.

12. The head-mounted display device is coupled to a mobile platform on which it is installed and is disposed movably with respect to the mobile platform. The method according to any one of claims 1 to 9.

13. Satisfies at least one of the following two: The predetermined posture is a posture for displaying the display screen of the head-mounted display device at a predetermined position in the coordinate system of the head-mounted display device; The predetermined posture is a posture for aligning the display screen of the head-mounted display device in the direction of gravity. The method according to any one of claims 1 to 12.

14. An electronic device, comprising: A memory for storing a computer program product; A processor for executing the computer program product stored in the memory, wherein when the computer program product is executed, a method for displaying a screen on the head-mounted display device according to any one of claims 1 to 13 is realized. An electronic device characterized by including a processor.

15. In a computer-readable storage medium storing computer program instructions. A computer-readable storage medium, characterized in that when the computer program instructions are executed by a processor, a method for displaying a screen on the head-mounted display device according to any one of claims 1 to 13 is realized.

Images