Method and device for displaying picture on head-mounted display equipment and electronic equipment

Abstract

The invention discloses a method and device for displaying a picture on head-mounted display equipment and electronic equipment. A specific implementation scheme is: determining a reference attitude for processing a to-be-displayed screen at a current moment based on angular velocity data of the head-mounted display device at the current moment and a first correction attitude for processing a display screen of the head-mounted display device at a previous moment (110); processing an attitude error between the reference attitude and a preset attitude for processing the display screen to obtain a corrected attitude difference (120); compensating the reference attitude by using the correction attitude difference value to obtain a second correction attitude for processing the to-be-displayed image at the current moment (130); and based on the second correction attitude, rendering a to-be-displayed image of the head-mounted display device to display the to-be-displayed image of the head-mounted display device (140).

Description

Method, device and electronic device for displaying images on head-mounted display device

[0001] This disclosure claims priority to a Chinese patent application filed with the Patent Office of China on May 11, 2022, with application number CN202210510750.8 and invention name “Method, device and electronic device for displaying images on a head-mounted display device,” the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The present disclosure relates to the technical field of head-mounted display devices, and in particular to a method, apparatus, and electronic device for displaying images on a head-mounted display device. Background Art

[0003] Head-mounted display devices are increasingly being used. However, they can be subject to vibration and shaking, which can affect their display quality. Ensuring the display quality of head-mounted display devices and enhancing the user experience is a pressing issue for those skilled in the art.

[0004] Summary of the Invention

[0005] Embodiments of the present disclosure provide a method, apparatus, and electronic device for displaying images on a head-mounted display device.

[0006] According to one aspect of the present disclosure, a method for displaying an image on a head-mounted display device is provided, comprising: determining a reference posture for processing an image to be displayed at the current moment based on angular velocity data of the head-mounted display device at the current moment and a first correction posture used to process an image displayed by the head-mounted display device at the previous moment; processing an attitude error between the reference posture and a preset posture for processing the image to be displayed to obtain a correction posture difference; compensating the reference posture using the correction posture difference to obtain a second correction posture for processing the image to be displayed at the current moment; and rendering the image to be displayed by the head-mounted display device based on the second correction posture to display the image to be displayed by the head-mounted display device.

[0007] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores a computer program for executing the above-mentioned method for displaying images on a head-mounted display device.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the above-mentioned method for displaying images on a head-mounted display device.

[0009] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other purposes, features, and advantages of the present disclosure will become more apparent through a more detailed description of the embodiments of the present disclosure in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and are not intended to limit the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] FIG1 is a flowchart of a method for displaying images on a head-mounted display device provided by an exemplary embodiment of the present disclosure.

[0012] FIG2 is a schematic diagram of a method for processing a display screen of a head-mounted display device in an embodiment of the present disclosure.

[0013] FIG3 is a schematic diagram of another method for processing a display image of a head-mounted display device in an embodiment of the present disclosure.

[0014] FIG4 is a schematic diagram of a scenario to which an embodiment of the present disclosure is applicable.

[0015] FIG5 is a flowchart of a method for displaying images on a head-mounted display device provided by another exemplary embodiment of the present disclosure.

[0016] FIG6 is a schematic diagram of a flow chart of determining a second correction posture in an embodiment of the present disclosure.

[0017] FIG7 is a flowchart of a method for displaying images on a head-mounted display device provided by yet another exemplary embodiment of the present disclosure.

[0018] FIG8 is a flowchart of a method for displaying images on a head-mounted display device provided by yet another exemplary embodiment of the present disclosure.

[0019] FIG9 is a flowchart of a method for displaying images on a head-mounted display device provided by yet another exemplary embodiment of the present disclosure.

[0020] FIG10 is a flowchart of a method for displaying images on a head-mounted display device provided by yet another exemplary embodiment of the present disclosure.

[0021] FIG11-1 is a flowchart of a method for displaying images on a head-mounted display device provided by another exemplary embodiment of the present disclosure.

[0022] FIG11-2 is a flowchart of a method for displaying images on a head-mounted display device provided by another exemplary embodiment of the present disclosure.

[0023] FIG11-3 is a flowchart of a method for displaying images on a head-mounted display device provided by another exemplary embodiment of the present disclosure.

[0024] FIG11-4 is a flowchart of a method for displaying images on a head-mounted display device provided by another exemplary embodiment of the present disclosure.

[0025] FIG12 is a flowchart of a method for displaying images on a head-mounted display device provided by yet another exemplary embodiment of the present disclosure.

[0026] FIG13 is a schematic structural diagram of an apparatus for displaying images on a head-mounted display device provided by an exemplary embodiment of the present disclosure.

[0027] FIG14 is a schematic structural diagram of an apparatus for displaying images on a head-mounted display device provided by another exemplary embodiment of the present disclosure.

[0028] FIG15 is a structural diagram of an apparatus for displaying images on a head-mounted display device provided by yet another exemplary embodiment of the present disclosure.

[0029] FIG16 is a structural diagram of an apparatus for displaying images on a head-mounted display device provided by yet another exemplary embodiment of the present disclosure.

[0030] FIG17 is a structural diagram of an apparatus for displaying images on a head-mounted display device provided by yet another exemplary embodiment of the present disclosure.

[0031] FIG18 is a structural diagram of an apparatus for displaying images on a head-mounted display device provided by yet another exemplary embodiment of the present disclosure.

[0032] FIG19 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0034] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0035] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate a necessary logical order between them.

[0036] It should also be understood that in the embodiments of the present disclosure, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0037] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0038] In addition, the term "and / or" in this disclosure is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the related objects are in an "or" relationship.

[0039] It should also be understood that the description of the various embodiments in this disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0040] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0041] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0042] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0043] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate in conjunction with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, among others.

[0044] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media, including storage devices.

[0045] Exemplary Methods

[0046] Figure 1 is a flowchart of a method for displaying images on a head-mounted display device provided by an exemplary embodiment of the present disclosure. The method shown in Figure 1 may include steps 110, 120, 130, and 140, each of which is described below.

[0047] Step 110 : Determine a reference posture for processing the image to be displayed at the current moment based on the angular velocity data of the head-mounted display device at the current moment and the first correction posture used to process the display image of the head-mounted display device at the previous moment.

[0048] It should be noted that a head-mounted display device may also be referred to as a head-mounted display (HMD) or head-mounted display. A head-mounted display device can be used to achieve extended reality (XR) effects, such as augmented reality (AR), virtual reality (VR), and mixed reality (MR). Alternatively, the head-mounted display device can be used to achieve AR effects, in which case the head-mounted display device can be AR glasses.

[0049] Generally speaking, a head-mounted display device (HMD) includes an inertial measurement unit (IMU) and an optical engine. The IMU may include an accelerometer, gyroscope, and magnetometer. The accelerometer collects acceleration data, while the gyroscope collects angular velocity data. The optical engine is the imaging system of the HMD and may include a display screen and optical components.

[0050] Normally, the user may not be able to directly see the display screen of the head-mounted display device. Instead, the display screen is provided to the user through the following scheme. The display screen is used to emit light for the display screen, and the optical element can process the above light so that the light of the display screen is projected into the user's eyes, and ultimately the user can see the display screen. However, since when the user wears the head-mounted display device, there is almost no relative displacement and rotation between the head-mounted display device and the user, the head-mounted display device will move with the user's movement. Therefore, it may be necessary to process the display screen based on the position and posture of the head-mounted display device (which can actually be regarded as the user), so that the display screen can always be positioned at a certain position, or within a certain area, or can move with the user's movement, etc., to match the corresponding application scenario or user needs.

[0051] Prior to step 110, the current angular velocity data of the IMU can be obtained, such as the angular velocity data collected by the gyroscope in the IMU, and the first calibration posture used to process the display image of the head-mounted display device at the previous moment can be obtained. The first calibration posture can be obtained in a manner similar to the second calibration posture described below. For the sake of clarity, the method for obtaining the first calibration posture is not described in detail here.

[0052] In step 110, the angular velocity data at the current moment and the first correction posture can be used for integration processing with reference to the integration method from the gyroscope angular velocity to the angular position to obtain a reference posture for processing the image to be displayed (the image to be displayed can be the image to be displayed by the head-mounted display device) at the current moment. Assume that the angular velocity data at the current moment is represented by Gyro k , the first correction posture is expressed as Pose k-1 , the reference posture is expressed as Then we have:

[0053] Step 120 : Processing the posture error between the reference posture and the preset posture for processing the display image to obtain a corrected posture difference.

[0054] It should be noted that the preset posture can be a posture pre-set for the head-mounted display device that can ensure the display effect of the head-mounted display device. In other words, the preset posture can be a posture used to process the display image in order to ensure the display effect.

[0055] In an optional example, the preset posture is a posture for causing the display screen of the head-mounted display device to be displayed at a preset position in the coordinate system of the head-mounted display device. It is understandable that it can also be a preset position in the IMU coordinate system.

[0056] It is understood that the preset position in the coordinate system of the head-mounted display device can be a fixed position in the coordinate system of the head-mounted display device, for example, so that the display screen is displayed in a position that can be viewed by the user, such as directly in front of the head-mounted display device, in the upper left, or lower right, etc. It should be noted that most users tend to display the display screen in the center of their eyes. In the embodiments of the present disclosure, by setting the preset posture to a posture that matches the usage habits of most users, the display effect of the head-mounted display device can be made to meet the needs of most users as much as possible, which is conducive to improving the user experience.

[0057] In an optional example, the preset posture is a posture for aligning a display screen of the head-mounted display device with the direction of gravity (in this posture, the heading angle (Yaw) of the head-mounted display device may be 0).

[0058] Aligning the display of a head-mounted display device with the direction of gravity can be understood as ensuring that, when a user observes the display through the head-mounted display device, the direction in which the plane of the display extends is the same as the direction of gravity in the real world. This ensures that the display is substantially perpendicular to the user's eye level, consistent with viewing habits. It should be noted that most users tend to align the display of a head-mounted display device with the direction of gravity. In the embodiments of the present disclosure, by setting the preset posture to one that matches the usage habits of most users, the display effect of the head-mounted display device can meet the needs of most users as much as possible, thereby improving the user experience.

[0059] Of course, users can also set a preset posture that can make the display screen of the head-mounted display device appear in a customized position according to their needs.

[0060] In step 120, the reference posture and the preset posture can be subtracted to obtain the posture error between the reference posture and the preset posture. The posture error can be in the form of 3×1 axis angle. It should be noted that the subtraction here is not a simple subtraction, but a generalized function. The "difference" in the subtraction refers to the difference. The subtraction can refer to the relative rotation between the two angles. Assume that the reference posture is expressed as The preset posture is represented as Target, and the posture error is represented as error, then:

[0061] After the attitude error is obtained, the attitude error can be processed to achieve attitude error correction, thereby obtaining a corrected attitude difference.

[0062] It is understood that the preset posture can be a posture pre-set for the head-mounted display device. If the preset posture and the reference posture are in the same coordinate system, and the preset posture is a posture with zero rotation, for program efficiency, the step of subtracting the reference posture from the preset posture can be omitted, and the reference posture can be directly used as the subsequent processing variable. In other words, in the subsequent steps, the reference posture is directly processed to correct the posture error, thereby obtaining a corrected posture difference.

[0063] Step 130 : Compensating the reference posture using the correction posture difference to obtain a second correction posture for processing the image to be displayed at the current moment.

[0064] In step 130, the correction posture difference can be used as a compensation value to update the reference posture to obtain a second correction posture for processing the display image at the current moment. The second correction posture is represented by Pose k , then:

[0065] It should be noted that the compensation value here may represent a relative rotation, and updating the compensation value to the reference posture may be performed by superimposing a relative rotation on the reference posture to obtain another posture (ie, the second correction posture).

[0066] It is understood that the correction posture difference is used to update the reference posture to obtain the second correction posture. Regardless of whether the correction posture difference is obtained by processing the reference posture or the posture error between the reference posture and the preset posture, the second correction posture is updated based on the correction posture difference.

[0067] Step 140: Render the image to be displayed on the head-mounted display device based on the second correction posture to display the image to be displayed on the head-mounted display device.

[0068] It is understood that both the first calibration posture and the second calibration posture are postures used to process the display image of the head-mounted display device, and can also be referred to as calibration postures. For example, when the head-mounted display device executes this method for the first time and does not have a calibration posture at a previous moment, the calibration posture at a previous moment can also be the actual posture of the head-mounted display device, a preset posture for processing the display image, or another pre-set posture.

[0069] It should be noted that there are various specific implementation forms for processing the display image of the head-mounted display device based on the second correction posture or the correction posture, which are introduced below with examples.

[0070] In an optional implementation, the rendering engine of the head-mounted display device can use the second correction posture or the correction posture to render the image to obtain a corresponding display image. The obtained display image can be updated to the optical engine of the head-mounted display device for display, thereby correcting the display image of the head-mounted display device. It is easy to see that this implementation uses software to process the display image of the head-mounted display device. For example, this solution can adjust the overall parameters of the image to be displayed based on the second correction posture or the correction posture. The overall parameters may include the three-dimensional size of the display image.

[0071] In another optional implementation, as shown in FIG2 , the second correction posture (eg, Pose in FIG2 ) can be used to correct the posture of the user. k ) or a corrected posture for rendering, adjusting the display position of the image on the display screen (e.g., the display in Figure 2) within the optical engine of the head-mounted display device to achieve correction of the head-mounted display image. This implementation allows for processing of the head-mounted display image, for example, adjusting the display position of the head-mounted display image up, down, left, and right. For example, this solution can adjust the two-dimensional size of the image to be displayed based on the second corrected posture or the corrected posture.

[0072] In another optional implementation, as shown in FIG3 , the image rendering can be performed based on the second correction posture or the correction posture, and the optical machine in the head-mounted display device can be configured. For example, the overall position of the optical machine in the head-mounted display module or the position of the display screen in the optical machine can be adjusted through the electronic control chassis to achieve the correction of the display screen of the head-mounted display device (this is equivalent to holding the display screen in front of the user and making corresponding shock absorption adjustments). It is easy to see that this implementation uses hardware to implement the processing of the display screen of the head-mounted display device. The execution logic of this implementation can be: second correction posture → electronic control chassis → optical machine rotation.

[0073] In an embodiment of the present disclosure, a reference posture for processing the image to be displayed at the current moment can be determined by combining the current angular velocity data of the head-mounted display device and a first correction posture used to process the image displayed by the head-mounted display device at a previous moment. Subsequently, a posture error between the reference posture and a preset posture used to process the image can be processed, or the reference posture can be processed to obtain a correction posture difference, and the correction posture difference can be used to compensate the reference posture to obtain a second correction posture for processing the image to be displayed at the current moment. The second correction posture can be considered to be determined by reference to the current angular velocity data of the head-mounted display device, the first correction posture, and the preset posture, and is adapted to the posture of the head-mounted display device (which can actually be regarded as the user) at the current moment. The image to be displayed by the head-mounted display device is rendered based on this posture, which facilitates correcting the display effect of the head-mounted display device based on the display effect of the head-mounted display device in the preset posture. In this way, no matter whether a small angle change occurs during the use of the head-mounted display device (bumping, shaking, trembling, etc.) or a large angle change occurs during the use of the head-mounted display device (such as a large angle turn), it can ensure that the head-mounted display device adjusts the display effect accordingly to the preset posture, thereby improving the user experience.

[0074] Optionally, each posture involved in the embodiments of the present disclosure may be in the form of a rotation matrix; each correction posture involved in the embodiments of the present disclosure may be a posture for image rendering by a rendering engine of a head-mounted display device.

[0075] It should be noted that for head-mounted display devices, there are currently multiple modes when presenting content, which may include a first mode and a second mode. For example, the first mode may be a 3-degree-of-freedom (dof) mode. In the first mode, the head-mounted display device performs positioning processing on the display screen. At this time, the content of the display screen of the head-mounted display device is related to the user's posture, and the display screen of the head-mounted display device can remain rotationally unchanged with respect to the world coordinate system. The second mode is similar to the screen projection mode from a mobile phone to a TV. In the second mode, the head-mounted display device does not perform positioning processing on the display screen. At this time, the content of the display screen of the head-mounted display device may be related to the content playback progress, but not to the user's posture. The display screen of the head-mounted display device moves with the movement of the user's head. For the sake of convenience, the second mode may be referred to as a 0-degree-of-freedom (dof) mode.

[0076] It is understandable that users can use head-mounted display devices on vehicles, ships, airplanes, or other types of mobile platforms. In 0dof mode, if the mobile platform operates smoothly, the display effect of the head-mounted display device is acceptable. However, if there is bumping or shaking, for example, due to uneven road conditions, the moving vehicle bumps or shakes, or due to airflow, the user may shake or shake the head-mounted display device. In this 0dof mode, the display screen will also shake or shake. In this case, the user will not be able to stably and clearly observe the content of the display screen of the head-mounted display device, which will reduce the user experience.

[0077] In 3Dof mode, if the aforementioned bumps and shakes occur, the display image is positioned so that even if the head-mounted display device shakes or shakes, the positioning of the display image remains unchanged, and the user can clearly see the content displayed on the head-mounted display device. However, if the direction of the mobile platform changes (for example, a 90-degree turn), since the user (head-mounted display device) also changes direction with the mobile platform, and the positioning of the display image remains unchanged, the user must return to the posture before the direction change to view the desired display image, which will also reduce the user experience.

[0078] Therefore, through the embodiments of the present disclosure, the display effect of the head-mounted display device can be calibrated based on the display effect of the head-mounted display device in a preset posture. For example, the display image of the head-mounted display device can be processed based on the calibrated posture so that when the angle changes during use of the head-mounted display device are small, the display effect is similar to 3Dof mode, and when the angle changes are large, the display effect is similar to 0Dof mode, thereby improving the user experience.

[0079] In an optional example, the head-mounted display device is configured to be coupled to a mobile platform and movable relative to the mobile platform.

[0080] It should be noted that the head-mounted display device can be coupled to the mobile platform through the user, and the mobile platform can be in motion. Optionally, as shown in Figure 4, the head-mounted display device can be worn on the head of a user (such as a passenger in a vehicle), and the head-mounted display device and the user remain relatively stationary. The user can sit or stand inside the vehicle, and when the vehicle moves, the user will move with the vehicle. The user can also move inside the vehicle (for example, from one position to another, or from one direction to another, etc.). In addition, when the mobile platform is of other types, such as a ship, the user can also sit or stand on the side of the ship.

[0081] It should be noted that for scenarios where the head-mounted display device is configured to be coupled to the mobile platform on which it is located and movable relative to the mobile platform, the mobile platform may be bumpy or shaken during movement, or the mobile platform may turn during movement. In view of this, the method steps disclosed in the present invention can be performed for this scenario to ensure the display effect of the head-mounted display device and enhance the user experience.

[0082] It's worth noting that currently, one solution for stabilizing the display of mobile phones (or computers) on mobile platforms involves the user manually adjusting the relative position and angle between the phone's actual display screen and the user's eyes. This means that the phone's actual display screen experiences relative displacement and rotation relative to the user's eyes. In contrast, with the stabilization solution in the embodiments of this disclosure, the head-mounted display device experiences virtually no relative displacement or rotation relative to the user's eyes.

[0083] Based on the embodiment shown in FIG1 , as shown in FIG5 , step 120 includes:

[0084] Step 1201: Low-pass filter the attitude error corresponding to the current moment to obtain a corrected attitude difference.

[0085] It should be noted that the attitude error obtained by subtracting the reference attitude determined in step 110 from the preset attitude is the attitude error corresponding to the current moment. If the preset attitude and the reference attitude are in the same coordinate system, and the preset attitude is a zero-rotation attitude, the reference attitude can be low-pass filtered to obtain a corrected attitude difference.

[0086] Optionally, the attitude error corresponding to the current moment can be low-pass filtered using low-pass filters such as infinite impulse response (IIR) digital filters and finite impulse response (FIR) digital filters. Each output value of the IIR digital filter can only depend on a previous sampling value, so it is very convenient for engineering applications. The low-pass filtering process of the FIR digital filter involves a sliding window, and each output value of the FIR digital filter can be obtained by calculating a number of previous sampling values. Assuming that the attitude error corresponding to the current moment is expressed as error, and the corrected attitude difference is expressed as error1, then: error1=LowPassfilter(error)

[0087] It is understandable that under ideal conditions, only data below the cutoff frequency can pass through, and the value of the correction attitude difference error1 is not 0, that is, the second correction attitude after the reference attitude is subsequently compensated by the correction attitude difference error1 is different from the reference attitude. If data above the cutoff frequency cannot pass through, the value of the correction attitude difference error1 is 0, that is, the second correction attitude after the reference attitude is subsequently compensated by the correction attitude difference error1 is the same as the reference attitude. However, under actual working conditions, the value of the correction attitude difference error1 may rarely be 0, and may be a very small value that is not noticeable to the user.

[0088] In the embodiment of the present disclosure, as shown in FIG6 , the angular velocity data Gyro collected by the gyroscope in the IMU at the current moment is obtained. k Afterwards, you can use the current angular velocity data Gyro k and the first correction posture Pose used to process the display image of the head-mounted display device at the previous moment k-1 Perform integration processing to obtain the reference posture for processing the display image at the current moment By By subtracting the error from the preset attitude Target, we can get the attitude error error corresponding to the current moment. For example, by low-pass filtering the error, we can get the corrected attitude difference error1. error1 can be used as the compensation value to update Finally, the second correction pose Pose is obtained k . Pose k It can be considered as a posture that can achieve anti-shake, Pose k It can be used to process the display image at the current moment. k-1 Get Pose as input k In the following, a similar method can be used to convert Pose k Get Pose as input k+1 , change Pose k+1 Get Pose as input k+2Etc. In this way, by performing low-pass filtering on the posture error, high-frequency jitter and shaking (such as short-term, small-angle movements) can be filtered out, and the display screen of the head-mounted display device will tend to be stationary, making it easier for the user to clearly and stably observe the content of the display screen of the head-mounted display device, which can improve the user experience. In addition, low-frequency movements (such as long-term, large-angle movements) will be retained. If the user turns their head or the mobile platform turns (for example, the mobile platform turns and the head turns with it), causing the head-mounted display device to move at a large angle, after compensation, the display screen of the head-mounted display device will be positioned to follow the user's head, so that the user can observe the desired display screen, which can also improve the user experience.

[0089] Based on the embodiment shown in FIG5 , as shown in FIG7 , step 1201 includes:

[0090] Step 12011: low-pass filter the attitude error corresponding to the current moment.

[0091] The method further includes steps 150 and 160 .

[0092] Step 150: Receive user input.

[0093] Optionally, the user input operation includes, but is not limited to, voice input operation, touch input operation, etc. For example, the user input operation may be an operation in which the user says "adjust the cutoff frequency to 2 Hz (or 5 Hz or other frequencies)". Alternatively, the head-mounted display device may have a paired controller, and the user input operation may be an operation in which the user clicks a cutoff frequency adjustment button on the controller. Alternatively, a cutoff frequency selection box or input box may be displayed on the display screen, and the user may directly select or input the desired cutoff frequency.

[0094] Step 160 : In response to a user input operation, adjust the cutoff frequency of the low-pass filter.

[0095] In step 160 , the cutoff frequency required by the user may be determined according to the user input operation, and then the cutoff frequency of the low-pass filter may be adjusted to the cutoff frequency required by the user in any feasible manner.

[0096] As described above, through the low-pass filtering process, when the head-mounted display device's movement frequency is no higher than the low-pass filter's cutoff frequency, the head-mounted display device's display image changes with the head-mounted display device's movement. However, when the head-mounted display device's movement frequency is higher than the low-pass filter's cutoff frequency, the head-mounted display device's display image tends to remain stationary. It is understandable that this method may be less sensitive to movement closer to the cutoff frequency. Thus, the higher the head-mounted display device's movement frequency is above the cutoff frequency, the closer the head-mounted display device's actual mode is to the 3Dof mode. The lower the head-mounted display device's movement frequency is below the cutoff frequency, the closer the head-mounted display device's actual mode is to the 0Dof mode. Therefore, it is understandable that the higher the cutoff frequency, the more likely the frequency of most movement is below the cutoff frequency. Therefore, this solution compensates for most movement of the head-mounted display device, ensuring that the display image follows the movement of the head-mounted display device, and the closer the head-mounted display device's actual mode is to the 0Dof mode. The lower the cutoff frequency, the more likely the frequency of most movements is to be higher than the cutoff frequency. In this case, the scheme will compensate for most movements of the head-mounted display device so that the display image tends to be stationary, and the actual mode of the head-mounted display device will be closer to the 3Dof mode.

[0097] In the embodiment of the present disclosure, by adjusting the cutoff frequency of the low-pass filter, the user can make the actual mode of the head-mounted display device closer to the 0dof mode or closer to the 3dof mode according to actual needs.

[0098] It should be pointed out that the above description is about the case where the cutoff frequency of the low-pass filter is adjustable. In actual implementation, the cutoff frequency of the low-pass filter may also be set to an unadjustable fixed value.

[0099] Based on the embodiment shown in FIG1 , as shown in FIG8 , step 120 includes:

[0100] Step 1203: Correct the posture error corresponding to the current moment using the correction proportional coefficient to obtain a corrected posture difference. The correction proportional coefficient is greater than 0 and less than 1.

[0101] Optionally, the correction proportional coefficient may be 0.2, 0.5, 0.6, 0.8 or other values ​​between 0 and 1, which are not listed here one by one.

[0102] It can be understood that when the preset posture and the reference posture are postures in the same coordinate system and the preset posture is a posture with zero rotation, the reference posture can be corrected using the correction scale coefficient to obtain a corrected posture difference.

[0103] In step 1203, the posture error corresponding to the current moment may be multiplied by the correction proportional coefficient, and the multiplication result is used as the correction posture difference.

[0104] It is understandable that in the presence of jitter and shaking, the user's head moves back and forth, causing the actual posture of the head-mounted display device to be positive or negative, and the posture error to be positive or negative. After compensation, the display screen of the head-mounted display device will tend to be stationary, making it easier for the user to clearly and stably observe the content of the display screen of the head-mounted display device, which can improve the user's experience. In addition, if the user turns his head or the mobile platform turns (for example, the mobile platform turns and causes the user's head to turn), causing the head-mounted display device to turn, the actual posture of the head-mounted display device is always positive or negative, and the posture error is also always positive or negative. After compensation, the display screen of the head-mounted display device will follow the direction of the user's head, so that the user can observe the desired display screen, which can also improve the user's experience.

[0105] It should be noted that the larger the correction scale factor, the closer the correction posture difference is to the posture error. When the correction posture difference is used to determine the second correction posture, the second correction posture will be closer to the preset posture. Accordingly, the head-mounted display device's display image will be corrected faster. Thus, the larger the correction scale factor, the closer the actual mode of the head-mounted display device is to the 0dof mode, while the smaller the correction scale factor, the closer the actual mode of the head-mounted display device is to the 3dof mode.

[0106] In the embodiment of the present disclosure, by introducing a correction ratio coefficient to correct the posture error, the correction posture of the head-mounted display device can be made close to the preset posture, and the correction posture is used for rendering the display screen of the head-mounted display device, which is conducive to correcting the display effect of the head-mounted display device according to the display effect of the head-mounted display device in the preset posture. In this way, whether the head-mounted display device undergoes a small angle change during use (bumping, shaking, jittering, etc.), or the head-mounted display device undergoes a large angle change during use (such as a large angle turn), it can ensure that the head-mounted display device adjusts the display effect accordingly to the preset posture, thereby improving the user experience. In addition, by controlling the correction ratio coefficient, the user can make the actual mode of the head-mounted display device closer to the 0dof mode or closer to the 3dof mode according to actual needs.

[0107] Based on the embodiment shown in FIG8 , as shown in FIG9 , the method further includes:

[0108] Step 115: Determine a correction scale coefficient based on compensation reference information, where the compensation reference information includes at least one of the following: a posture error corresponding to the current moment, a cumulative value of the posture error, an environmental condition of the mobile platform where the head-mounted display device is located, and steering information of the mobile platform.

[0109] Optionally, step 115 can be performed before step 120, that is, after determining the correction proportional coefficient based on the compensation reference information, the determined correction proportional coefficient can be used to correct the posture error. Alternatively, step 115 can be performed after step 120, that is, after correcting the posture error using the correction proportional coefficient, the correction proportional coefficient can be determined based on the compensation reference information. In order to use the determination result to dynamically update the correction proportional coefficient. In this way, the updated correction proportional coefficient can be subsequently used to correct the posture error. The embodiments of the present disclosure do not impose any restrictions on the execution order of steps 115 and 120. Therefore, it can be understood that the correction proportional coefficient in the embodiments of the present disclosure can be dynamically adjusted based on the above factors.

[0110] It can be understood that when the preset posture and the reference posture are postures in the same coordinate system and the preset posture is a posture of zero rotation, the posture error corresponding to the current moment and the cumulative value of the posture error can be understood as: the reference posture corresponding to the current moment and the cumulative value of the reference posture.

[0111] The environmental conditions of the mobile platform on which the head-mounted display device resides can be determined based on information collected by devices installed on the platform (e.g., cameras, radar, etc.). If the mobile platform on which the head-mounted display device resides is a vehicle, the environmental conditions of the mobile platform on which the head-mounted display device resides may include road surface information of the road on which the vehicle resides, which may include road surface smoothness. If the mobile platform on which the head-mounted display device resides is an aircraft, the environmental conditions of the mobile platform on which the head-mounted display device resides may include airflow intensity at the aircraft's location.

[0112] The steering information of the mobile platform on which the head mounted display device is mounted may include a steering degree, which may be represented by a steering angle. In the case where the mobile platform is a vehicle, the steering information may be detected by an onboard steering system.

[0113] Optionally, the correction scale coefficient is positively correlated with the posture error corresponding to the current moment.

[0114] The greater the difference between the reference posture and the preset posture, the larger the correction scale coefficient. A larger correction scale coefficient is conducive to correcting the display screen of the head-mounted display device according to the preset posture more quickly, thereby helping to ensure the display effect of the head-mounted display device.

[0115] Optionally, the correction scale coefficient is positively correlated with the cumulative value of the posture error. The cumulative value of the posture error can be the cumulative value of the posture error within a preset time period, or it can be the cumulative value of the continuous posture error. Compared with the above example, the correction scale coefficient is determined by the overall movement trend before the screen is displayed, which can make the correction scale coefficient more consistent with the actual posture error. For example, when the mobile platform makes a large-angle turn, the display screen of the head-mounted display device can be quickly corrected according to the preset posture, which is conducive to ensuring the display effect of the head-mounted display device.

[0116] It is understood that the value of the posture error can be either positive or negative. For example, if the value of the posture error in a first direction is defined as a positive number, then the value of the posture error in a second direction opposite the first direction is defined as a negative number. Both positive and negative posture errors can be processed. When evaluating the "large," "small," and "positive correlation" of posture errors, the absolute value of the posture error can be used as a reference.

[0117] Optionally, the environmental conditions include road surface smoothness, and the correction scaling factor is positively correlated with the road surface smoothness. Thus, the less smooth the road surface, the smaller the correction scaling factor. A smaller correction scaling factor allows the head-mounted display device's display image to be corrected more slowly based on the preset posture, thereby facilitating image stability.

[0118] Optionally, the steering information includes a steering degree, and the correction scale factor is positively correlated with the steering degree. Thus, the greater the change in the direction of the mobile platform, the larger the correction scale factor. A larger correction scale factor facilitates faster correction of the head-mounted display image according to the preset posture, thereby ensuring that the head-mounted display image follows the user's posture.

[0119] In the embodiments of the present disclosure, the correction proportional coefficient can be reasonably determined with reference to at least one of the posture error corresponding to the current moment, the environmental conditions of the mobile platform where the head-mounted display device is located, and the steering information of the mobile platform, so that the determined correction proportional coefficient is adapted to the actual situation.

[0120] Of course, it is understandable that, similar to the adjustment scheme of the cutoff frequency, the user can also adjust the above correction proportional coefficient according to usage requirements.

[0121] Based on the embodiment shown in FIG. 8 , as shown in FIG. 10 , the method further includes step 180 and step 190 .

[0122] Step 180: determining a variation pattern of the attitude error based on the attitude error sequence, where the attitude error sequence includes attitude errors corresponding to multiple moments.

[0123] It should be noted that after determining the reference posture corresponding to the current moment by executing step 110, the posture error corresponding to the current moment can be determined by subtracting the reference posture corresponding to the current moment from the preset posture. In a similar manner, the posture error corresponding to each moment can be determined, thereby obtaining multiple posture errors corresponding to multiple moments. The multiple posture errors can be arranged in order from earliest to latest according to the corresponding moments to form a posture error sequence.

[0124] It can be understood that when the preset posture and the reference posture are postures in the same coordinate system and the preset posture is a posture with zero rotation, the posture error variation rule can be determined based on the reference posture sequence.

[0125] In an optional example, the posture error sequence may include 20 posture errors arranged in sequence, namely R1, R2, R3, ..., R20, where the time corresponding to R1 may be t1, the time corresponding to R2 may be t2, the time corresponding to R3 may be t3, ..., and the time corresponding to R20 may be t20. Optionally, t20 may be the current time.

[0126] By analyzing the posture error sequence, the variation pattern of the posture error can be determined. The variation pattern of the posture error includes but is not limited to the following information: whether the overall trend of the posture error is increasing or decreasing; the time at which the maximum and minimum values ​​of the posture error correspond; the number of maximum and minimum values ​​of the posture error; the time at which each maximum and minimum value corresponds; and the relationship between the posture error corresponding to each time after a certain time (which can be considered as a reference time) and the posture error corresponding to the reference time.

[0127] Step 190: Adjust the correction scale coefficient based on the change rule of the posture error.

[0128] It should be noted that step 190 can be implemented in various forms, which are described below with examples.

[0129] In an optional implementation, as shown in FIG11-1 , step 190 includes:

[0130] Step 1901: When it is determined based on the attitude error variation rule that the attitude error corresponding to each moment in the target time period is greater than or equal to the attitude error corresponding to the target moment, the correction proportional coefficient is increased.

[0131] The target time, the start time of the target time period, and the end time of the target time period are each one of the multiple times. The start time of the target time period is later than the target time, and the duration of the target time period is greater than or equal to the preset duration.

[0132] Optionally, the preset duration may be 3 seconds, 4 seconds, 5 seconds, 8 seconds or other durations, which are not listed here one by one.

[0133] In an optional example, the posture error sequence may include 20 posture errors arranged in sequence, namely R1, R2, R3, ..., R20, where the time corresponding to R1 may be t1, the time corresponding to R2 may be t2, the time corresponding to R3 may be t3, ..., and the time corresponding to R20 may be t20. Optionally, t20 may be the current time.

[0134] Assuming t13 among t1 to t20 is the target time, the change pattern of the posture error can include the magnitude relationship between each of R14 to R20 and R13. Assuming that the interval between any two adjacent moments is 0.5 seconds, the preset duration is 3 seconds, the start time of the target time period is t14, and the end time of the target time period is t20, then if each of R14 to R20 is greater than or equal to R13, it can be considered that the posture error has been continuously high since time t13. In other words, when correcting the posture error based on the current correction scale coefficient, the correction effect is not ideal. In this case, the correction scale coefficient can be increased, for example, from 0.5 to 0.6 or 0.7. By increasing the correction scale coefficient, the display image of the head-mounted display device can be corrected more quickly according to the preset posture, which helps to ensure the display effect of the head-mounted display device.

[0135] In another optional implementation, as shown in FIG11-2 , step 190 includes:

[0136] Step 1903 : When it is determined based on the change rule of the posture error that the posture error corresponding to each moment in the target time period is smaller than the posture error corresponding to the target moment, the correction proportional coefficient is reduced.

[0137] The target time, the start time of the target time period, and the end time of the target time period are each one of the multiple times. The start time of the target time period is later than the target time, and the duration of the target time period is greater than or equal to the preset duration.

[0138] Optionally, the preset duration may be 3 seconds, 4 seconds, 5 seconds, 8 seconds or other durations, which are not listed here one by one.

[0139] Continuing with the example from the previous implementation, if each of R14 to R20 is less than R13, it can be assumed that the posture error has remained low since time t13. In other words, when correcting the posture error based on the current correction scale factor, the correction effect is satisfactory. In this case, the correction scale factor can be reduced, for example, from 0.5 to 0.3 or 0.4. By reducing the correction scale factor, the head-mounted display's display image can be corrected more slowly according to the preset posture, thereby helping to ensure the stability of the head-mounted display's display image.

[0140] In yet another optional implementation, as shown in FIG11-3 , step 190 includes:

[0141] Step 1905: When it is determined based on the change law of the posture error that the change trend of the posture error is an increasing trend, the correction proportional coefficient is increased.

[0142] Assuming that based on the law of change of posture error, the change trend of posture error is determined to be an increasing trend, for example, continuing the example in the first implementation form, if R1, R2, R3, ..., R20 increase in sequence, or although R1, R2, R3, ..., R20 do not increase in sequence, but the overall trend is gradually increasing (for example, the first 10 posture errors of 20 posture errors increase in sequence, the 11th posture error and the 12th posture error are the same, and the last 8 posture errors increase in sequence), then the correction proportional coefficient can be increased. For example, the correction proportional coefficient can be adjusted from 0.5 to 0.6 or 0.7. By increasing the correction proportional coefficient, the display screen of the head-mounted display device can be corrected more quickly according to the preset posture, which is conducive to ensuring the display effect of the head-mounted display device.

[0143] In yet another optional implementation, as shown in FIG11-4 , step 190 includes:

[0144] Step 1907: When it is determined based on the change law of the posture error that the change trend of the posture error is a decreasing trend, the correction proportional coefficient is reduced.

[0145] Assuming that based on the law of change of posture error, the change trend of posture error is determined to be a decreasing trend, for example, continuing the example in the first implementation form, if R1, R2, R3, ..., R20 decreases in sequence, or although R1, R2, R3, ..., R20 do not decrease in sequence, the overall trend is gradually decreasing (for example, the first 10 posture errors of 20 posture errors decrease in sequence, the 11th posture error is the same as the 12th posture error, and the last 8 posture errors decrease in sequence), then the correction proportional coefficient can be reduced. For example, the correction proportional coefficient can be adjusted from 0.5 to 0.3 or 0.4. By reducing the correction proportional coefficient, the speed at which the display screen of the head-mounted display device is corrected according to the preset posture can be made more gentle, which is conducive to ensuring the stability of the display screen of the head-mounted display device.

[0146] It can be seen that in the embodiments of the present disclosure, by referring to the change law of the posture error and adjusting the correction proportional coefficient, the display image of the head-mounted display device can be corrected at a more appropriate speed according to the actual situation, thereby ensuring the display effect and display image stability of the head-mounted display device.

[0147] Based on the embodiment shown in FIG. 1 , as shown in FIG. 12 , the method further includes step 102 , step 104 and step 106 .

[0148] Step 102: Identify the type of platform on which the head-mounted display device is located.

[0149] It should be noted that there are various implementations for identifying the type of platform on which the head-mounted display device is located, which are described below with examples.

[0150] In an optional implementation, the head-mounted display device can establish a communication connection with the platform on which it is located. During the process of establishing the communication connection, relevant information about the platform can be obtained. The relevant information about the platform includes, but is not limited to, identification information and type information. By extracting the type information from the obtained relevant information and identifying the category represented by the type information, it can be determined whether the platform is a mobile platform.

[0151] In another optional implementation, the head-mounted display device may further include a camera. The head-mounted display device may use the camera to capture images of its surroundings and perform object detection on the images to determine whether the images match a mobile platform such as a vehicle. If so, the platform on which the head-mounted display device resides may be determined to be a mobile platform. Otherwise, the platform on which the head-mounted display device resides may be determined to be not a mobile platform.

[0152] In another optional implementation, acceleration data collected by an accelerometer or angular velocity data collected by a gyroscope in the head-mounted display device's IMU can be obtained and compared with a preset threshold. If the acceleration of the head-mounted display device is determined to be greater than the preset acceleration threshold based on the acceleration data, or if the angular velocity of the head-mounted display device is determined to be greater than the preset angular velocity threshold based on the angular velocity data, then the type of platform on which the head-mounted display device resides can be determined to be a mobile platform. Otherwise, it can be determined that the type of platform on which the head-mounted display device resides is not a mobile platform.

[0153] Step 104 : When the platform on which the head mounted display device is located is a mobile platform, control the head mounted display device to switch to a target mode.

[0154] In an embodiment of the present disclosure, the head mounted display device may include at least three modes, and the at least three modes may include the above-mentioned 0dof mode, 3dof mode, and the target mode in step 104.

[0155] Step 106 , determining whether the head mounted display device is in the target mode, and if so, triggering step 110 .

[0156] In an embodiment of the present disclosure, the type of platform on which the head-mounted display device is located can be identified. If the category of the platform is not a mobile platform, the head-mounted display device can be placed in a conventional 0dof mode or 3dof mode. If the category of the platform is a mobile platform, the head-mounted display device can be automatically switched to a target mode, so that in the target mode, step 110 is triggered to implement the execution of steps 110 to 140 above. As described above, the target mode is a mode between the 0dof mode and the 3dof mode. In the target mode, when the mobile platform is bumpy or shaking, the display screen of the head-mounted display device can be adjusted to compensate for the bumps and shakes to keep the screen stable. Moreover, when the mobile platform or the user's head turns, the display screen of the head-mounted display device can be kept positioned following the user's head, thereby improving the user experience.

[0157] It should be noted that the above description automatically switches the head-mounted display device to target mode when the head-mounted display device is on a mobile platform. Of course, users can also manually select any mode among 0dof mode, 3dof mode, or target mode according to actual needs.

[0158] It should be noted that the mobile platform involved in the embodiments of the present disclosure may also be referred to as a sports vehicle; the angular velocity data in the embodiments of the present disclosure may also be collected not by the gyroscope in the IMU, but by other sensors that can obtain angular velocity data.

[0159] In summary, in the embodiments of the present disclosure, without using the acceleration data collected by the IMU, by using the angular velocity data collected by the IMU, the display effect of the head-mounted display device can be corrected according to the display effect of the head-mounted display device in a preset posture. For the head-mounted display device, the display effect can be guaranteed regardless of whether it is a short-term small-angle movement or a long-term large-angle movement.

[0160] Any of the methods for displaying images on a head-mounted display device provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server. Alternatively, any of the methods for displaying images on a head-mounted display device provided in the embodiments of the present disclosure can be executed by a processor, such as by invoking corresponding instructions stored in a memory to execute any of the methods for displaying images on a head-mounted display device mentioned in the embodiments of the present disclosure. This will not be further described below.

[0161] An embodiment of the present disclosure also provides a method for displaying an image on a head-mounted display device, the method comprising: acquiring angular velocity data of the head-mounted display device; determining a reference posture for processing the image to be displayed based on the angular velocity data and a first correction posture previously used to process the display image of the head-mounted display device; processing the reference posture to obtain a correction posture difference; compensating the reference posture using the correction posture difference to obtain a second correction posture for processing the image to be displayed; and rendering the image to be displayed of the head-mounted display device based on the second correction posture to display the image to be displayed of the head-mounted display device.

[0162] An embodiment of the present disclosure also provides a method for displaying an image on a head-mounted display device, the method comprising: acquiring angular velocity data of the head-mounted display device, and determining a reference posture for processing the image to be displayed based on the angular velocity data and a first correction posture previously used to process the display image of the head-mounted display device; processing a posture error between the reference posture and a preset posture for processing the display image to obtain a correction posture difference; compensating the reference posture using the correction posture difference to obtain a second correction posture for processing the image to be displayed; and rendering the image to be displayed of the head-mounted display device based on the second correction posture to display the image to be displayed of the head-mounted display device.

[0163] An embodiment of the present disclosure also provides a method for displaying an image on a head-mounted display device, the method comprising: determining a reference posture for processing the image to be displayed at the current moment based on angular velocity data of the head-mounted display device at the current moment and a first correction posture used to process the display image of the head-mounted display device at the previous moment; processing the reference posture to obtain a correction posture difference; compensating the reference posture using the correction posture difference to obtain a second correction posture for processing the image to be displayed at the current moment; and rendering the image to be displayed of the head-mounted display device based on the second correction posture to display the image to be displayed of the head-mounted display device.

[0164] In some optional embodiments, in response to a user selection, the head-mounted display device display mode is switched to one of a 0-DOF mode, a target mode, and a 3-DOF mode. When the head-mounted display device is in the target mode, a step is performed to determine a reference posture for processing the image to be displayed at the current moment based on the angular velocity data of the head-mounted display device and a first correction posture used to process the image displayed by the head-mounted display device at a previous moment.

[0165] In some optional embodiments, the method for displaying an image on a head-mounted display device further includes: receiving a user input operation; and adjusting the correction scale coefficient in response to the user input operation.

[0166] In some optional embodiments, the method for displaying an image on a head-mounted display device further includes: after correcting the posture error using the correction scale coefficient, the correction scale coefficient can be determined based on the compensation reference information, so as to dynamically update the correction scale coefficient using the determination result.

[0167] Exemplary devices

[0168] Figure 13 is a structural diagram of an apparatus for displaying images on a head-mounted display device provided by an exemplary embodiment of the present disclosure. The apparatus shown in Figure 13 includes a first determination module 1310, a first processing module 1320, a compensation module 1330 and a second processing module 1340.

[0169] A first determining module 1310 is configured to determine a reference posture for processing a display image at a current moment based on the angular velocity data of the head-mounted display device at a current moment and a first correction posture used to process a display image of the head-mounted display device at a previous moment;

[0170] A first processing module 1320 is configured to process a posture error between a reference posture and a preset posture for processing a display image to obtain a corrected posture difference;

[0171] A compensation module 1330 is configured to compensate the reference posture using the correction posture difference to obtain a second correction posture for processing the image to be displayed at a current moment;

[0172] The second processing module 1340 is configured to render the image to be displayed on the head-mounted display device based on the second correction posture, so as to display the image to be displayed on the head-mounted display device.

[0173] In an optional example, the first processing module 1320 is configured to:

[0174] The attitude error corresponding to the current moment is low-pass filtered to obtain the corrected attitude difference.

[0175] In an alternative example,

[0176] The first processing module 1320 is configured to:

[0177] Perform low-pass filtering on the attitude error corresponding to the current moment through a low-pass filter;

[0178] As shown in FIG14 , the device further includes:

[0179] Receiving module 1350, used to receive user input operation;

[0180] The first adjustment module 1360 is configured to adjust the cutoff frequency of the low-pass filter in response to a user input operation.

[0181] In an optional example, the first processing module 1320 is configured to:

[0182] The correction proportional coefficient is used to correct the posture error corresponding to the current moment to obtain a corrected posture difference, wherein the correction proportional coefficient is greater than 0 and less than 1.

[0183] In an optional example, as shown in FIG15 , the apparatus further includes:

[0184] The second determination module 1315 is used to determine the correction scale coefficient based on the compensation reference information, where the compensation reference information includes at least one of the following: the posture error corresponding to the current moment, the accumulated value of the posture error, the environmental conditions of the mobile platform where the head-mounted display device is located, and the steering information of the mobile platform.

[0185] In an optional example, at least one of the following four items is satisfied:

[0186] The correction scale coefficient is positively correlated with the attitude error corresponding to the current moment;

[0187] The correction scale coefficient is positively correlated with the cumulative value of the attitude error;

[0188] Environmental conditions include road surface smoothness, and the correction proportional coefficient is positively correlated with road surface smoothness;

[0189] The steering information includes the steering degree, and the correction proportional coefficient is positively correlated with the steering degree.

[0190] In an optional example, as shown in FIG16 , the apparatus further includes:

[0191] A third determining module 1380 is configured to determine a change pattern of the posture error based on a posture error sequence, where the posture error sequence includes posture errors corresponding to a plurality of moments;

[0192] The second adjustment module 1390 is used to adjust the correction scale coefficient based on the change rule of the posture error.

[0193] In an optional example, the second adjustment module 1390 includes at least one of the first adjustment unit 13901 , the second adjustment unit 13903 , the third adjustment unit 13905 , and the fourth adjustment unit 13907 shown in FIG16 ;

[0194] The first adjustment unit 13901 is configured to increase the correction proportional coefficient when it is determined based on a change rule of the posture error that the posture error corresponding to each moment in the target time period is greater than or equal to the posture error corresponding to the target moment;

[0195] The second adjustment unit 13903 is configured to reduce the correction proportional coefficient when it is determined based on the change law of the posture error that the posture error corresponding to each moment in the target time period is smaller than the posture error corresponding to the target moment;

[0196] The third adjustment unit 13905 is configured to increase the correction proportional coefficient when it is determined that the change trend of the posture error is an increasing trend based on the change law of the posture error;

[0197] The fourth adjustment unit 13907 is configured to reduce the correction proportional coefficient when it is determined that the change trend of the posture error is a decreasing trend based on the change law of the posture error;

[0198] The target time, the start time of the target time period, and the end time of the target time period are each one of the multiple times, the start time of the target time period is later than the target time, and the duration of the target time period is greater than or equal to the preset duration.

[0199] In an alternative example,

[0200] As shown in FIG17 , the device further includes:

[0201] Identification module 1302, used to identify the type of platform where the head mounted display device is located;

[0202] The control module 1304 is used to control the head mounted display device to switch to the target mode when the type is a mobile platform;

[0203] The triggering module 1306 is configured to trigger the first determining module 1310 when the head mounted display device is in the target mode.

[0204] In an optional example, the head-mounted display device is configured to be coupled to a mobile platform and movable relative to the mobile platform.

[0205] In an optional example, one of the following two conditions is met:

[0206] The preset posture is a posture for causing the display screen of the head-mounted display device to be displayed at a preset position in the coordinate system of the head-mounted display device;

[0207] The preset posture is a posture used to align the display screen of the head-mounted display device with the direction of gravity.

[0208] In an optional example, as shown in FIG18 , the apparatus further includes:

[0209] The configuration module 1395 is used to configure the optical machine in the head-mounted display device based on the second correction posture.

[0210] Exemplary electronic devices

[0211] 19 , an electronic device according to an embodiment of the present disclosure is described. The electronic device may be either or both of the first and second devices, or a standalone device independent of the first and second devices, and the standalone device may communicate with the first and second devices to receive collected input signals from them.

[0212] FIG19 illustrates a block diagram of an electronic device 1900 according to an embodiment of the present disclosure.

[0213] As shown in FIG. 19 , the electronic device 1900 includes one or more processors 1910 and a memory 1920 .

[0214] The processor 1910 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1900 to perform desired functions.

[0215] The memory 1920 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1910 may execute the program instructions to implement the method for displaying images on a head-mounted display device and / or other desired functions of the various embodiments of the present disclosure described above. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0216] In one example, the electronic device 1900 may further include an input device 1930 and an output device 1940 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0217] For example, when the electronic device 1900 is a first device or a second device, the input device 1930 may be a microphone or a microphone array. When the electronic device 1900 is a standalone device, the input device 1930 may be a communication network connector for receiving collected input signals from the first device and the second device.

[0218] In addition, the input device 1930 may also include, for example, a keyboard, a mouse, and the like.

[0219] The output device 1940 can output various information to the outside, and can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto.

[0220] Of course, for simplicity, Figure 19 only shows some of the components related to the present disclosure in the electronic device 1900, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1900 may further include any other appropriate components depending on specific applications.

[0221] Exemplary computer program products and computer-readable storage media

[0222] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method for displaying an image on a head-mounted display device according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0223] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0224] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the method for displaying images on a head-mounted display device according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0225] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0226] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. It should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and non-limiting, and should not be construed as necessarily possessed by each embodiment of the present disclosure. The specific details disclosed above are provided for illustrative purposes and to facilitate understanding, and are not intended to be limiting. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0227] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For system embodiments, since they largely correspond to method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.

[0228] The block diagrams of the devices, devices, equipment, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0229] The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0230] It should also be noted that in the apparatus, device, and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0231] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0232] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for displaying an image on a head-mounted display device, comprising: Determining a reference posture for processing a display image at a current moment based on the angular velocity data of the head-mounted display device at a current moment and a first correction posture used to process a display image of the head-mounted display device at a previous moment; Processing a posture error between the reference posture and a preset posture for processing a display image to obtain a corrected posture difference; Compensating the reference posture using the correction posture difference to obtain a second correction posture for processing the image to be displayed at the current moment; Based on the second correction posture, the image to be displayed of the head-mounted display device is rendered to display the image to be displayed of the head-mounted display device.

2. The method according to claim 1, wherein The step of processing a posture error between the reference posture and a preset posture for processing a display image to obtain a corrected posture difference comprises: The posture error corresponding to the current moment is corrected using a correction proportional coefficient to obtain a corrected posture difference, wherein the correction proportional coefficient is greater than 0 and less than 1.

3. The method according to claim 2, further comprising: The correction proportional coefficient is determined based on compensation reference information, where the compensation reference information includes at least one of the following: a posture error corresponding to a current moment, a cumulative value of the posture error, an environmental condition of a mobile platform where the head-mounted display device is located, and steering information of the mobile platform.

4. The method according to claim 3, wherein: At least one of the following four conditions is met: The correction proportional coefficient is positively correlated with the posture error corresponding to the current moment; The correction proportional coefficient is positively correlated with the cumulative value of the posture error; The environmental condition includes road surface smoothness, and the correction proportional coefficient is positively correlated with the road surface smoothness; The steering information includes a steering degree, and the correction proportional coefficient is positively correlated with the steering degree.

5. The method according to claim 2, further comprising: Determining a change pattern of the attitude error based on an attitude error sequence, wherein the attitude error sequence includes attitude errors corresponding to a plurality of moments; Based on the change rule of the posture error, the correction proportional coefficient is adjusted.

6. The method according to claim 5, wherein: The adjusting of the correction proportional coefficient based on the change rule of the posture error includes at least one of the following four items: When it is determined based on the attitude error variation rule that the attitude error corresponding to each moment in the target time period is greater than or equal to the attitude error corresponding to the target moment, increasing the correction proportional coefficient; When it is determined based on the change law of the posture error that the posture error corresponding to each moment in the target time period is smaller than the posture error corresponding to the target moment, reducing the correction proportional coefficient; When it is determined based on the change law of the posture error that the change trend of the posture error is an increasing trend, increasing the correction proportional coefficient; When it is determined based on the change law of the posture error that the change trend of the posture error is a decreasing trend, reducing the correction proportional coefficient; Among them, the target moment, the starting moment of the target time period, and the ending moment of the target time period are respectively one moment among the multiple moments, the starting moment of the target time period is later than the target moment, and the duration of the target time period is greater than or equal to the preset duration.

7. The method according to claim 2, further comprising: Receive user input operations; In response to a user input operation, the correction scale factor is adjusted.

8. The method according to claim 1, wherein The step of processing a posture error between the reference posture and a preset posture for processing a display image to obtain a corrected posture difference comprises: The posture error corresponding to the current moment is subjected to low-pass filtering to obtain the corrected posture difference.

9. The method according to claim 8, wherein The low-pass filtering of the attitude error corresponding to the current moment includes: Perform low-pass filtering on the attitude error corresponding to the current moment through a low-pass filter; The method further comprises: Receive user input operations; In response to the user input operation, the cutoff frequency of the low-pass filter is adjusted.

10. The method according to any one of claims 1 to 9, further comprising: In response to the user's selection, the head mounted display device display mode is switched to one of the 0 degree of freedom mode, the target mode, and the 3 degree of freedom mode, When the head-mounted display device is in the target mode, the step of determining a reference posture for processing the display image at the current moment based on the angular velocity data of the head-mounted display device at the current moment and the first correction posture used to process the display image of the head-mounted display device at the previous moment is performed.

11. The method according to any one of claims 1 to 9, further comprising: Identifying the type of platform on which the head-mounted display device is located; In a case where the type is a mobile platform, controlling the head mounted display device to switch to a target mode; When the head-mounted display device is in the target mode, the step of determining a reference posture for processing the display image at the current moment based on the angular velocity data of the head-mounted display device at the current moment and the first correction posture used to process the display image of the head-mounted display device at the previous moment is performed.

12. The method according to any one of claims 1 to 9, wherein The head mounted display device is configured to be coupled to a mobile platform and movable relative to the mobile platform.

13. The method according to any one of claims 1 to 12, wherein At least one of the following two conditions is met: The preset posture is a posture for causing the display screen of the head-mounted display device to be displayed at a preset position in the coordinate system of the head-mounted display device; The preset posture is a posture for aligning a display screen of the head-mounted display device with a direction of gravity.

14. An electronic device comprising: a memory for storing a computer program product; A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, implements the method for displaying an image on a head-mounted display device according to any one of claims 1 to 13.

15. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method for displaying images on a head-mounted display device according to any one of claims 1 to 13 is implemented.

Images