Display method, device, chip and storage medium of virtual image

Abstract

This application relates to the field of augmented reality display technology, and discloses a method, device, chip, and storage medium for displaying virtual images. The method is applied to a head-mounted display device and includes: acquiring the head posture data of a wearer wearing the head-mounted display device and the original projection matrix of the head-mounted display device; extracting high-frequency vibration signals from the posture data; determining a high-frequency rotation matrix based on the high-frequency vibration signals; compensating the original projection matrix in reverse according to the high-frequency rotation matrix to obtain a compensation matrix for the head-mounted display device; and rendering and displaying a virtual image based on the compensation matrix to ensure stable display of the virtual image in the wearer's field of vision. Through the above method, this application eliminates the jitter of the virtual image in the wearer's field of vision and reduces the cost of the head-mounted display device.

Description

Technical Field

[0001] This application relates to the field of augmented reality display technology, specifically to a method, device, chip, and storage medium for displaying virtual images. Background Technology

[0002] Existing head-mounted display (HMD) devices, such as AR (Augmented Reality) glasses, typically overlay virtual images onto the user's field of view using optical waveguides, projection, or optical synthesis. Because the HMD is rigidly connected to the wearer's head, high-frequency vibrations of the head during movement are directly transmitted to the HMD's optomechanical structure, causing corresponding jitter in the virtual image within the user's field of vision.

[0003] Existing technologies can correct high-frequency jitter by using mechanical devices within head-mounted displays to translate the display screen or optical elements, thereby altering the physical position of the image relative to the wearer's eyes. However, these solutions require additional mechanical structures within the head-mounted display to precisely control the physical movement of the display elements, which undoubtedly increases the cost of the device. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a method, device, chip and storage medium for displaying virtual images, which eliminates the jitter of virtual images in the wearer's field of vision and reduces the cost of head-mounted display devices.

[0005] According to one aspect of the embodiments of this application, a method for displaying a virtual image is provided. The method is applied to a head-mounted display device and includes: acquiring posture data of a wearer's head wearing the head-mounted display device and the original projection matrix of the head-mounted display device; extracting high-frequency vibration signals from the posture data; determining a high-frequency rotation matrix based on the high-frequency vibration signals; compensating the original projection matrix in reverse according to the high-frequency rotation matrix to obtain a compensation matrix for the head-mounted display device; and rendering and displaying a virtual image according to the compensation matrix so that the virtual image is stably displayed in the wearer's field of vision.

[0006] In one alternative approach, the compensation matrix of the head-mounted display device is obtained by inversely compensating the original projection matrix based on the high-frequency rotation matrix, further comprising: determining the inverse matrix of the high-frequency rotation matrix; and determining the compensation matrix of the head-mounted display device based on the inverse matrix of the high-frequency rotation matrix and the original projection matrix.

[0007] In one alternative approach, the compensation matrix for the head-mounted display is determined based on the inverse matrix of the high-frequency rotation matrix and the original projection matrix, including: according to the formula Calculate the compensation matrix ,in, Represents the original projection matrix. It represents the inverse matrix of the high-frequency rotation matrix.

[0008] In an alternative approach, the method further includes: obtaining the viewpoint matrix of the head-mounted display device; rendering and displaying a virtual image based on the compensation matrix; and further includes: multiplying the compensation matrix by the viewpoint matrix to obtain a target matrix of the head-mounted display device; and rendering and displaying a virtual image based on the target matrix.

[0009] In one alternative approach, the posture data includes a rotation angle and the frequency at which the wearer's head rotates; extracting a high-frequency vibration signal from the posture data further includes: determining whether the rotation angle is greater than a first preset angle and less than a second preset angle, wherein the first preset angle is less than the second preset angle and the absolute values ​​of the first preset angle and the second preset angle are equal; if the rotation angle is greater than the first preset angle and less than the second preset angle, determining whether the frequency is greater than a preset frequency threshold; if the frequency is greater than the preset frequency threshold, determining the rotation angle as a high-frequency vibration signal.

[0010] In one alternative approach, the rotation angle includes single-axis rotation angles about the X-axis, Y-axis, and Z-axis; determining the high-frequency rotation matrix based on the high-frequency vibration signal further includes: determining rotation matrices about the X-axis, Y-axis, and Z-axis respectively based on the single-axis rotation angles about the X-axis, Y-axis, and Z-axis; and multiplying the rotation matrices about the X-axis, Y-axis, and Z-axis sequentially to obtain the high-frequency rotation matrix.

[0011] In one optional approach, extracting high-frequency vibration signals from the posture data further includes: acquiring the delay time of the head-mounted display device, the delay time being the time interval between the generation of the posture data and the display of a virtual image based on the posture data; determining the predicted posture data for the current moment based on the posture data of the previous moment; correcting the predicted posture data based on the posture data of the current moment to obtain corrected posture data; determining the posture data for a delayed display moment based on the corrected posture data, the delayed display moment being separated from the current moment by the delay time; extracting the high-frequency vibration signal for the delayed display moment from the posture data of the delayed display moment; obtaining a compensation matrix for the head-mounted display device by inverse compensation of the original projection matrix based on the high-frequency rotation matrix; rendering and displaying the virtual image based on the compensation matrix to ensure stable display of the virtual image in the wearer's field of vision, further including: inverse compensation of the original projection matrix based on the high-frequency rotation matrix for the delayed display moment to obtain a compensation matrix for the delayed display moment of the head-mounted display device; rendering and displaying the virtual image for the current moment based on the compensation matrix for the delayed display moment to ensure stable display of the virtual image for the current moment in the wearer's field of vision.

[0012] According to another aspect of the embodiments of this application, a head-mounted display device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the virtual image display method provided in any of the above embodiments.

[0013] According to another aspect of the embodiments of this application, a chip is provided for use in a head-mounted display device. The chip stores a computer program, and the chip executes the computer program to implement the virtual image display method provided in any of the above embodiments.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the virtual image display method provided in any of the above embodiments.

[0015] This application first extracts high-frequency vibration signals from the head posture data of the wearer wearing the head-mounted display device, and converts the high-frequency vibration signals into a high-frequency rotation matrix. Then, the original projection matrix of the head-mounted display device is compensated in reverse using the high-frequency rotation matrix to obtain a compensation matrix. Finally, a virtual image is rendered and displayed using the compensation matrix. In the rendering stage, the virtual image can be compensated in reverse to adjust its position and angle in the wearer's field of vision, making the virtual image displayed stably in the wearer's field of vision and offsetting the effects of high-frequency head shaking. This eliminates the shaking of the virtual image in the wearer's field of vision without the need to add additional mechanical structures to the head-mounted display device, thus reducing the cost of the head-mounted display device.

[0016] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0017] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0018] Figure 1 A flowchart illustrating the virtual image display method provided in an embodiment of this application is shown;

[0019] Figure 2 A flowchart illustrating a method for displaying virtual images according to another embodiment of this application is shown;

[0020] Figure 3 A schematic diagram of the structure of the virtual image display device provided in an embodiment of this application is shown;

[0021] Figure 4 A schematic diagram of the structure of the head-mounted display device provided in an embodiment of this application is shown. Detailed Implementation

[0022] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein.

[0023] Existing head-mounted display devices typically superimpose virtual images onto the user's field of view using optical waveguides, projection, or optical synthesis. Because the head-mounted display device is rigidly connected to the wearer's head, when the wearer is in motion (such as walking, running, or riding in a vehicle), the high-frequency vibrations of the head are directly transmitted to the optomechanical structure of the head-mounted display device, causing the virtual image to jitter in the user's field of view.

[0024] Traditional image stabilization algorithms are primarily used in video shooting, eliminating image instability caused by camera shake through digital correction of sensors or images. However, in AR glasses, this shake occurs at the projection end and is tied to the coordinate mapping of the virtual content, making traditional stabilization algorithms inapplicable. Therefore, there is an urgent need for a technical solution that can compensate for shake at the display end, ensuring that AR images remain stably displayed in the wearer's field of vision even in motion scenarios.

[0025] Some existing solutions correct high-frequency jitter by using mechanical devices within the head-mounted display to translate the display screen or optical elements, thereby altering the physical position of the image relative to the wearer's eyes. However, these solutions require additional mechanical structures within the head-mounted display to precisely control the physical movement of the display elements, which undoubtedly increases the cost of the device.

[0026] Based on this, this application provides a method for displaying virtual images. This method collects the posture data of the wearer's head, extracts high-frequency vibration signals from the posture data, and converts them into a high-frequency rotation matrix for compensating for high-frequency vibrations. Then, the high-frequency rotation matrix is ​​used to inversely compensate the original projection matrix of the head-mounted display device to obtain a compensation matrix. The virtual image is then rendered and displayed using the compensation matrix. This method can adjust the position and angle of the virtual image in the wearer's field of vision, making the virtual image display stable in the wearer's field of vision. It eliminates the jitter of the virtual image in the wearer's field of vision without adding additional mechanical structures to the head-mounted display device, thereby reducing the cost of the head-mounted display device.

[0027] Figure 1A flowchart illustrating a method for displaying virtual images according to an embodiment of this application is shown. This method can be executed by an AR chip or a video processing unit (VPU) in a head-mounted display device, such as AR glasses. Figure 1 As shown, the method includes the following steps:

[0028] Step 100: Obtain the head posture data of the wearer wearing the head-mounted display device and the original projection matrix of the head-mounted display device.

[0029] Posture data refers to the three-dimensional posture information of the wearer's head in space, which may include rotation angles and the frequency of the wearer's head rotating. Among them, rotation angles may include single-axis rotation angles around the X-axis, Y-axis, and Z-axis.

[0030] Preferably, an inertial measurement unit (IMU) can be set in the head-mounted display device to collect the head posture data of the wearer of the head-mounted display device in real time.

[0031] The original projection matrix is ​​a matrix used by head-mounted display devices to project 3D scenes onto a 2D display plane. It describes the position and size of the virtual image on the screen.

[0032] For example, when a wearer walks while wearing AR glasses, the image processor of the head-mounted display can generate an initial projection matrix for initial rendering based on the current virtual scene settings. Simultaneously, the IMU can detect head posture data, such as minute rotation angles and the frequency of those rotations.

[0033] Step 200: Extract high-frequency vibration signals from attitude data.

[0034] This step can be implemented using either a high-pass filtering algorithm or a Fast Fourier Transform (FFT). Specifically, a high-pass filtering algorithm sets a cutoff frequency to filter out normal head rotation signals below that frequency from the attitude data, retaining only the attitude data with frequencies above that frequency, thus separating the high-frequency vibration signal. A Fast Fourier Transform converts the time-domain attitude data into frequency-domain data, identifies the high-frequency vibration components through spectral analysis, and separates them to obtain the high-frequency vibration signal.

[0035] Specifically, when the attitude data includes a rotation angle, step 200 may include the following steps:

[0036] Step 210: Determine whether the rotation angle is greater than the first preset angle and less than the second preset angle, the first preset angle is less than the second preset angle, and the absolute values ​​of the first preset angle and the second preset angle are equal.

[0037] Specifically, the shaking of the virtual image in this application is mainly caused by the low-amplitude, high-frequency movement of the wearer's head, such as frequent small-angle rotation of the head. Therefore, in order to counteract the shaking caused by the low-amplitude, high-frequency movement, it is first necessary to determine the posture data belonging to the low-amplitude, high-frequency movement from the collected posture data.

[0038] In this case, the absolute values ​​of the first preset angle and the second preset angle in this embodiment can be set to a smaller value. The range of the first preset angle can be [-0.5°, -0.1°], and the range of the second preset angle can be [0.1°, 0.5°]. For example, in application scenarios where head shaking is relatively gentle, the first preset angle can be set to -0.3° and the second preset angle can be set to +0.3°; in application scenarios where head shaking is relatively severe, the first preset angle can be set to -0.5° and the second preset angle can be set to 0.5°.

[0039] Step 220: If the rotation angle is greater than the first preset angle and less than the second preset angle, then determine whether the frequency is greater than the preset frequency threshold.

[0040] The frequency in this step refers to the frequency at which the wearer's head rotates. The preset frequency threshold can be set according to the actual application scenario and jitter characteristics to determine high-frequency, low-amplitude posture data from the wearer's low- and high-amplitude head posture data. In scenarios with frequent jitter, the preset frequency threshold can be set higher; in scenarios with less jitter, the preset frequency threshold can be set lower. The preset frequency threshold can range from [8Hz, 15Hz], for example, in a specific embodiment, the preset frequency threshold can be set to 10Hz.

[0041] Step 230: If the frequency is greater than the preset frequency threshold, the rotation angle is determined as a high-frequency vibration signal.

[0042] For example, when the wearer's head rotates at an angle of +0.2° and the frequency is 20Hz, it indicates that the wearer's head is vibrating at a high frequency. The rotation angle collected at this time is then identified as a high-frequency vibration signal.

[0043] When the wearer's head rotates at an angle of 5° and a frequency of 5Hz, it indicates that the wearer's head has undergone a large-amplitude low-frequency movement. The head-mounted display device recognizes this posture data as normal head movement and does not perform reverse compensation based on this posture data.

[0044] Through steps 210-230, reverse compensation can be performed based solely on low-amplitude and high-frequency posture data of the wearer's head, avoiding the occurrence of virtual image drift caused by reverse compensation based on posture data of normal head movement.

[0045] Step 300: Determine the high-frequency rotation matrix based on the high-frequency vibration signal.

[0046] Specifically, after extracting the high-frequency vibration signal, the rotation matrices around the X-axis, Y-axis, and Z-axis can be determined based on the single-axis rotation angles around the X-axis, Y-axis, and Z-axis of the high-frequency vibration signal, i.e., the rotation angles around the X-axis, Y-axis, and Z-axis. Then, the rotation matrices around the X-axis, Y-axis, and Z-axis are multiplied sequentially to obtain the high-frequency rotation matrix.

[0047] Assuming the rotation angles are (θx, θy, θz), where θx is the single-axis rotation angle around the X-axis, θy is the single-axis rotation angle around the Y-axis, and θz is the single-axis rotation angle around the Z-axis, then the rotation matrix around the X-axis is:

[0048] ,

[0049] The rotation matrix about the Y-axis is:

[0050] ,

[0051] The rotation matrix around the Z-axis is:

[0052] ,

[0053] Finally, the high-frequency rotation matrix is:

[0054] .

[0055] Step 400: Compensate the original projection matrix in reverse using the high-frequency rotation matrix to obtain the compensation matrix for the head-mounted display device.

[0056] Inverse compensation refers to using the inverse of a high-frequency rotation matrix to counteract the jitter factors contained in the original projection matrix.

[0057] First, the inverse of the high-frequency rotation matrix can be determined. Then, the compensation matrix for the head-mounted display device can be determined based on the inverse of the high-frequency rotation matrix and the original projection matrix. Specifically, the compensation matrix can be calculated using the following formula. :

[0058] ,

[0059] in, Represents the original projection matrix. It represents the inverse matrix of the high-frequency rotation matrix.

[0060] For example, if the high-frequency vibration of the head causes the virtual image to jitter 0.5° to the upper right, the virtual image rendered after compensating the original projection matrix based on the inverse matrix of the high-frequency rotation matrix will rotate 0.5° to the lower left. After the jitter 0.5° to the upper right and the rotation 0.5° to the lower left are superimposed, the jitter component in the projection matrix is ​​canceled out.

[0061] Step 500: Render and display the virtual image according to the compensation matrix to make the virtual image stably displayed in the wearer's field of vision.

[0062] Stable display means that the relative position of the virtual image in the wearer's eyes does not shift or shake significantly with high-frequency vibrations of the head.

[0063] Specifically, the viewpoint matrix of the head-mounted display device can be obtained, and then the compensation matrix can be multiplied by the viewpoint matrix to obtain the target matrix of the head-mounted display device. Finally, the virtual image is rendered and displayed based on the target matrix. Rendering and displaying the virtual image based on the target matrix also means multiplying the target matrix by the world coordinates of the virtual object to obtain the coordinates of the virtual object in the screen coordinates, thereby correctly drawing the virtual object onto the display screen of the head-mounted display device and realizing real-time cancellation of high-frequency jitter.

[0064] The viewpoint matrix is ​​a transformation matrix used to describe the position and orientation of the camera in a head-mounted display device. It transforms the coordinates of objects in the world coordinate system to the camera coordinate system, determining the wearer's viewing angle in the virtual image. For example, when the wearer's head turns 30 degrees to the left, the viewpoint matrix is ​​updated accordingly, causing objects in the virtual scene to undergo a visual displacement to the right relative to the observer, thus simulating a realistic viewing experience.

[0065] Multiplying the compensation matrix by the viewpoint matrix is ​​equivalent to adding a reverse high-frequency micro-motion correction to the observation viewpoint. Assuming the current viewpoint matrix represents the wearer looking straight ahead, and the compensation matrix includes a reverse rotation to counteract upward jitter, the resulting target matrix represents a slight downward pre-shift correction of the image while the wearer is looking straight ahead. Because the target matrix already incorporates the reverse high-frequency jitter compensation component, the rendered virtual image automatically cancels out the wearer's head vibrations in physical space.

[0066] For example, in AR navigation scenarios, the virtual turning arrows need to always be aligned with the intersection of the real road. Even if the wearer moves up and down due to vehicle bumps, the arrow image rendered by the target matrix will also move up and down in the opposite direction, so that it will be displayed stably in the wearer's field of vision and will not produce ghosting or violent shaking due to vehicle vibration.

[0067] This application first extracts high-frequency vibration signals from the head posture data of the wearer wearing the head-mounted display device, and converts the high-frequency vibration signals into a high-frequency rotation matrix. Then, the original projection matrix of the head-mounted display device is compensated in reverse using the high-frequency rotation matrix to obtain a compensation matrix. Finally, a virtual image is rendered and displayed using the compensation matrix. In the rendering stage, the virtual image can be compensated in reverse to adjust its position and angle in the wearer's field of vision, making the virtual image displayed stably in the wearer's field of vision and offsetting the effects of high-frequency head shaking. This eliminates the shaking of the virtual image in the wearer's field of vision without the need to add additional mechanical structures to the head-mounted display device, thus reducing the cost of the head-mounted display device.

[0068] In general, when a wearer's head experiences high-frequency shaking while wearing a head-mounted display device, the device detects the wearer's head posture data (t1), calculates a high-frequency rotation matrix (t2) based on the posture data, calculates a compensation matrix (t3) based on the high-frequency rotation matrix, and renders and displays a virtual image (t4) based on the compensation matrix. There is a delay between detecting the posture data (t1) and rendering and displaying the virtual image (t4). If the posture data at time t1 is used directly to compensate for the virtual image at time t4, the compensation of the high-frequency rotation matrix is ​​outdated, and the virtual image will still shake in the wearer's field of vision.

[0069] Therefore, in order to avoid the above problems, Figure 2 A flowchart illustrating a method for displaying a virtual image according to another embodiment of this application is shown, the method being performed by the aforementioned head-mounted display device. Figure 2 As shown, the method includes the following steps:

[0070] Step 201: Obtain the head posture data of the wearer wearing the head-mounted display device and the original projection matrix of the head-mounted display device.

[0071] This step can be referenced. Figure 1 Step 110 in the illustrated embodiment will not be repeated here.

[0072] Step 202: Obtain the latency of the head-mounted display device, which is the time interval between the generation of the posture data and the display of the virtual image based on the posture data.

[0073] Specifically, the time t when the attitude data is generated can be recorded. start And the time t for recording the display of the virtual image after inverse compensation rendering based on pose data. start By calculating the difference between two moments, the latency of the head-mounted display device can be determined.

[0074] For example, if the head-mounted display device has a heavy processing load, and it takes 20ms from the generation of posture data to the display of the virtual image on the screen, then the latency is 20ms.

[0075] Step 203: Determine the predicted attitude data for the current moment based on the attitude data from the previous moment.

[0076] Predicted posture data refers to the head posture calculated at the current moment based on historical posture data.

[0077] For example, if the wearer's head was turning to the right at an angular velocity of 10 degrees / second at the previous moment, and the head angle at the previous moment was 0°, after 10ms, the predicted head angle at the current moment may be 0.1°, that is, the predicted head posture data at the current moment is 0.1°.

[0078] Step 204: Correct the predicted attitude data based on the attitude data at the current moment to obtain the corrected attitude data.

[0079] Once the IMU of the head-mounted display device actually collects the wearer's current head posture data, it compares and merges the current posture data with the predicted posture data to correct the predicted posture data.

[0080] Specifically, various data fusion algorithms can be used to correct the predicted attitude data. For example, in one instance, the average of the current attitude data and the predicted attitude data can be used to determine the corrected attitude data. For instance, if the predicted attitude data indicates that the head has rotated 0.1° to the right, but the actual detected attitude data indicates that the head has rotated 0.12° to the right, then the corrected attitude data would be a head rotation of 0.11° to the right. Alternatively, the weighted average of the current attitude data and the predicted attitude data can be used to determine the corrected attitude data, with the weights dynamically adjusted based on the sensor's accuracy. Those skilled in the art can also use more complex state estimation algorithms, such as Kalman filters, to correct the predicted attitude data.

[0081] Step 205: Determine the attitude data for the delayed display time based on the corrected attitude data, with the delay time being the interval between the delayed display time and the current time.

[0082] The delayed display moment refers to the future moment when the virtual image is actually displayed on the screen, and the pose data of the delayed display moment refers to the predicted pose of the head at the future moment.

[0083] For example, in a simplified example, assuming the head maintains a constant speed for 40ms (i.e., the delay time) after the current time (t=40ms), and the angle of the head is determined to be 1.195° and the angular velocity is 0.0375° / ms based on the corrected attitude data, then the angle of the head corresponding to the attitude data at the delayed display time (t=80ms) can be predicted to be 1.195° + 0.0375° × 40 = 2.695°.

[0084] Step 206: Extract the high-frequency vibration signal at the delayed display time from the attitude data at the delayed display time.

[0085] Step 207: Determine the high-frequency rotation matrix based on the high-frequency vibration signal.

[0086] Step 208: Compensate the original projection matrix in reverse using the high-frequency rotation matrix of the delayed display time to obtain the compensation matrix for the delayed display time of the head-mounted display device.

[0087] Steps 206-208 can be referenced. Figure 1 Steps 120-140 in the illustrated embodiment will not be repeated here.

[0088] Step 209: Render and display the virtual image at the current moment based on the compensation matrix of the delayed display moment, so that the virtual image at the current moment is stably displayed in the wearer's field of vision.

[0089] Assuming a delay of 40ms and the current time is 40ms, when the head-mounted display detects attitude data at 40ms, it is currently using the attitude data detected at 0ms to render and display a virtual image. Therefore, to compensate for the high-frequency jitter at the current time, a compensation matrix for the delayed display time needs to be used to render and display the virtual image at the current time; for example, an 80ms compensation matrix can be used to render and display the virtual image at 40ms.

[0090] In this way, when a head-mounted display device uses a compensation matrix for delayed display times to render a virtual image of the current moment, although the compensation matrix is ​​calculated for a future moment, due to the delay time, when the virtual image is finally displayed on the screen, it happens to be at the delayed display time. The pre-set inverse compensation in the virtual image can then precisely offset the high-frequency jitter that actually occurs at that moment. For example, after rendering, the virtual image is displayed on the screen. At this time, the wearer's head happens to move to the predicted angle and produce the predicted jitter. Because the virtual image has undergone pre-processed inverse compensation, the virtual image is displayed stably in the wearer's field of vision.

[0091] Figure 3A schematic diagram of the structure of the virtual image display device provided in the embodiments of this application is shown, such as... Figure 3 As shown, the virtual image display device 600 includes: an acquisition module 610, an extraction module 620, a determination module 630, a reverse compensation module 640, and a rendering and display module 650. The acquisition module 610 acquires the head posture data of the wearer wearing the head-mounted display device and the original projection matrix of the head-mounted display device; the extraction module 620 extracts high-frequency vibration signals from the posture data; the determination module 630 determines a high-frequency rotation matrix based on the high-frequency vibration signals; the reverse compensation module 640 reverse-compensates the original projection matrix based on the high-frequency rotation matrix to obtain a compensation matrix for the head-mounted display device; and the rendering and display module 650 renders and displays the virtual image based on the compensation matrix to ensure stable display of the virtual image in the wearer's field of vision.

[0092] The specific implementation process and beneficial effects of each module in the virtual image display device 600 of the above embodiments to achieve the above functions can be referred to the foregoing. Figure 1 The embodiment of the virtual image display method shown will not be described in detail here.

[0093] The virtual image display device 600 of this application embodiment also includes various modules for performing the steps of the above-described virtual image display method embodiment, which will not be described in detail here.

[0094] Figure 4 The diagram shows a schematic of the structure of a head-mounted display device provided in an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the head-mounted display device.

[0095] like Figure 4 As shown, the head-mounted display device 700 may include a processor 702 and a memory 704.

[0096] The memory 704 is used to store the computer program 706. The memory 704 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive. The computer program 706 may include computer-executable instructions.

[0097] The processor 702 is used to execute the computer program 706 to implement the above-described virtual image display method embodiment.

[0098] The processor 702 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The head-mounted display device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0099] This application provides a chip suitable for head-mounted display devices. The chip stores a computer program, and the chip executes the computer program to implement the above-described virtual image display method.

[0100] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described virtual image display method embodiment.

[0101] This application provides a computer program that can be executed by a processor to implement the above-described virtual image display method embodiment.

[0102] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described virtual image display method embodiment.

[0103] In the several embodiments provided in this application, any function, if implemented as a software functional module / unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, part or all of the technical solutions of this application can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or other electronic device) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of this application are not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0105] It should be noted that the above embodiments are illustrative of this application and not restrictive, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In claims enumerating several means, several units or modules of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

[0106] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A display method of a virtual image applied to a head-mounted display device, characterized by, The method includes: Acquire the head posture data of the wearer wearing the head-mounted display device and the original projection matrix of the head-mounted display device; High-frequency vibration signals are extracted from the attitude data; Determine the high-frequency rotation matrix based on the high-frequency vibration signal; The compensation matrix of the head-mounted display device is obtained by inversely compensating the original projection matrix using the high-frequency rotation matrix. The virtual image is rendered and displayed according to the compensation matrix to ensure that the virtual image is stably displayed in the wearer's field of vision; The posture data includes the rotation angle and the frequency at which the wearer's head generates the rotation angle; The step of extracting high-frequency vibration signals from the attitude data further includes: Determine whether the rotation angle is greater than a first preset angle and less than a second preset angle, wherein the first preset angle is less than the second preset angle, and the absolute values ​​of the first preset angle and the second preset angle are equal; If the rotation angle is greater than the first preset angle and less than the second preset angle, then determine whether the frequency is greater than the preset frequency threshold. If the frequency is greater than a preset frequency threshold, then the rotation angle is determined as the high-frequency vibration signal; The method further includes: Obtain the viewing angle matrix of the head-mounted display device; The step of rendering and displaying the virtual image according to the compensation matrix further includes: Multiplying the compensation matrix by the viewpoint matrix yields the target matrix of the head-mounted display device. The virtual image is rendered and displayed based on the target matrix.

2. The method of claim 1, wherein, The step of obtaining the compensation matrix of the head-mounted display device by inversely compensating the original projection matrix based on the high-frequency rotation matrix further includes: Determine the inverse matrix of the high-frequency rotation matrix; The compensation matrix of the head-mounted display device is determined based on the inverse matrix of the high-frequency rotation matrix and the original projection matrix.

3. The method of claim 2, wherein, Determining the compensation matrix of the head-mounted display device based on the inverse matrix of the high-frequency rotation matrix and the original projection matrix includes: According to the formula Calculate the compensation matrix ,in, Represents the original projection matrix. This represents the inverse matrix of the high-frequency rotation matrix.

4. The method of claim 1, wherein, The rotation angles include single-axis rotation angles about the X-axis, Y-axis, and Z-axis; The step of determining the high-frequency rotation matrix based on the high-frequency vibration signal further includes: The rotation matrices around the X-axis, Y-axis, and Z-axis are determined based on the single-axis rotation angles around the X-axis, Y-axis, and Z-axis, respectively. The high-frequency rotation matrix is ​​obtained by multiplying the rotation matrices around the X-axis, Y-axis, and Z-axis in sequence.

5. The method of claim 1, wherein, The step of extracting high-frequency vibration signals from the attitude data further includes: The delay time of the head-mounted display device is obtained, wherein the delay time is the time interval between the generation of the posture data and the display of a virtual image based on the posture data; The predicted attitude data for the current moment is determined based on the attitude data from the previous moment. The predicted attitude data is corrected based on the attitude data at the current moment to obtain the corrected attitude data; The attitude data for the delayed display time is determined based on the corrected attitude data, wherein the delayed display time is spaced from the current time by the delay time. Extract the high-frequency vibration signal at the delayed display time from the attitude data at the delayed display time; The step of inversely compensating the original projection matrix according to the high-frequency rotation matrix to obtain the compensation matrix of the head-mounted display device, and rendering and displaying a virtual image according to the compensation matrix to make the virtual image stably displayed in the wearer's field of vision, further includes: The original projection matrix is ​​inversely compensated based on the high-frequency rotation matrix of the delayed display time to obtain the compensation matrix of the delayed display time of the head-mounted display device; The virtual image at the current moment is rendered and displayed according to the compensation matrix of the delayed display moment, so that the virtual image at the current moment is stably displayed in the wearer's field of vision.

6. A head-mounted display device comprising a memory, a processor, and a computer program stored on the memory, wherein, The processor executes the computer program to implement the virtual image display method according to any one of claims 1 to 5.

7. A chip applied to a head-mounted display device, characterized by, The chip stores a computer program, and the chip executes the computer program to implement the virtual image display method according to any one of claims 1 to 5.

8. A computer readable storage medium having stored thereon a computer program, characterized in that, When executed by a processor, the computer program implements the method for displaying virtual images as described in any one of claims 1 to 5.

Images