Multi-layer display method and head-mounted display device

Abstract

The invention relates to the technical field of three-dimensional display, and provides a multi-layer display method and head-mounted display equipment. In order to solve the problem that the range of a display screen of the head-mounted display equipment is limited, a first layer is used for displaying an application picture, a second layer is used for displaying prompt information, and the second layer is always located in front of the first layer, so that the prompt information does not occupy the position of the first layer, the display area of the application picture is larger, the immersion of a user is stronger, and the user experience is improved. According to the technical scheme, multiple display modes are set according to the situation of the two layers, the positions of the first layer and the second layer in different display modes are different in control mode when the head moves, flexible switching can be achieved according to user requirements, and therefore the method is suitable for multiple use scenes and higher in expansibility. Wherein the transparency of the second image layer is specially processed, so that the interference of the second image layer on the first image layer is reduced while the user can timely see the prompt information, and the normal watching of the user is ensured.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional display technology and provides a multi-layer display method and a head-mounted display device. Background Art

[0002] Head-mounted display devices (such as AR glasses, VR glasses, MR glasses, etc.) are an emerging wearable device that can provide a 3D immersive experience through computer technology and have been widely used in various fields such as gaming, navigation, and education.

[0003] Currently, the display solutions for head-mounted display devices mainly include the following:

[0004] 1. Optical waveguide: transmits light through total internal reflection inside the lens, projecting the image of a micro display (such as LCoS / Micro-OLED) into the human eye;

[0005] 2. Free-form surface: The display screen content is projected to the human eye by reflecting light through an asymmetric curved prism;

[0006] 3. BirdBath: Light is reflected by a beam splitter onto a curved mirror, and then back to the human eye, creating a superposition of virtual and real life.

[0007] 4. Off-axis reflection: The display screen is placed on the side of the lens, and the light enters the human eye through the off-axis parabolic reflector.

[0008] However, the common problems with the above display solutions are: the field of view of the head-mounted display device is small, and long-term use can easily cause visual fatigue. At the same time, due to the small size of the display screen, the display screen is simple, the user's immersion is poor, and the usage scenarios are relatively single and the scalability is poor.

[0009] Therefore, improving the viewing comfort and usage scenarios of head-mounted display devices has become an urgent problem to be solved in the field of three-dimensional display technology. Summary of the Invention

[0010] The embodiments of the present application provide a multi-layer display method and a head-mounted display device, which are used to improve the viewing comfort of the head-mounted display device and expand the usage scenarios.

[0011] In a first aspect, an embodiment of the present application provides a multi-layer display method, comprising:

[0012] After the device is started, a currently selected target mode is determined from a plurality of preset display modes; wherein each display mode includes a first layer and a second layer, wherein the first layer is used to display an application screen, and the second layer is used to display prompt information, and the second layer is always located in front of the first layer, and the transparency of the second layer is fully transparent so that the second layer is not visible;

[0013] When the target mode is the static mode, the posture states of the first layer and the second layer are set to 0 degrees of freedom, so that the first layer and the second layer move with the movement of the user's head and are located in the center of the user's field of view; the static mode corresponds to scenes where the amount of movement does not meet the preset conditions;

[0014] When the target mode is motion mode, the posture state of the first layer is set to 3 degrees of freedom and the posture state of the second layer is set to 0 degrees of freedom, so that the second layer moves with the movement of the user's head and is located at the center of the user's field of view, while the first layer does not move with the movement of the user's head; the motion mode corresponds to a scene in which the amount of motion meets a preset condition;

[0015] Upon receiving the prompt information, the transparency of the second layer is set to opaque so that the prompt information is visible.

[0016] The beneficial effects of the above technical solution are as follows: to address the problem of limited display screen range of the head-mounted display device, two layers are used, the first layer is used as the main layer, for displaying the application screen, and the second layer is used as the auxiliary layer, for displaying prompt information of the system or application. The second layer is located in front of the first layer and is fully transparent, that is, the second layer is closer to the human eye and is not seen by the user. In this way, the prompt information will not occupy the position of the first layer where the application screen is located, and the display area of ​​the application screen is larger, thereby enhancing the user's immersion. Moreover, in the case of two layers, the second layer is invisible, so that when the prompt information is not received, the display of the application screen is not disturbed, ensuring the user's normal viewing.

[0017] On the other hand, in order to adapt to different usage scenarios, the method can set multiple display modes. The control methods of the positions of the first layer and the second layer are different in different display modes. In this way, the user can choose between still mode and motion mode according to the current body state. The still mode corresponds to the scene where the amount of movement does not meet the preset conditions, such as the sitting or lying scene with small amplitude and frequency of head movement. Therefore, the posture state of the first layer and the second layer is set to 0 degrees of freedom. In this way, the first layer and the second layer do not have motion independence in the Unity coordinate system, and thus move with the movement of the user's head and are located in the center of the user's field of view. Since the still mode corresponds to the scene where the amount of movement does not meet the preset conditions, the position of the first layer will not change at a high frequency, so the user will not feel dizzy when watching the application screen, and the second layer will not change at a high frequency. The position change of the layer can ensure that the user sees the prompt information in time; the motion mode corresponds to the scene where the amount of movement meets the preset conditions, such as walking or running scenes with large head movement amplitude and frequency. If the position of the application screen changes frequently, it will cause dizziness. Therefore, the posture state of the first layer is set to 3 degrees of freedom and the posture state of the second layer is set to 0 degrees of freedom. In this way, the first layer can move independently in the Unity coordinate system and have its own posture, while the second layer has no motion independence in the Unity coordinate system. Therefore, the first layer does not move with the movement of the user's head. The position of the application screen seen by the user during exercise is fixed, thereby reducing the dizziness caused by the screen shaking. The second layer moves with the movement of the user's head and is located in the center of the user's field of view, ensuring that the user can see the prompt information in time. These two display modes can be flexibly switched between according to user needs, thus enabling a variety of usage scenarios and greater scalability.

[0018] Optionally, in the sports mode, the method further includes:

[0019] When a preset calibration condition is triggered, the position of the first layer is calibrated according to the rotation data of the user's head so that the first layer is located at the center of the user's field of view.

[0020] The beneficial effect of the above technical solution is: the position of the first layer is calibrated by the rotation data of the user's head, so that the user's viewing comfort can still be ensured when the user's head moves.

[0021] Optionally, the rotation data is represented in quaternion form, and calibrating the position of the first layer according to the rotation data of the user's head includes:

[0022] Calculating the current orientation of the user's head based on the quaternion of the current position of the user's head, with the positive direction of the Z axis representing the depth direction as a reference;

[0023] Calculate the deflection angles of the first layer on the Y-axis and the X-axis according to the current orientation, and convert the two rotation angles into a quaternion matrix;

[0024] The position of the first layer is determined according to the two quaternion matrices.

[0025] The beneficial effects of the above technical solution are: since quaternions perform well in numerical stability, the position of the first layer is calibrated by quaternions, thereby reducing precision loss and calculation errors, thereby improving calibration accuracy and further improving user viewing comfort.

[0026] Optionally, the second layer includes a plurality of cells, and setting the transparency of the second layer to be opaque so that the prompt information is visible includes:

[0027] According to the gaze point position of the binocular eyes on the second layer, the prompt information is rendered on the second layer, and the transparency of the target cell occupied by the prompt information on the second layer is set to opaque so that the target cell is visible and the transparency of other cells remains fully transparent.

[0028] The beneficial effects of the above technical solution are: after receiving the prompt information, the prompt information is rendered according to the position of the binocular gaze point, and by setting the local transparency of the second layer to fully transparent, while ensuring that the user can see the prompt information in time, the interference with the display screen of the first layer is reduced.

[0029] Optionally, after setting the transparency of the target cell occupied by the prompt information on the second layer to be opaque, the method further includes:

[0030] When the display time of the prompt information is greater than a first time threshold, setting the transparency of the target cell on the second layer to be fully transparent so that the prompt information is invisible;

[0031] When the display time of the prompt information is less than or equal to the first time threshold, the change in the gaze point position is greater than the preset range, and the stay time after the gaze point position changes is greater than the second time threshold, the transparency of the target cell on the second layer is set to fully transparent to make the prompt information invisible.

[0032] The beneficial effect of the above technical solution is: by setting the time threshold and the change of the gaze point position, the local transparency of the second layer is adjusted, thereby avoiding the interruption of the user by the prompt information being displayed for a long time and improving the viewing quality.

[0033] Optionally, the preset condition is that the rotation angle of the user's head is greater than a preset angle threshold, and the dwell time after rotation is greater than a third time threshold.

[0034] The beneficial effect of the above technical solution is: by setting a preset angle threshold and time threshold, the user's viewing comfort is ensured while avoiding the computing resources occupied by frequent position calibration of the first layer.

[0035] In a second aspect, an embodiment of the present application provides a head-mounted display device, comprising a processor, a memory, and a display screen, wherein the display screen, the memory, and the processor are connected via a bus;

[0036] The display screen is used to display prompt information and application screens;

[0037] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0038] After the device is started, a currently selected target mode is determined from a plurality of preset display modes; wherein each display mode includes a first layer and a second layer, wherein the first layer is used to display an application screen, and the second layer is used to display prompt information, and the second layer is always located in front of the first layer, and the transparency of the second layer is fully transparent so that the second layer is not visible;

[0039] When the target mode is the static mode, the posture states of the first layer and the second layer are set to 0 degrees of freedom, so that the first layer and the second layer move with the movement of the user's head and are located in the center of the user's field of view; the static mode corresponds to scenes where the amount of movement does not meet the preset conditions;

[0040] When the target mode is motion mode, the posture state of the first layer is set to 3 degrees of freedom and the posture state of the second layer is set to 0 degrees of freedom, so that the second layer moves with the movement of the user's head and is located at the center of the user's field of view, while the first layer does not move with the movement of the user's head; the motion mode corresponds to a scene in which the amount of movement meets a preset condition;

[0041] Upon receiving the prompt information, the transparency of the second layer is set to opaque so that the prompt information is visible.

[0042] Optionally, in the motion mode, the processor further executes:

[0043] When a preset calibration condition is triggered, the position of the first layer is calibrated according to the rotation data of the user's head so that the first layer is located at the center of the user's field of view.

[0044] Optionally, the rotation data is represented in quaternion form, and the processor calibrates the position of the first layer according to the rotation data of the user's head, specifically by:

[0045] Calculating the current orientation of the user's head based on the quaternion of the current position of the user's head, with the positive direction of the Z axis representing the depth direction as a reference;

[0046] Calculate the deflection angles of the first layer on the Y-axis and the X-axis according to the current orientation, and convert the two rotation angles into a quaternion matrix;

[0047] The position of the first layer is determined according to the two quaternion matrices.

[0048] Optionally, the second layer includes a plurality of cells, and the processor sets the transparency of the second layer to opaque so that the prompt information is visible, specifically by:

[0049] According to the gaze point position of the binocular eyes on the second layer, the prompt information is rendered on the second layer, and the transparency of the target cell occupied by the prompt information on the second layer is set to opaque so that the target cell is visible and the transparency of other cells remains fully transparent.

[0050] Optionally, after the processor sets the transparency of the target cell occupied by the prompt information on the second layer to be opaque, it further executes:

[0051] When the display time of the prompt information is greater than a first time threshold, setting the transparency of the target cell on the second layer to be fully transparent so that the prompt information is invisible;

[0052] When the display time of the prompt information is less than or equal to the first time threshold, the change in the gaze point position is greater than the preset range, and the stay time after the gaze point position changes is greater than the second time threshold, the transparency of the target cell on the second layer is set to fully transparent to make the prompt information invisible.

[0053] Optionally, the preset condition is that the rotation angle of the user's head is greater than a preset angle threshold, and the dwell time after rotation is greater than a third time threshold.

[0054] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned multi-layer display methods are implemented.

[0055] The technical effects brought about by any implementation method in the second to third aspects can refer to the technical effects brought about by the corresponding implementation method in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0057] Figure 1 A schematic diagram of a multi-layer display method according to an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of the positional relationship between the first layer and the second layer;

[0059] Figure 3 This is a top view of the position changes of the two layers in static mode;

[0060] Figure 4 This is a top view of the position changes of the two layers in motion mode;

[0061] Figure 5 This is a schematic diagram of the transparency change process of the second layer;

[0062] Figure 6A This is a schematic diagram of the display effect of the prompt information on the second layer;

[0063] Figure 6B Schematic diagram of the process of position calibration for the first layer;

[0064] Figure 7 This is a schematic diagram of the coordinate system of the two layers;

[0065] Figure 8A and Figure 8B This is a schematic diagram of the position calibration process for the first layer;

[0066] Figure 9 A schematic diagram of a multi-layer display method according to an embodiment of the present invention;

[0067] Figure 10 This is a structural diagram of the head-mounted display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of the technical solutions of this application, but not all of them. Based on the embodiments described in this application document, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the technical solutions of this application.

[0069] Based on the exemplary embodiments shown in this application, all other embodiments obtained by persons of ordinary skill in the art without inventive effort are within the scope of protection of this application. In addition, although the disclosure in this application is presented based on one or several exemplary examples, it should be understood that each aspect of the disclosure can independently constitute a complete technical solution.

[0070] In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0071] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0072] An embodiment of the present application provides a multi-layer display method. To address the problem of limited display screen range of a head-mounted display device, two layers are used. The first layer serves as a main layer for displaying an application screen, and the second layer serves as an auxiliary layer for displaying prompt information of the system or application. The second layer is located in front of the first layer and is fully transparent, that is, the second layer is closer to the human eye and is not seen by the user. In this way, the prompt information will not occupy the position of the first layer where the application screen is located, and the display area of ​​the application screen is larger, thereby enhancing the user's immersion. Moreover, in the case of two layers, the second layer is invisible, so that when the prompt information is not received, the display of the application screen is not disturbed, thereby ensuring the user's normal viewing.

[0073] On the other hand, in order to adapt to different usage scenarios, the method can set multiple display modes. The control methods of the positions of the first layer and the second layer are different in different display modes. In this way, the user can choose between still mode and motion mode according to the current body state. The still mode corresponds to the scene where the amount of movement does not meet the preset conditions, such as the sitting or lying scene with small amplitude and frequency of head movement. Therefore, the posture state of the first layer and the second layer is set to 0 degrees of freedom. In this way, the first layer and the second layer do not have motion independence in the Unity coordinate system, and thus move with the movement of the user's head and are located in the center of the user's field of view. Since the still mode corresponds to the scene where the amount of movement does not meet the preset conditions, the position of the first layer will not change at a high frequency, so the user will not feel dizzy when watching the application screen, and the second layer will not change at a high frequency. The position change of the layer can ensure that the user sees the prompt information in time; the motion mode corresponds to the scene where the amount of movement meets the preset conditions, such as walking or running scenes with large head movement amplitude and frequency. If the position of the application screen changes frequently, it will cause dizziness. Therefore, the posture state of the first layer is set to 3 degrees of freedom and the posture state of the second layer is set to 0 degrees of freedom. In this way, the first layer can move independently in the Unity coordinate system and have its own posture, while the second layer has no motion independence in the Unity coordinate system. Therefore, the first layer does not move with the movement of the user's head. The position of the application screen seen by the user during exercise is fixed, thereby reducing the dizziness caused by the screen shaking. The second layer moves with the movement of the user's head and is located in the center of the user's field of view, ensuring that the user can see the prompt information in time. These two display modes can be flexibly switched between according to user needs, thus enabling a variety of usage scenarios and greater scalability.

[0074] See also Figure 1 , is a flow chart of a multi-layer display method provided in an embodiment of the present application. The process is executed by a head-mounted display device and mainly includes the following steps:

[0075] S101: After the device is started, a currently selected target mode is determined from a plurality of preset display modes.

[0076] In the case where multiple display modes are set, the head-mounted display device can automatically select a currently used display mode after startup, that is, the head-mounted display device is set with a default display mode.

[0077] In one embodiment, after the head-mounted display device is started, the user can select a target mode currently in use from multiple display modes through the configuration interface according to his or her physical state (such as sitting state, exercise state, etc.). When the user does not select a display mode, the head-mounted display device automatically uses the default display mode.

[0078] Among them, the multiple display modes include at least a still mode and a motion mode. The still mode corresponds to a situation where the amount of exercise does not meet the preset conditions, such as a sitting or lying scene with a small amplitude and frequency of head movement. The motion mode corresponds to a scene where the amount of exercise meets the preset conditions, such as a walking or running scene with a large amplitude and frequency of head movement.

[0079] In one embodiment, each display mode includes a first layer and a second layer, wherein the first layer serves as the main layer for displaying application screens, including but not limited to text, pictures, and videos, and the second layer serves as the auxiliary layer for displaying prompt information, which may be system prompt information, such as a battery icon, a WiFi signal icon, a time icon, a Bluetooth icon, etc., or application prompt information, such as WeChat message reminders, email message reminders, etc. The second layer is always in front of the first layer, that is, the second layer is closer to the human eye, and the transparency of the second layer is initially set to fully transparent, thereby reducing interference with the first layer.

[0080] like Figure 2 , which is a schematic diagram of the positional relationship between the first layer and the second layer, wherein the solid line represents the first layer and the dotted line represents the second layer.

[0081] S102: Determine the type of display mode. If it is a still mode, execute S103; if it is a sports mode, execute S104.

[0082] When there are multiple display modes, the head-mounted display device needs to determine the display mode currently in use to reasonably control the first layer and the second layer.

[0083] S103: Setting the posture states of the first layer and the second layer to 0 degrees of freedom, so that the first layer and the second layer move with the movement of the user's head and are located in the center of the user's field of view.

[0084] In static mode, the posture state of the first layer and the second layer is set to 0 degrees of freedom. At this time, the device or system does not track the movement of the user's head at all, and the user cannot change the perspective and position in the virtual environment by moving the head. In other words, the first layer and the second layer cannot be moved and rotated independently in the Unity coordinate system, thereby achieving the effect of binding the first layer and the second layer to the display screen respectively. In this way, when the head movement drives the head-mounted display device (i.e., the display screen) to move, the first layer and the second layer can be controlled to move with the movement of the user's head and always remain in the center of the user's field of view.

[0085] like Figure 3As shown, this is a top view of the position changes of the first layer and the second layer in static mode. When the user's head moves to the left, the first layer and the second layer both move to the left and remain in the center of the user's field of view. When the user's head moves to the right, the first layer and the second layer both move to the right and remain in the center of the user's field of view.

[0086] In Still Mode, by setting the first and second layers to zero degrees of freedom, they are controlled to move with the user's head movements and remain centered in the user's field of view, achieving the effect of fixing the first and second layers to the display. Furthermore, since Still Mode typically corresponds to scenes with less frequent and ample head movements, the position of the first layer doesn't fluctuate frequently, minimizing jitter as the image follows the line of sight. This prevents motion sickness and improves viewing comfort. Furthermore, the changing position of the second layer ensures that prompts are visible to the user.

[0087] S104: Setting the posture state of the first layer to 3 degrees of freedom and the posture state of the second layer to 0 degrees of freedom, so that the second layer moves with the movement of the user's head and is located at the center of the user's field of view, while the first layer does not move with the movement of the user's head.

[0088] In motion mode, by setting the posture state of the second layer to 0 degrees of freedom, the first and second layers are constrained in the Unity coordinate system and cannot move and rotate independently, so as to achieve the effect of binding the second layer to the display screen. In this way, when the head movement drives the head-mounted display device (i.e., the display screen) to move, the second layer can be controlled to move with the movement of the user's head and always remain in the center of the user's field of view, thereby ensuring that the user sees the prompt information in time. At the same time, by setting the position state of the first layer to 3 degrees of freedom, the first layer has the characteristic of independent movement in the Unity coordinate system, which can keep the posture of the first layer itself unchanged, thereby avoiding jitter of the application screen caused by head movement.

[0089] like Figure 4 As shown, this is a top view of the position changes of the first and second layers in motion mode. When the user's head moves to the left, the first layer remains in position, and the second layer moves to the left and remains in the center of the user's field of view. When the user's head moves to the right, the first layer remains in position, and the second layer moves to the right and remains in the center of the user's field of view.

[0090] S105: After receiving the prompt information, the transparency of the second layer is set to opaque to make the prompt information visible.

[0091] When the head-mounted display device displays the application screen through the first layer, before receiving the prompt information, the transparency of the second layer is set to fully transparent to make it invisible to the user. The user will only see the application screen displayed on the first layer, which will not interfere with the display of the first layer. When the prompt information is received, in order to enable the user to see the second prompt information, the transparency of the second layer is set to transparent, and after ensuring that the user sees the prompt information, the transparency of the second layer is restored to fully transparent.

[0092] See also Figure 5 , which is a flowchart for changing the transparency of the second layer, mainly includes the following steps:

[0093] S1051: Check whether the prompt information is received, if not, execute S1052, if so, execute S1053.

[0094] The prompt information affects the transparency of the second layer, so it is necessary to detect whether the prompt information is received.

[0095] S1052: Keep the transparency of the second layer fully transparent so that the second layer is invisible.

[0096] Before receiving the prompt information, the transparency of the second layer is kept fully transparent, so that the second layer is invisible and avoids blocking the application screen displayed by the first layer.

[0097] S1053: Render prompt information on the second layer according to the gaze point position of the binocular eyes on the second layer.

[0098] In one embodiment, the second layer may be divided into M cells (M is an integer greater than 1) according to the resolution, and each cell contains H*L pixels.

[0099] For example, if the resolution is 1920*1280 (pixels), the second layer is divided into 128 cells of 15*10 pixels each.

[0100] To reduce the interference of prompt information display on the application screen, the second layer can be set to be partially visible in the form of cells. In specific implementation, the head-mounted display device uses the deployed eye tracking algorithm to detect the gaze point of both eyes on the second layer in real time and render the prompt information on the second layer based on the gaze point.

[0101] S1054: The transparency of the target cell occupied by the prompt information on the second layer is set to be opaque, so that the target cell is visible, and the transparency of other cells remains fully transparent.

[0102] After the prompt information is rendered, the transparency of the target cell occupied by the prompt information on the second layer is quickly reduced, making the transparency of the target cell opaque, and keeping the transparency of other cells fully transparent, so that the prompt information on each target cell can be clearly seen. At the same time, other cells will not block the application screen displayed on the first layer.

[0103] In one embodiment, in order to avoid the long-term display of the prompt information interfering with the application screen, the prompt information can be hidden. The details are as follows:

[0104] S1055: Check whether the display time of the prompt information is greater than the first time threshold. If so, execute S1056; otherwise, execute S1057.

[0105] Optionally, the first time threshold is 15 seconds.

[0106] It should be noted that the size of the first time threshold can be set according to actual needs, and the embodiment of the present application does not impose any restrictive requirements.

[0107] S1056: Set the transparency of the target cell on the second layer to be fully transparent so that the prompt information is invisible.

[0108] When the display time of the prompt information is greater than the first time threshold, it indicates that the user has enough time to get the prompt information. Therefore, the transparency of the target cell can be set to fully transparent to make the prompt information invisible, thereby reducing interference with the application screen.

[0109] S1057: Determine whether the change in the gaze point position is greater than a preset range, and whether the dwell time after the gaze point position change is greater than a third time threshold. If so, execute S1058; otherwise, execute S1059.

[0110] The change in gaze position and dwell time can reflect whether the user has seen the content of the prompt information.

[0111] S1058: Set the transparency of the target cell on the second layer to be fully transparent so that the prompt information is invisible.

[0112] When the change in the gaze point position is greater than a preset range (such as: the X-axis coordinate change is greater than 30 pixels, the Y-axis change is greater than 20 pixels, or the X-axis coordinate change is greater than 30 pixels and the Y-axis change is greater than 20 pixels), and the dwell time after the gaze point position changes is greater than a second time threshold (such as 5 seconds), it indicates that the user has seen the content of the prompt information. When the prompt information is not important, the user may not process it but continue to watch the application screen. At this time, there is no need to display the prompt information. Therefore, the transparency of the target cell can be quickly changed to fully transparent to make the prompt information invisible, thereby reducing the interference of the prompt information on the application screen.

[0113] It should be noted that the preset range size of the gaze point position change and the third time threshold can be set according to actual needs, and the embodiment of the present application does not impose any restrictive requirements.

[0114] S1059: Maintain the transparency of the target cell on the second layer as opaque so that the prompt information is visible.

[0115] When at least one of the following conditions is met: the display time of the prompt information is less than or equal to the first time threshold, the change in the gaze point position is less than or equal to the preset range, and the dwell time after the gaze point position changes is less than or equal to the second time threshold, the transparency of the target cell occupied by the prompt information is kept opaque so that the user can see the prompt information.

[0116] like Figure 6A The figure below shows the display effect of the prompt information on the second layer. S(x, y) represents the binocular gaze point on the second layer. The prompt information is rendered starting from S(x, y). The prompt information "Hello" occupies four cells. The transparency of these four cells is set to opaque, that is, the transparency value is 0. The other cells are set to fully transparent, that is, the transparency value is 1, and are filled with gray. When the prompt information is displayed, the transparency value of these four cells quickly returns to 1.

[0117] In the case of multiple layers, before receiving the prompt information, the transparency of the second layer is set to be fully transparent, thereby reducing the interference of the second layer on the first layer and ensuring the normal viewing of the application screen on the first layer. After receiving the prompt information, by setting the local transparency of the second layer to be fully transparent, while ensuring that the user sees the prompt information in time, the interference with the display screen of the first layer is reduced. In addition, the local transparency of the second layer is adjusted by setting the time threshold and the change of the gaze point position, thereby avoiding disturbing the user due to the long-term display of the prompt information and improving the viewing quality.

[0118] In one embodiment, in motion mode, the range of head movement varies significantly. If the position of the first layer remains unchanged, it may result in unresolved viewing. Therefore, the position of the first layer can be controlled by setting a calibration condition that triggers the change of the first layer's position. The calibration condition is that the rotation angle of the user's head is greater than a preset angle threshold, and the dwell time after rotation is greater than a third time threshold. Thus, when the calibration condition is triggered, the position of the first layer can be calibrated based on the user's head rotation data so that the first layer is located in the center of the user's field of view. This ensures viewing comfort even when the user's head moves, while avoiding image jitter caused by frequent position calibration of the first layer.

[0119] It should be noted that the preset angle threshold and the third time threshold can be set according to the actual scenario, and the embodiment of the present application does not impose any restrictive requirements. Among them, the third time threshold should be greater than or equal to the time required for the position calibration algorithm, such as being set to 5 seconds.

[0120] In one embodiment, the rotation data can be represented in the form of quaternions. The position calibration process of the first layer is described in Figure 6B , mainly includes the following steps:

[0121] S106: Calculate the current orientation of the user's head based on the quaternion of the current position of the user's head, taking the positive direction of the Z axis representing the depth direction as a reference.

[0122] For example, the image rendering of the first and second layers follows the left-hand coordinate system. Figure 7 As shown, the Z axis represents the depth direction. Taking the positive direction of the Z axis as the reference, the formula for the direction corresponding to the current position after the head is deflected is as follows:

[0123] LookDir=Forward*CurrentQuat Formula 1

[0124] Among them, Forward is the reference direction vector (0.0f, 0.0f, 1.0f) in the positive direction of the Z axis, CurrentQuat = (x1, y1, z1, w1) represents the quaternion of the current position, and LookDir = (x2, y2, z2, w2) represents the current direction corresponding to the current position.

[0125] S107: Calculate the deflection angles of the first layer on the Y-axis and the X-axis according to the current orientation, and convert the two rotation angles into a quaternion matrix.

[0126] First, calibrate the position of the first layer on the Y axis. Assume that the deflection angle on the Y axis after the head is deflected is RotateRadsY, that is, the yaw angle. Use the trigonometric function atan2 to calculate it as follows:

[0127] RotateRadsY=atan2(x2,z2) Formula 2

[0128] Then, rotate the position of the first layer in the X-axis direction. Assuming that the deflection angle of the head in the X-axis is RotateRadsX, that is, the pitch angle, use the inverse trigonometric function asin to calculate it as follows:

[0129] RotateRadsX =asin(y2) Formula 3

[0130] Furthermore, a quaternion matrix RecenterRotY for reverse rotation around the Y axis and a quaternion matrix RecenterZRotX for reverse rotation around the X axis are generated.

[0131] Specifically, the existing OpenGL function glm::rotate can be used to calculate the quaternion matrix. The formula is as follows:

[0132] RecenterRotY=glm::rotate((0.0f,0.0f,0.0f,1.0f),RotateRadsY,(0.0f,1.0f,0.0f)) Formula 4

[0133] RecenterRotX=glm::rotate((0.0f,0.0f,0.0f,1.0f),RotateRadsX,(1.0f,1.0f,0.0f)) Formula 5

[0134] Here, f represents a floating-point number type.

[0135] S108: Determine the position of the first layer according to the two quaternion matrices.

[0136] Finally, based on the two quaternion matrices, the formula for calculating the position of the first layer is:

[0137] RecenterRot=RecenterRotY*RecenterRotX Formula 6

[0138] like Figure 8A The figure shows a schematic diagram of the process of calibrating the head deflection and the position of the first layer in motion mode. Assume that the user's head deflects to the right, starting from point A and stopping at point B. The user's head stays at point B for 5 seconds, and the deflection angle from point A to point B exceeds the preset angle threshold α, which triggers the position calibration of the first layer. After calibration, the first layer appears at point B.

[0139] like Figure 8BFigure 2 shows a schematic diagram of the process of calibrating the head deflection and the first layer position in another motion mode. Assume that the user's head rotates from point A to point B, but the calibration conditions are not met at point B, such as the deflection angle is less than α or the dwell time at point B is less than 5 seconds. Therefore, the first layer does not appear at point B. When the user's head briefly stops at point B and then rotates to point C, the rotation angle from point B to point C is greater than α, and the dwell time at point C is greater than 5 seconds, which triggers the calibration of the first layer position. After calibration, the first layer appears at point C.

[0140] After the position of the first layer is calibrated, it can still be reproduced directly in front of the user's field of view when the head deflection meets the calibration conditions. In addition, since quaternions perform well in terms of numerical stability, quaternions are used to calibrate the position of the first layer, thereby reducing precision loss and calculation errors, thereby improving calibration accuracy and further enhancing user viewing comfort.

[0141] Take AR glasses as an example, which use the static mode by default. Figure 9 The multi-layer display method provided in the embodiment of the present application mainly includes the following steps:

[0142] S1: Detect whether the user switches the display mode. If so, execute S2; otherwise, execute S3.

[0143] S2: Start motion mode, set the pose state of the first layer to 3 degrees of freedom, and the pose state of the second layer to 0 degrees of freedom.

[0144] The second layer includes multiple cells, is located in front of the first layer, and has full transparency.

[0145] S3: Start the static mode and set the pose state of the first and second layers to 0 degrees of freedom.

[0146] The second layer includes multiple cells, is located in front of the first layer, and has full transparency.

[0147] S4: Rendering the application screen on the first layer.

[0148] S5: Check whether the prompt information is received. If so, execute S6, otherwise end.

[0149] S6: Use an eye tracking algorithm to determine the gaze point position of the two eyes on the second layer.

[0150] S7: Rendering prompt information on the second layer based on the gaze point position.

[0151] S8: Set the transparency of the target cells occupied by the prompt information on the second layer to be opaque.

[0152] S9: When the user's head movement is detected, determine whether it is in the static mode. If so, execute S10, otherwise S11.

[0153] S10: The first layer and the second layer move as the user's head moves and remain in the center of the user's field of view.

[0154] S11: Determine whether the calibration conditions are met, if not, execute S12, if so, execute S13.

[0155] S12: The position of the first layer remains unchanged, and the second layer moves with the user's head movement and remains in the center of the user's field of view.

[0156] S13: The position of the first layer is calibrated according to the rotation data of the user's head, and remains unchanged after calibration. At the same time, the second layer moves with the movement of the user's head and remains in the center of the user's field of view.

[0157] S14: Count whether the display time of the prompt information is greater than the first time threshold. If so, execute S15; otherwise, execute S16.

[0158] S15: Set the transparency of the target cells on the second layer to fully transparent.

[0159] S16: Detect whether the change in the gaze point position is greater than a preset range, and whether the dwell time after the gaze point position changes is greater than a second time threshold. If so, execute S17, otherwise end.

[0160] S17: Set the transparency of the target cells on the second layer to fully transparent.

[0161] It should be noted that the static mode and motion mode provided in the embodiment of the present application do not need to be selected to use the default display mode, or you can switch to the appropriate display mode before entering the application, or you can switch to the appropriate display mode during the application running.

[0162] In an embodiment of the present application, the first layer and the second layer display different images respectively. The first layer displays the application screen, and the second layer displays the prompt information. In this way, the prompt information does not occupy the position of the first layer. The first layer can display the application screen in full screen, which makes the screen larger and more immersive. The position of the prompt information on the second layer is combined with the gaze point position detected by the eye tracking algorithm, so that the user can see the prompt information in time. At the same time, since the transparency of the second layer is processed, the display of the prompt information reduces the interference with the application screen. In addition, two display modes are provided. In static mode, the application screen and prompt information can follow the movement of the head and always remain in front of the user's field of vision. In motion mode, the application screen can be fixed at a certain position in front of the user's field of vision, so that the screen will not shake when it is slightly deflected. When it is greatly deflected, the position of the first layer can be automatically calibrated so that the application screen is always in front of the user's field of vision.

[0163] Based on the same technical concept, an embodiment of the present application provides a head-mounted display device that can implement the steps of the above-mentioned multi-layer display method and achieve the same technical effect.

[0164] See also Figure 10 , the positioning device includes a processor 1001, a memory 1002 and a display screen 1003, and the display screen 1003, the memory 1002 and the processor 1001 are connected via a bus 1004;

[0165] Display screen 1003 is used to display prompt information and application screens;

[0166] The memory 1002 stores a computer program, and the processor 1001 executes the computer program according to the computer program. Figure 1 Steps of the method shown.

[0167] It should be noted that Figure 10 The head-mounted display device shown is only an example. Optionally, the head-mounted display device may also include conventional devices such as a pickup, a microphone, a power supply, a communication interface, an IMU, and a handle.

[0168] exist Figure 10In the embodiment, the memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc. The memory may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory may also be a non-volatile memory (non-volatile memory), such as a read-only memory, a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD); or the memory may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be a combination of the above memories. The processor may include one or more central processing units (CPUs), GPUs or digital processing units, etc. The processor is configured to implement the steps of any of the above-mentioned visual map generation methods and visual map-based spatial positioning methods when calling the computer program stored in the memory.

[0169] In the embodiment of the present application, the bus can be divided into an address bus, a data bus, a control bus, etc. For ease of description, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0170] For the convenience of description, the present invention can be divided into modules (or units) according to their functions and described separately. Of course, when implementing the present invention, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0171] Those skilled in the art will appreciate that various aspects of the present application may be implemented as systems, methods, or program products. Therefore, various aspects of the present application may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0172] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions. When these instructions are executed, the steps of any one of the multi-layer display methods in the aforementioned embodiments can be completed.

[0173] An embodiment of the present application also provides a computer program product for storing a computer program, which is used to execute the steps of any one of the multi-layer display methods in the aforementioned embodiments.

[0174] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0175] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device that implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0178] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A multi-layer display method, characterized in that: include: After the device is started, a currently selected target mode is determined from a plurality of preset display modes; wherein each display mode includes a first layer and a second layer, wherein the first layer is used to display an application screen, and the second layer is used to display prompt information, and the second layer is always located in front of the first layer, and the transparency of the second layer is fully transparent so that the second layer is not visible; When the target mode is the static mode, the posture states of the first layer and the second layer are set to 0 degrees of freedom, so that the first layer and the second layer move with the movement of the user's head and are located in the center of the user's field of view; the static mode corresponds to scenes where the amount of movement does not meet the preset conditions; When the target mode is motion mode, the posture state of the first layer is set to 3 degrees of freedom and the posture state of the second layer is set to 0 degrees of freedom, so that the second layer moves with the movement of the user's head and is located at the center of the user's field of view, while the first layer does not move with the movement of the user's head; the motion mode corresponds to a scene in which the amount of movement meets a preset condition; Upon receiving the prompt information, the transparency of the second layer is set to opaque so that the prompt information is visible.

2. The method according to claim 1, wherein In the sports mode, the method further includes: When a preset calibration condition is triggered, the position of the first layer is calibrated according to the rotation data of the user's head so that the first layer is located at the center of the user's field of view.

3. The method according to claim 2, wherein The rotation data is represented in a quaternion form, and calibrating the position of the first layer according to the rotation data of the user's head includes: Calculating the current orientation of the user's head based on the quaternion of the current position of the user's head, with the positive direction of the Z axis representing the depth direction as a reference; Calculate the deflection angles of the first layer on the Y-axis and the X-axis according to the current orientation, and convert the two rotation angles into a quaternion matrix; The position of the first layer is determined according to the two quaternion matrices.

4. The method according to any one of claims 1 to 3, wherein The second layer includes a plurality of cells, and setting the transparency of the second layer to be opaque so that the prompt information is visible includes: According to the gaze point position of the binocular eyes on the second layer, the prompt information is rendered on the second layer, and the transparency of the target cell occupied by the prompt information on the second layer is set to opaque so that the target cell is visible and the transparency of other cells remains fully transparent.

5. The method according to claim 4, wherein After setting the transparency of the target cell occupied by the prompt information on the second layer to be opaque, the method further includes: When the display time of the prompt information is greater than a first time threshold, setting the transparency of the target cell on the second layer to be fully transparent so that the prompt information is invisible; When the display time of the prompt information is less than or equal to the first time threshold, the change in the gaze point position is greater than the preset range, and the stay time after the gaze point position changes is greater than the second time threshold, the transparency of the target cell on the second layer is set to fully transparent to make the prompt information invisible.

6. The method according to any one of claims 1 to 3, wherein The preset condition is that the rotation angle of the user's head is greater than a preset angle threshold, and the stay time after rotation is greater than a third time threshold.

7. A head-mounted display device, characterized in that: It includes a processor, a memory and a display screen, wherein the display screen, the memory and the processor are connected via a bus; The display screen is used to display prompt information and application screens; The memory stores a computer program, and the processor stores a computer program to perform the following operations: After the device is started, a currently selected target mode is determined from a plurality of preset display modes; wherein each display mode includes a first layer and a second layer, wherein the first layer is used to display an application screen, and the second layer is used to display prompt information, and the second layer is always located in front of the first layer, and the transparency of the second layer is fully transparent so that the second layer is not visible; When the target mode is the static mode, the posture states of the first layer and the second layer are set to 0 degrees of freedom, so that the first layer and the second layer move with the movement of the user's head and are located in the center of the user's field of view; the static mode corresponds to scenes where the amount of movement does not meet the preset conditions; When the target mode is motion mode, the posture state of the first layer is set to 3 degrees of freedom and the posture state of the second layer is set to 0 degrees of freedom, so that the second layer moves with the movement of the user's head and is located at the center of the user's field of view, while the first layer does not move with the movement of the user's head; the motion mode corresponds to a scene in which the amount of movement meets a preset condition; Upon receiving the prompt information, the transparency of the second layer is set to opaque so that the prompt information is visible.

8. The head-mounted display device according to claim 7, wherein: In the sports mode, the processor is further configured to: When a preset calibration condition is triggered, the position of the first layer is calibrated according to the rotation data of the user's head so that the first layer is located at the center of the user's field of view.

9. The head-mounted display device according to claim 7 or 8, wherein: The second layer includes a plurality of cells, and the processor sets the transparency of the second layer to opaque so that the prompt information is visible. Specifically, the operation is as follows: According to the gaze point position of the binocular eyes on the second layer, the prompt information is rendered on the second layer, and the transparency of the target cell occupied by the prompt information on the second layer is set to opaque so that the target cell is visible and the transparency of other cells remains fully transparent.

10. The head-mounted display device according to claim 9, wherein: After the processor sets the transparency of the target cell occupied by the prompt information on the second layer to be opaque, it further executes: When the display time of the prompt information is greater than a first time threshold, setting the transparency of the target cell on the second layer to be fully transparent so that the prompt information is invisible; When the display time of the prompt information is less than or equal to the first time threshold, the change in the gaze point position is greater than the preset range, and the stay time after the gaze point position changes is greater than the second time threshold, the transparency of the target cell on the second layer is set to fully transparent to make the prompt information invisible.

Images