Window display method and head-mounted display device

Abstract

The invention discloses a window display method and head-mounted display equipment. The method comprises the following steps: acquiring a motion state of a user, wherein the motion state comprises a motion direction; and moving, shrinking or hiding a target window in a display interface of the head-mounted display equipment according to the motion state so as to reduce the shielding of the view of a user.

Description

Technical Field

[0001] This application relates to the field of extended reality technology, and more particularly to a window display method and a head-mounted display device. Background Technology

[0002] As is well known, with the development of Extended Reality (XR) technology, head-mounted display devices with XR functionality have been widely used, such as XR glasses. Head-mounted displays can switch between application windows and ambient views. However, currently, the display of windows in extended reality glasses is usually actively controlled by the user through gestures or buttons. This means that if the user's state changes and they do not adjust the window display in time, they may not be able to see their surroundings, potentially leading to falls or bumps. Therefore, existing technologies present a safety issue with head-mounted display devices. Summary of the Invention

[0003] The purpose of this application is to provide a window display method and a head-mounted display device that can solve the problem of poor security in the use of head-mounted display devices.

[0004] In a first aspect, embodiments of this application provide a window display method applied to a head-mounted display device, including:

[0005] Acquire the user's motion state, including the direction of motion;

[0006] The target window in the display interface of the head-mounted display device can be moved, shrunk, or hidden according to the motion state to reduce obstruction of the user's field of vision.

[0007] Secondly, embodiments of this application provide a head-mounted display device, including:

[0008] An inertial measurement module is used to acquire the user's motion state, including the direction of motion;

[0009] The display module is used to move, shrink, or hide the target window in the display interface of the head-mounted display device according to the motion state, so as to reduce obstruction of the user's field of vision.

[0010] Thirdly, embodiments of this application provide a head-mounted display device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0011] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0012] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0013] In this embodiment, by acquiring the user's motion state, including the direction of motion, the target window in the display interface of the head-mounted display device is moved, contracted, or hidden according to the motion state to reduce obstruction of the user's field of vision. This allows for proactive triggering of the target window's movement, contraction, or hiding based on the motion state, thereby reducing obstruction of the user's field of vision and improving the safety of using the head-mounted display device. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is one of the example diagrams for window positioning in related technologies;

[0016] Figure 2 This is the second example diagram of window positioning in related technologies;

[0017] Figure 3 This is a flowchart illustrating a window display method provided in an embodiment of this application;

[0018] Figure 4 This is one of the example scenarios in which the window display method provided in the embodiments of this application is applied;

[0019] Figure 5 This is an example diagram of the user's movement trajectory in the window display method provided in the embodiments of this application;

[0020] Figure 6 This is the second example diagram illustrating the application scenario of the window display method provided in this application embodiment;

[0021] Figure 7 This is the third example of a scenario in which the window display method provided in this application is applied;

[0022] Figure 8This is the fourth example of a scenario in which the window display method provided in this application is applied;

[0023] Figure 9 This is a flowchart illustrating another window display method provided in an embodiment of this application;

[0024] Figure 10 This is a schematic diagram of the structure of a head-mounted display device provided in an embodiment of this application;

[0025] Figure 11 This is a schematic diagram of the structure of a head-mounted display device provided in an embodiment of this application;

[0026] Figure 12 This is a schematic diagram of another head-mounted display device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0029] To better understand this application, some technical aspects involved in this application will be explained below.

[0030] 1. Zero Degree of Freedom (0DoF): refers to the interactive mode in head-mounted display devices (such as XR glasses) where the screen moves completely with the head's movements, that is, the screen moves in real time with the head's left and right turns, up and down tilts, etc.

[0031] 2. Six Degrees of Freedom (6DoF): This refers to the ability of head-mounted display devices (such as XR glasses) to track their own three translational movements (forward / backward, left / right, up / down) and three rotational movements (pitch, yaw, roll) in three-dimensional space. This allows the system to accurately know the real-time position and orientation of the user's head.

[0032] III. Visual Simultaneous Localization and Mapping (VSLM): One of the core technologies for achieving spatial positioning. By using cameras on augmented reality glasses to capture images of the surrounding environment in real time, extracting object feature points, and simultaneously calculating the object's position and orientation based on these feature points, a 3D map of the surrounding environment is constructed.

[0033] IV. Inertial Measurement Unit (IMU): A functional module that uses accelerometers and gyroscopes to measure the motion state of an object in three-dimensional space.

[0034] V. Currently, the window positioning methods for XR glasses include:

[0035] 1. 6DoF fixed window + VSLAM: The window's position in space is fixed and will not change position as the XR glasses move. Figure 1 As shown. This positioning method is a commonly used window positioning technique in XR glasses.

[0036] 2. 0DoF fixed window: The window always moves with the XR glasses, such as... Figure 2 As shown. This positioning method is mainly used for window positioning in mobile scenarios.

[0037] The 0DoF fixed window technology is mainly suitable for high-speed movement scenarios such as traveling by car, high-speed rail, subway, and airplane, where the user does not need to move their relative position within the vehicle. However, if a user is wearing XR glasses indoors and needs to move (for example, while watching a movie with XR glasses, they want to walk from the living room to the bedroom), the application window under 0DoF may obstruct the user's view, causing the user to trip over obstacles and creating safety issues. Therefore, the window display method proposed in this application is presented.

[0038] The window display method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0039] See Figure 3 , Figure 3 This is a flowchart of a window display method provided in an embodiment of this application, applied to a head-mounted display device, such as... Figure 1 As shown, the method for displaying this window includes the following steps:

[0040] Step 301: Obtain the user's motion state, including the direction of motion;

[0041] Step 302: Move, shrink, or hide the target window in the display interface of the head-mounted display device according to the motion state to reduce obstruction of the user's field of vision.

[0042] In this embodiment, the head-mounted display device may include an IMU (Integrated Measurement Unit) capable of detecting the motion state of the head-mounted display device. For example, the IMU may have a built-in accelerometer and gyroscope for detecting the motion state of the head-mounted display device in three-dimensional space. Specifically, the data output by the accelerometer and gyroscope can be understood as displacement data. Based on this displacement data, the motion state of the head-mounted display device in three-dimensional space can be determined, and based on this motion state, it can be determined whether to adjust the display state of a window. If it is determined that the display state of the window should be adjusted, the target window in the display interface of the head-mounted display device can be moved, collapsed, or hidden.

[0043] It should be noted that the head-mounted display device in this application embodiment may include, but is not limited to, XR glasses, mixed reality (MR) glasses, or augmented reality (AR) glasses. The aforementioned window may be a 2D window or a 3D window. It should be understood that the head-mounted display device may display one or more windows, and when multiple windows are displayed, the multiple windows may be arranged in an array.

[0044] Optionally, in some embodiments, moving the target window can be understood as moving the target window's position from its initial position to a target position. During this movement, the initial display size of the target window can remain unchanged, or the display size of the target window can be reduced.

[0045] Optionally, in some embodiments, shrinking the target window can be understood as reducing the display size of the target window from its initial size to a target size. The size of this target size can be set according to actual needs and is not further limited here. After shrinking, the display position of the target window can be set according to actual needs. For example, after all windows are shrunk, it can be displayed in a designated area, such as the edge or corner area of ​​the display interface of the head-mounted display device; or, for example, it can be displayed in the edge or corner area of ​​the previous display area of ​​the target window. Optionally, shrinking the target window can also be understood as minimizing the target window.

[0046] Optionally, in some embodiments, the above-mentioned hiding can be understood as minimizing the target window. After minimizing the target window, the icons associated with the target window can be displayed. The icon associated with each window in the target window can be a specified image or a thumbnail of that window. The icon associated with the target window can be displayed in a specified area, such as in a non-human-eye core area of ​​the display interface. Specifically, as follows... Figure 4 As shown.

[0047] Optionally, in some embodiments, the above-mentioned hiding can also be understood as canceling the display of the target window, and not displaying any information of the target window in the display interface of the head-mounted display device. At this time, the user needs to enter the specified function interface to display the hidden target window, or restore the display of the target window based on the user's status or the operation command input by the user.

[0048] Optionally, the target window can be understood as a window within a specified area, or all windows. By moving, shrinking, or hiding the target window, users wearing head-mounted displays can see objects in their current environment. This allows users to avoid obstacles while in motion, thereby improving the safety of using head-mounted displays.

[0049] In this embodiment, by acquiring the user's motion state, including the direction of motion, the target window in the display interface of the head-mounted display device is moved, contracted, or hidden according to the motion state to reduce obstruction of the user's field of vision. This allows for proactive triggering of the target window's movement, contraction, or hiding based on the motion state, thereby reducing obstruction of the user's field of vision and improving the safety of using the head-mounted display device.

[0050] Optionally, in some embodiments, the direction of movement includes the vertical direction;

[0051] As the user moves along the vertical direction, the target window can be moved, collapsed, or hidden.

[0052] In this embodiment of the application, when the head-mounted display device detects that the user wearing the head-mounted display device is moving along a numerical direction, the target window can be moved, shrunk, or hidden.

[0053] For example, when a user is in scenarios such as riding an elevator, sitting down, standing up, going uphill, going downhill, or climbing stairs, it can be considered that the user is in a scenario where collisions are likely to occur or where there is a potential danger. In such cases, by moving, shrinking, or hiding the target window, obstruction of the user's field of vision can be reduced, preventing the user from colliding or falling in scenarios where collisions are likely to occur or where there is a potential danger. Therefore, the embodiments of this application improve the safety of using head-mounted display devices.

[0054] Optionally, in some embodiments, the direction of movement includes a vertical direction and a horizontal direction; the vertical direction is perpendicular to the horizontal direction, and the target window is moved, collapsed, or hidden when the angle between the user's direction of movement and the horizontal direction is greater than 25°.

[0055] Optionally, the angle between the user's movement direction and the horizontal direction can be understood as the angle of the displacement generated by the user's movement in the vertical direction. For example, in some embodiments, this angle can be 40° or 45°, etc., and is not further limited here.

[0056] In this embodiment of the application, when the angle between the user's direction of movement and the horizontal direction is greater than 25°, it can be determined that the user is in a scenario such as going up or down stairs, riding an escalator, going uphill, or going downhill. For example, when a user is walking up an indoor staircase, there is a height difference, and the user's range of movement may be 360°. Therefore, during the movement, the user will form a circular main range of movement with the user's standing point as the center, and the displacement generated by the user's movement will have an angle greater than 25° in the vertical direction.

[0057] It should be noted that, in this embodiment of the application, by detecting the angle between the vertical and horizontal directions, dangerous scenarios such as riding an elevator, sitting down, standing up, going uphill, going downhill, and going up and down stairs can be identified. When it is determined that the user is in a dangerous scenario such as riding an elevator, sitting down, standing up, going uphill, going downhill, or going up and down stairs, the target window can be actively moved, shrunk, or hidden to increase the user's field of vision of the current environment and improve the safety of using the head-mounted display device.

[0058] Optionally, in some embodiments, the direction of motion includes a vertical direction, and the motion state also includes the vertical motion acceleration of the user moving along the vertical direction. The user's motion state is obtained by the accelerometer of the inertial measurement module of the head-mounted display device. If the motion state confirms that the user is in at least one of the following scenarios: going up or down stairs, standing up, or sitting down, the target window is moved, collapsed, or hidden.

[0059] In this embodiment, acceleration data collected by an accelerometer can be acquired at preset time intervals to determine the current acceleration. This acceleration may include vertical motion acceleration (moving in the vertical direction) and / or horizontal motion acceleration (moving in the horizontal direction). Based on the vertical motion acceleration, it can be determined whether the user is in a scenario such as going up or down stairs, standing up, or sitting down. For example, if the vertical motion acceleration is greater than a preset value, it can be determined that the user is in a scenario such as going up or down stairs, standing up, or sitting down. This embodiment can determine whether the user is in a scenario such as going up or down stairs, standing up, or sitting down through vertical motion acceleration, and when it is determined that the user is in such a scenario, the target window is moved, collapsed, or hidden, thereby improving the safety of using the head-mounted display device.

[0060] Optionally, in some embodiments, the direction of movement includes a horizontal direction; the method further includes:

[0061] As the user moves along the horizontal direction, the relative displacement between the user and the target object in the real scene is confirmed by the camera of the head-mounted display device.

[0062] When the user moves relative to the target object, the target window is moved, collapsed, or hidden.

[0063] The target window continues to be displayed while the user remains stationary relative to the target object.

[0064] In this embodiment, whether a user is moving in the horizontal direction can be determined by the horizontal acceleration and the horizontal velocity along the horizontal direction. For example, in some embodiments, the horizontal acceleration and horizontal velocity can be determined based on the acceleration output by the accelerometer. If the horizontal acceleration is greater than a set threshold and / or the horizontal velocity is greater than a set threshold, it can be determined that the user is moving in the horizontal direction. At this time, the relative displacement between the user and the target object in the real scene is confirmed by the camera of the head-mounted display device, thereby determining whether the user is actively or passively moving.

[0065] It should be understood that continuing to display the target window can be interpreted as maintaining the target window's display state unchanged, or in other words, not adjusting the target window. For example, continuing to maintain the target window in a first state; wherein, the first state is the display state of the target window before it was moved, collapsed, or hidden.

[0066] For example, when a user moves relative to the target object, it indicates that the user has moved within the actual environment, which can be considered as active movement. In this case, because the user is actively moving, there may be potential security risks. By moving, shrinking, or hiding the target window, the user's field of vision of the real scene can be improved, avoiding potential risks during movement.

[0067] For example, if the user is stationary relative to the target object, it means the user has not moved in the actual environment and can be considered as passively moving, such as in a scenario where they are using public transportation, where there is no potential risk. In this case, the head-mounted display device can be controlled to continue maintaining the target window in its first state, i.e., without triggering adjustments to the target window. This avoids accidentally triggering adjustments to the target window's display state in movement scenarios without potential risks (such as using public transportation), thereby preventing interruptions to video viewing or gaming operations on the head-mounted display device due to target window adjustments, thus improving the reliability of the head-mounted display device.

[0068] Optionally, the target object can be understood as a fixed object whose position usually remains unchanged in the actual scene, such as including but not limited to at least one of the following: the inner wall of a vehicle, roadside buildings, and ground texture. Since the user is stationary relative to the target object whose position remains unchanged, there is no potential risk. When the user moves along the horizontal direction, the camera of the head-mounted display device confirms the relative displacement between the user and the target object in the real scene, which can avoid accidentally triggering the adjustment of the target window.

[0069] It should be noted that the camera of the head-mounted display device can capture environmental images in real time and determine the relative displacement between the user and the target object in the real scene; or, the camera of the head-mounted display device can capture environmental images in real time and, when the user moves along the horizontal direction, trigger the determination of the relative displacement between the user and the target object in the real scene; or, when the user moves along the horizontal direction, trigger the camera of the head-mounted display device to capture environmental images and determine the relative displacement between the user and the target object in the real scene.

[0070] Optionally, in some embodiments, the movement of the user relative to the target object includes: within a unit time range, the relative displacement between the user and the target object is greater than a first threshold.

[0071] The condition that the user is stationary relative to the target object includes: within a unit time range, the relative displacement between the user and the target object is less than the first threshold.

[0072] In this embodiment of the application, the specific time setting of the above-mentioned unit time range can be set according to actual needs. For example, the above-mentioned unit time range can be 0.5 seconds, 1 second, or 2 seconds, etc.

[0073] Optionally, in some embodiments, the environmental image captured by the camera of the head-mounted display device can be acquired every second. When the user moves relative to the target object, the position of the target object in two adjacent environmental images will change. The relative displacement between the user and the target object can be determined by comparing the position information of the target object in two adjacent environmental images.

[0074] It should be understood that, in this embodiment of the application, by setting a first threshold, the user is determined to be stationary or moving relative to the target object based on the comparison between the first threshold and the relative displacement between the user and the target object. This allows for a certain degree of jitter error, improves the accuracy of user status judgment, and avoids erroneous triggering of target window adjustments.

[0075] Optionally, in some embodiments, the direction of movement includes a horizontal direction; the motion state also includes the user's horizontal movement speed and / or horizontal movement acceleration along the horizontal direction, and moving, shrinking, or hiding the target window in the display interface of the head-mounted display device according to the motion state includes:

[0076] The motion state includes the horizontal motion speed. If the change in the horizontal motion speed per unit time is greater than a second threshold, it is determined that the user is moving along the horizontal direction, and the target window is moved, collapsed, or hidden. If the change in the horizontal motion speed per unit time is less than the second threshold, it is determined that the user is not moving along the horizontal direction, and the target window remains in a first state; and / or,

[0077] The motion state includes the horizontal motion acceleration. If the horizontal motion acceleration is greater than a third threshold, the user is determined to be moving along the horizontal direction, and the target window is moved, collapsed, or hidden. If the horizontal motion acceleration is less than the third threshold, the target window is maintained in a first state; and / or,

[0078] The motion state includes the horizontal motion speed, which includes the speed in the direction of motion and the speed in the normal direction of motion. If the change in the speed in the normal direction of motion per unit time is greater than a fourth threshold, it is determined that the user is moving along the horizontal direction, and the target window is moved, collapsed, or hidden. If the change in the speed in the normal direction of motion per unit time is less than the fourth threshold and the change in the speed in the direction of motion per unit time is less than a fifth preset value, the target window is maintained in a first state.

[0079] The first state refers to the display state of the target window before it is moved, collapsed, or hidden.

[0080] It should be noted that, in the embodiments of this application, the determination of whether the window adjustment condition is met can be based solely on the horizontal motion velocity or solely on the horizontal motion acceleration, or a determination can be based on both the horizontal motion velocity and the horizontal motion acceleration. The window adjustment condition is used to indicate a change in the motion state or the occurrence of horizontal displacement. The condition for meeting the window adjustment condition may include at least one of the following:

[0081] The change in the horizontal motion speed per unit time is greater than the second threshold.

[0082] The horizontal acceleration is less than the third threshold.

[0083] The change in velocity in the normal direction of motion per unit time is greater than the fourth threshold.

[0084] Optionally, the magnitude of the second threshold can be set according to actual needs. For example, in some embodiments, the second threshold can be 0.25 m / s. That is, when the absolute value of the horizontal movement speed per unit time is greater than 0.25 m / s, the motion state of the terminal can be considered to have changed. Optionally, in scenarios such as a user starting to walk from a standstill, accelerating or decelerating during walking, the speed will exhibit a change characteristic greater than the second threshold.

[0085] Optionally, the magnitude of the third threshold can be set according to actual needs. For example, in some embodiments, the third threshold can be 0.25 m / s². That is, when a horizontal acceleration greater than 0.25 m / s² is detected, the motion state of the terminal can be considered to have changed. Optionally, in scenarios such as a user starting to walk from a standstill, accelerating or decelerating during walking, the acceleration will exhibit a change characteristic greater than the third threshold.

[0086] Optionally, in some embodiments, the vector of horizontal motion velocity can be decomposed into velocity in the direction of motion and velocity in the normal direction of motion. If the change in velocity in the normal direction of motion per unit time is greater than a fourth threshold, the current motion state can be considered to have a turning trend. The magnitude of the fourth threshold can be set according to actual needs; for example, in some embodiments, the fourth threshold can be 0.25 m / s. It should be understood that if the change in velocity in the normal direction of motion per unit time is less than the fourth threshold, and the change in velocity in the direction of motion per unit time is less than a fifth preset value, the user's motion state can be considered unchanged, or the user has not generated displacement, and in this case, there is no need to adjust the display state of the target window.

[0087] like Figure 5 As shown, when turning, the direction of motion can be understood as the tangent direction of the trajectory, and the direction perpendicular to this direction of motion can be understood as the normal direction of motion, such as... Figure 5 As shown, the solid black line represents the user's movement trajectory. When the user is at point A, the direction of movement is as shown by arrow A1, and the normal direction of movement is as shown by arrow A2.

[0088] Optionally, in some embodiments, the method further includes:

[0089] When the user moves along the horizontal direction, the visual-assisted positioning function of the head-mounted display device is activated to confirm the relative displacement between the user and the target object.

[0090] In this embodiment, the aforementioned visual-assisted positioning function can be understood as a VSLM (Visually Assisted Positioning) function. When horizontal movement of the user is detected, the visual-assisted positioning function can be activated, and based on this function, the relative displacement between the user and the target object can be determined. Since the visual-assisted positioning function of the head-mounted display device is activated when the user moves horizontally, power consumption can be reduced.

[0091] It's important to note that while the displacement data output by the IMU can sense the user's motion relative to the Earth's frame of reference, it cannot distinguish between the user's active motion and the user's passive motion while traveling on a vehicle. For example, when a user is traveling on a high-speed train, airplane, or ship, the IMU may detect significant changes in acceleration and velocity, misinterpreting this as a change in motion. However, in this situation, the user is stationary relative to the vehicle's interior environment, which is a safe scenario and does not require intervention. Visual-assisted positioning, on the other hand, allows for visual perception and calculation of the user's relative displacement to a target object, filtering out misinterpretations caused by passive motion and improving the accuracy of the judgment.

[0092] Optionally, in some embodiments, confirming the relative displacement between the user and the target object in the real-world scene via the camera of the head-mounted display device includes:

[0093] At least two frames of scene images, including the target object, are acquired through the camera of the head-mounted display device;

[0094] Feature points are extracted from each frame of the image;

[0095] Calculate the depth information of the feature points;

[0096] Generate a point cloud matrix of feature points corresponding to each frame of the scene image based on the depth information of the feature points;

[0097] Calculate the displacement change information of the feature points corresponding to the target object based on the point cloud matrix of the feature points corresponding to the at least two frames of scene images;

[0098] The relative displacement between the user and the target object in the real-world scene is determined based on the displacement change information.

[0099] In this embodiment, the aforementioned camera can be a binocular camera. A binocular camera can simultaneously capture the same scene from different angles. The same three-dimensional point will form different pixel positions on the left and right images. This position difference is called "parallax (d)". A frame of photographs captured by a binocular camera is an image with depth information formed based on the left and right images.

[0100] Optionally, the depth information of the feature points can be calculated as follows: Based on the principle of binocular vision, the three-dimensional depth of each point in the scene can be accurately calculated using the formula Z=(f×B) / d by fixing hardware parameters (such as camera focal length f and the distance between the baselines of the left and right cameras B), where Z represents the distance from the feature point to the camera.

[0101] Furthermore, a variable-scale stereo matching algorithm can be used to optimize the parallax calculation accuracy for different distances (near, middle, and far) and different texture features (high-texture and low-texture areas), reduce mismatches, and ensure the reliability of depth information.

[0102] Since the position of the feature points corresponding to the target object is relatively fixed in the real scene, when the camera (i.e. the user) moves relative to the scene, the relative position of the camera and each point in the point cloud matrix will change. By comparing the changes in the point cloud position of the point cloud matrix corresponding to different images, the actual moving distance of the camera (i.e. the user) relative to the real scene can be accurately calculated, and thus the relative displacement between the user and the target object in the real scene can be obtained.

[0103] like Figure 6 and Figure 7 As shown, Figure 6 This is an example image of the user's scene at the first moment. Figure 7 This is an example image of the user's scene at the second moment. Object A within the scene is relatively fixed. Therefore, when the camera moves within the scene, its position relative to the point cloud within the scene changes. The distance the head-mounted display device has moved can then be calculated from the point cloud positions in the point cloud matrix.

[0104] Optionally, in some embodiments, obtaining the user's motion state includes: obtaining the motion direction through the inertial measurement module of the head-mounted display device.

[0105] Optionally, in some embodiments, the motion state further includes motion speed; if the motion speed is greater than a first speed threshold, the target window is moved, collapsed, or hidden; and / or,

[0106] If the movement speed is less than a second speed threshold and the duration of the state where the movement speed is less than the second speed threshold is greater than a first time threshold, the target window is adjusted from the second state to the first state; if the movement speed is less than the second speed threshold and the duration of the state where the movement speed is less than the second speed threshold is less than the first time threshold, the target window is maintained in the second state.

[0107] The first state is the display state before the target window is moved, shrunk, or hidden, and the second state is the display state after the target window is moved, shrunk, or hidden.

[0108] In this embodiment, the aforementioned movement speed can be understood as the user's displacement speed, which may include the combined speed of horizontal and vertical displacement, or the instantaneous speed at which the user is currently moving in the horizontal direction, i.e., the combined speed of the speed in the horizontal movement direction and the speed in the horizontal movement normal direction. The magnitude of the aforementioned first speed threshold can be set according to actual needs; for example, in some embodiments, the first speed threshold can be 0.5 m / s.

[0109] Optionally, when the movement speed is less than the second speed threshold, the user can be considered to have stopped moving horizontally. By setting a first time threshold, if the duration of the state where the movement speed is less than the second speed threshold is greater than the first time threshold, the user can be considered to have stopped moving horizontally and has reached a stable state. The target window is then restored only when the user is in a stable state. This can avoid the target window being adjusted multiple times in a short period of time, which would result in a poor user experience.

[0110] Optionally, in some embodiments, the target window includes a window located at the center of the display interface of the head-mounted display device; and / or,

[0111] The step of moving the target window in the display interface of the head-mounted display device according to the motion state includes: moving the window located at the center of the display interface toward the edge of the display interface.

[0112] In this embodiment, the central area of ​​the display interface is the core area of ​​the human eye's field of vision. By moving the window located at the center of the display interface toward the edge of the display interface, the window can be prevented from obstructing the core area of ​​the human eye's field of vision, thereby allowing the user to better view the real-world scene and further improving the safety of using the head-mounted display device.

[0113] It should be understood that the core area of ​​human visual field can be the area within ±15° of the center of the field of vision, which is the core area of ​​the user's visual focus.

[0114] In some embodiments, when the target window is moved, it can be moved to a non-human visual field core area. The specific moving position can be located in the upper left or upper right corner of the display interface. Specifically, it can be set according to the user's dominant hand configuration. For example, when the right hand is the dominant hand, the upper left corner of the display interface can be selected as the display area after the target window is moved. If the left hand is the dominant hand, the upper right corner of the field of vision can be selected as the display area after the target window is moved.

[0115] Optionally, in some embodiments, in response to a first input, the target window is adjusted from a second state to a first state; the first input includes at least one of gesture input, voice input, touch input, eye-tracking input, and posture input in response to changes in head posture; and / or,

[0116] Based on the motion state, the user is determined to stop moving, and the target window is adjusted from the second state to the first state.

[0117] The first state is the display state before the target window is moved, shrunk, or hidden, and the second state is the display state after the target window is moved, shrunk, or hidden.

[0118] In this embodiment, the first input can be understood as user input. For example, in some embodiments, when the user determines that the security risk has been eliminated, they can input the first input. After receiving the first input from the user, the head-mounted display device can consider that the current security risk has been eliminated and can restore the display of the target window, that is, adjust the target window from the second state to the first state. This allows the user to actively restore the window display, thereby improving the flexibility of the window restoration operation.

[0119] It should be noted that when restoring the target window, only some windows can be restored, or all windows can be restored; no further limitations are imposed here. Optionally, different windows can be restored based on different initial inputs.

[0120] For example, in some embodiments, the first input can be voice input, and a specific user can control at least a portion of the window in the target window to return to a first state by inputting voice.

[0121] For example, in some embodiments, the first input can be a gesture input. A specific user can pre-set different gestures to restore different windows, or set a specific gesture to restore all windows, so that at least some windows in the target window can be restored to the first state by inputting gestures.

[0122] For example, in some embodiments, the first input can be a touch input. Specifically, the user can click the minimized preview icon to restore the specified window to the first state, or click a specific virtual button or physical button to restore at least a portion of the target window to the first state.

[0123] For example, in some embodiments, the first input can be eye-tracking input. Specifically, the user can pre-set different eye movements (e.g., one blink or two blinks) to be associated with a specific window, or set a specific eye movement (e.g., the eyeball moves to the left and continues for a period of time) to be associated with all windows, so that one or more windows in the target window can be restored to the first state based on eye-tracking input.

[0124] For example, in some embodiments, the first input can be a head posture change input, and the user can achieve posture input by moving their head left and right. At this time, all target windows are restored to the first state.

[0125] It should be noted that, based on the motion state, once it is determined that the user has stopped moving and / or the first input has been received, the safety hazard can be eliminated, thereby restoring the display of the target window. Specifically, as follows... Figure 8 As shown.

[0126] Optionally, in some embodiments, the motion state further includes the vertical acceleration of the user moving along the vertical direction and the horizontal acceleration of the user moving along the horizontal direction, and determining that the user stops moving based on the motion state includes:

[0127] The user is determined to stop moving if at least one of the following conditions is met:

[0128] The duration during which the horizontal motion acceleration is less than the third threshold is greater than the first duration;

[0129] The duration during which the change in the horizontal motion speed per unit time is less than the second threshold is greater than the second duration;

[0130] The duration during which the horizontal motion speed is less than the fifth threshold is greater than the third duration;

[0131] The duration during which the vertical velocity is less than the sixth threshold is greater than the fourth duration.

[0132] In the embodiments of this application, the first duration, second duration, third duration, and fourth duration may be the same or different, and no further limitation is made here. For example, in some embodiments, the first duration, second duration, third duration, and fourth duration may all be 2 seconds or 3 seconds.

[0133] Optionally, in some embodiments, the fifth threshold is 0.1 m / s²; the sixth threshold may be 0.1 m / s.

[0134] It should be noted that, taking the first, second, third, and fourth durations as all being 3 seconds as an example, if any of the following conditions are met within 3 seconds, it is necessary to re-determine whether the user has stopped moving:

[0135] The duration during which the horizontal motion acceleration is greater than the third threshold;

[0136] The duration during which the change in the horizontal motion speed per unit time is greater than the second threshold.

[0137] The duration during which the horizontal motion speed is less than the fifth threshold;

[0138] The duration during which the vertical velocity is less than the sixth threshold.

[0139] It should be understood that by determining that the user has stopped moving, the target window can be restored directly without waiting, and normal operation can be resumed immediately. Therefore, the embodiments of this application improve the flexibility of restoring the target window.

[0140] Optionally, in some embodiments, adjusting the target window from the first state to the second state includes:

[0141] The target window is adjusted from the first state to the second state according to a preset movement rate and / or a preset size magnification rate.

[0142] In this embodiment, a smooth transition of the target window is achieved by adjusting the target window according to a preset movement rate and a preset size magnification rate. For example, in some embodiments, the recovery time from the second state to the first state can be between 0.8 seconds and 1.2 seconds. Since it does not exceed 1.2 seconds, the waiting time is eliminated; and since it is not less than 0.8 seconds, visual impact is avoided. The specific time can be determined based on the size change and displacement change from the second state to the first state. For example, in the second state, the smaller the size, the longer the recovery time; and the larger the size, the shorter the recovery time.

[0143] Optionally, during the recovery process, the window's hierarchy is temporarily elevated to the "top layer of the current interface," i.e., above other application windows and system controls, to avoid being obscured and to ensure that the user can clearly perceive the recovery process.

[0144] It should be noted that the specific restoration content for adjusting the target window from the second state to the first state includes at least one of the following: restoring the display position, restoring the display size, and restoring the display state. The display state may include the progress of the content, the cursor input position, and the sub-menus that have been expanded within the window, which can ensure the continuity of user operation.

[0145] Optionally, in some embodiments, the triggering conditions for moving, shrinking, or hiding the target window include at least one of the following:

[0146] The head-mounted display device is in motion-linked mode;

[0147] The head-mounted display device has activated the function of adjusting the display status of the motion trigger window;

[0148] The target window belongs to a preset application, which is an application that disables motion linkage mode.

[0149] The motion linkage mode is a mode based on the adjustment of the motion status window display status.

[0150] In this embodiment, the above-mentioned motion linkage mode can be understood or replaced by the 0DoF mode. When the head-mounted display device is in a non-motion linkage mode, the judgment logic and window shrinkage control of this application are no longer applicable.

[0151] For example, when switching to 3DoF or 6DoF immersive mode, such as in immersive interactive scenarios of virtual reality (VR) devices or augmented reality (AR) head-mounted displays, motion linkage mode will be disabled to avoid frequent window changes.

[0152] For example, when switching to "Focus Mode", the system disables motion-linked mode by default to avoid frequent window changes interfering with concentration.

[0153] Optionally, in some embodiments, users can actively disable the motion linkage mode of certain applications, so that these applications will no longer be controlled by window shrinking, thereby improving the user experience of specific applications.

[0154] To better understand this application, the following is based on... Figure 9 The flowchart shown illustrates the control logic of this application.

[0155] like Figure 9 As shown, the process includes the following:

[0156] Step 91: Acquire IMU data to obtain motion status;

[0157] Step 92: Determine whether vertical movement has occurred based on the motion state; if vertical movement has occurred, proceed to step 93; otherwise, proceed to step 94.

[0158] Step 93: Move, shrink, or hide the target window in the display interface of the head-mounted display device;

[0159] Step 94: Monitor the horizontal movement status to determine if horizontal movement has occurred; if horizontal movement has occurred, proceed to step 95; otherwise, proceed to step 97.

[0160] Step 95: Obtain VLSM data, which involves capturing scene images with a camera, examining the point cloud matrix of feature points based on the captured scene images, and determining the displacement change information of the feature points corresponding to the target object. This VLSM data is the displacement change information of the feature points corresponding to the target object.

[0161] Step 96: Based on VLSM data, determine whether the user has moved relative to the target object in the real scene. If movement has occurred, proceed to step 93; otherwise, proceed to step 97.

[0162] Step 97: Continue to monitor motion status.

[0163] It should be noted that the window display method provided in this application embodiment can be executed by a head-mounted display device or a control module in the head-mounted display device for executing the loading window display method. This application embodiment uses the execution of the loading window display method by a head-mounted display device as an example to illustrate the window display method provided in this application embodiment.

[0164] See Figure 10 , Figure 10 This is a structural diagram of the head-mounted display device provided in the embodiments of this application, such as... Figure 10 As shown, the head-mounted display device 1000 includes:

[0165] An inertial measurement module 1001 is used to acquire the user's motion state, including the direction of motion;

[0166] The display module 1002 is used to move, shrink, or hide the target window in the display interface of the head-mounted display device according to the motion state, so as to reduce obstruction of the user's field of vision.

[0167] Optionally, the direction of movement includes the vertical direction;

[0168] As the user moves along the vertical direction, the target window can be moved, collapsed, or hidden.

[0169] Optionally, the movement direction includes a vertical direction and a horizontal direction; the vertical direction is perpendicular to the horizontal direction, and the display module 1002 is specifically used to move, shrink, or hide the target window when the angle between the user's movement direction and the horizontal direction is greater than 25°.

[0170] Optionally, the direction of motion includes the vertical direction, and the motion state also includes the vertical motion acceleration of the user moving along the vertical direction. The user's motion state is obtained by the accelerometer of the inertial measurement module of the head-mounted display device. The display module 1002 is specifically used to move, shrink, or hide the target window when the user is confirmed to be in at least one of the following scenarios: going up or down stairs, standing up, or sitting down, based on the motion state.

[0171] Optionally, the direction of movement includes the horizontal direction; the display module 1002 is specifically used for:

[0172] As the user moves along the horizontal direction, the relative displacement between the user and the target object in the real scene is confirmed by the camera of the head-mounted display device.

[0173] When the user moves relative to the target object, the target window is moved, collapsed, or hidden.

[0174] The target window continues to be displayed while the user remains stationary relative to the target object.

[0175] Optionally, the movement of the user relative to the target object includes: within a unit time range, the relative displacement between the user and the target object is greater than a first threshold.

[0176] The condition that the user is stationary relative to the target object includes: within a unit time range, the relative displacement between the user and the target object is less than the first threshold.

[0177] Optionally, the direction of movement includes a horizontal direction; the motion state also includes the horizontal velocity and / or horizontal acceleration of the user moving along the horizontal direction, and the display module 1002 is specifically used for:

[0178] The motion state includes the horizontal motion speed. If the change in the horizontal motion speed per unit time is greater than a second threshold, it is determined that the user is moving along the horizontal direction, and the target window is moved, collapsed, or hidden. If the change in the horizontal motion speed per unit time is less than the second threshold, it is determined that the user is not moving along the horizontal direction, and the target window is maintained in a first state.

[0179] The motion state includes the horizontal motion acceleration. If the horizontal motion acceleration is greater than a third threshold, it is determined that the user is moving along the horizontal direction, and the target window is moved, collapsed, or hidden. If the horizontal motion acceleration is less than the third threshold, the target window is maintained in a first state.

[0180] The motion state includes the horizontal motion speed, which includes the speed in the direction of motion and the speed in the normal direction of motion. If the change in the speed in the normal direction of motion per unit time is greater than a fourth threshold, it is determined that the user is moving along the horizontal direction, and the target window is moved, collapsed, or hidden. If the change in the speed in the normal direction of motion per unit time is less than the fourth threshold and the change in the speed in the direction of motion per unit time is less than a fifth preset value, the target window is maintained in a first state.

[0181] The first state refers to the display state of the target window before it is moved, collapsed, or hidden.

[0182] Optionally, the display module 1002 is further configured to: activate the visual-assisted positioning function of the head-mounted display device to confirm the relative displacement between the user and the target object when the user moves along the horizontal direction.

[0183] Optionally, the display module 1002 is specifically used for:

[0184] At least two frames of scene images, including the target object, are acquired through the camera of the head-mounted display device;

[0185] Feature points are extracted from each frame of the image;

[0186] Calculate the depth information of the feature points;

[0187] Generate a point cloud matrix of feature points corresponding to each frame of the scene image based on the depth information of the feature points;

[0188] Calculate the displacement change information of the feature points corresponding to the target object based on the point cloud matrix of the feature points corresponding to the at least two frames of scene images;

[0189] The relative displacement between the user and the target object in the real-world scene is determined based on the displacement change information.

[0190] Optionally, the inertial measurement module 1001 is specifically used to: obtain the direction of motion through the inertial measurement module of the head-mounted display device.

[0191] Optionally, the motion state further includes motion speed; if the motion speed is greater than a first speed threshold, the target window is moved, collapsed, or hidden; and / or,

[0192] If the movement speed is less than a second speed threshold and the duration of the state where the movement speed is less than the second speed threshold is greater than a first time threshold, the target window is adjusted from the second state to the first state; if the movement speed is less than the second speed threshold and the duration of the state where the movement speed is less than the second speed threshold is less than the first time threshold, the target window is maintained in the second state.

[0193] The first state is the display state before the target window is moved, shrunk, or hidden, and the second state is the display state after the target window is moved, shrunk, or hidden.

[0194] Optionally, the target window includes a window located at the center of the display interface of the head-mounted display device; and / or,

[0195] The display module 1002 is specifically used to move the window located at the center of the display interface toward the edge of the display interface.

[0196] Optionally, the display module 1002 is specifically configured to: respond to a first input and adjust the target window from a second state to a first state; the first input includes at least one of gesture input, voice input, touch input, eye-tracking input, and posture input in response to head posture changes; and / or, determine that the user has stopped moving based on the motion state and adjust the target window from a second state to a first state;

[0197] The first state is the display state before the target window is moved, shrunk, or hidden, and the second state is the display state after the target window is moved, shrunk, or hidden.

[0198] Optionally, the motion state further includes the vertical acceleration of the user moving along the vertical direction and the horizontal acceleration of the user moving along the horizontal direction. The display module 1002 is specifically used for:

[0199] The user is determined to stop moving if at least one of the following conditions is met:

[0200] The duration during which the horizontal motion acceleration is less than the third threshold is greater than the first duration;

[0201] The duration during which the change in the horizontal motion speed per unit time is less than the second threshold is greater than the second duration;

[0202] The duration during which the horizontal motion speed is less than the fifth threshold is greater than the third duration;

[0203] The duration during which the vertical velocity is less than the sixth threshold is greater than the fourth duration.

[0204] Optionally, the display module 1002 is specifically configured to: adjust the target window from a first state to a second state according to a preset movement rate and / or a preset size magnification rate. Further, the target window can also be adjusted from a second state to a first state according to a preset movement rate and / or a preset size magnification rate.

[0205] Optionally, the triggering conditions for moving, shrinking, or hiding the target window include at least one of the following:

[0206] The head-mounted display device is in motion-linked mode;

[0207] The head-mounted display device has activated the function of adjusting the display status of the motion trigger window;

[0208] The target window belongs to a preset application, which is an application that disables motion linkage mode.

[0209] The motion linkage mode is a mode based on the adjustment of the motion status window display status.

[0210] The head-mounted display device in this application embodiment can be a mobile electronic device. For example, the mobile electronic device can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., and this application embodiment does not specifically limit it.

[0211] The head-mounted display device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0212] The head-mounted display device provided in this application embodiment can achieve... Figure 3 The various processes implemented by the head-mounted display device in the method embodiment will not be described again here to avoid repetition.

[0213] Optional, refer to Figure 11 This application also provides a head-mounted display device 1100, including a memory 1101, a processor 1102, and a program or instructions stored in the memory 1101 and executable on the processor 1102. When the program or instructions are executed by the processor 1102, they implement the various processes of the above-described window display method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0214] Figure 12 A schematic diagram of the hardware structure of an electronic device for implementing the various embodiments of this application.

[0215] The electronic device 1200 includes, but is not limited to, components such as: radio frequency unit 1201, network module 1202, audio output unit 1203, input unit 1204, sensor 1205, display unit 1206, user input unit 1207, interface unit 1208, memory 1209, and processor 1210.

[0216] Those skilled in the art will understand that the electronic device 1200 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1210 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 12 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0217] The processor 1210 is used to acquire the user's motion state, including the direction of motion; and to move, shrink, or hide the target window in the display interface of the head-mounted display device according to the motion state, so as to reduce obstruction of the user's field of vision.

[0218] Optionally, the direction of movement includes the vertical direction;

[0219] As the user moves along the vertical direction, the target window can be moved, collapsed, or hidden.

[0220] Optionally, the movement direction includes a vertical direction and a horizontal direction; the vertical direction is perpendicular to the horizontal direction, and the processor 1210 is specifically used to move, shrink, or hide the target window when the angle between the user's movement direction and the horizontal direction is greater than 25°.

[0221] Optionally, the direction of motion includes the vertical direction, and the motion state also includes the vertical motion acceleration of the user moving along the vertical direction. The user's motion state is obtained by the accelerometer of the inertial measurement module of the head-mounted display device. The processor 1210 is specifically used to move, shrink, or hide the target window when it is confirmed through the motion state that the user is in at least one of the following scenarios: going up or down stairs, standing up, or sitting down.

[0222] Optionally, the direction of motion includes a horizontal direction; the processor 1210 is specifically used for:

[0223] As the user moves along the horizontal direction, the relative displacement between the user and the target object in the real scene is confirmed by the camera of the head-mounted display device.

[0224] When the user moves relative to the target object, the target window is moved, collapsed, or hidden.

[0225] The target window continues to be displayed while the user remains stationary relative to the target object.

[0226] Optionally, the movement of the user relative to the target object includes: within a unit time range, the relative displacement between the user and the target object is greater than a first threshold.

[0227] The condition that the user is stationary relative to the target object includes: within a unit time range, the relative displacement between the user and the target object is less than the first threshold.

[0228] Optionally, the direction of motion includes a horizontal direction; the motion state also includes the horizontal velocity and / or horizontal acceleration of the user moving along the horizontal direction, and the processor 1210 is specifically used for:

[0229] The motion state includes the horizontal motion speed. If the change in the horizontal motion speed per unit time is greater than a second threshold, it is determined that the user is moving along the horizontal direction, and the target window is moved, collapsed, or hidden. If the change in the horizontal motion speed per unit time is less than the second threshold, it is determined that the user is not moving along the horizontal direction, and the target window is maintained in a first state.

[0230] The motion state includes the horizontal motion acceleration. If the horizontal motion acceleration is greater than a third threshold, it is determined that the user is moving along the horizontal direction, and the target window is moved, collapsed, or hidden. If the horizontal motion acceleration is less than the third threshold, the target window is maintained in a first state.

[0231] The motion state includes the horizontal motion speed, which includes the speed in the direction of motion and the speed in the normal direction of motion. If the change in the speed in the normal direction of motion per unit time is greater than a fourth threshold, it is determined that the user is moving along the horizontal direction, and the target window is moved, collapsed, or hidden. If the change in the speed in the normal direction of motion per unit time is less than the fourth threshold and the change in the speed in the direction of motion per unit time is less than a fifth preset value, the target window is maintained in a first state.

[0232] The first state refers to the display state of the target window before it is moved, collapsed, or hidden.

[0233] Optionally, the processor 1210 is further configured to: activate the visual-assisted positioning function of the head-mounted display device to confirm the relative displacement between the user and the target object when the user moves along the horizontal direction.

[0234] Optionally, the processor 1210 is specifically used for:

[0235] At least two frames of scene images, including the target object, are acquired through the camera of the head-mounted display device;

[0236] Feature points are extracted from each frame of the image;

[0237] Calculate the depth information of the feature points;

[0238] Generate a point cloud matrix of feature points corresponding to each frame of the scene image based on the depth information of the feature points;

[0239] Calculate the displacement change information of the feature points corresponding to the target object based on the point cloud matrix of the feature points corresponding to the at least two frames of scene images;

[0240] The relative displacement between the user and the target object in the real-world scene is determined based on the displacement change information.

[0241] Optionally, the processor 1210 is specifically configured to: obtain the direction of motion through the inertial measurement module of the head-mounted display device.

[0242] Optionally, the motion state further includes motion speed; if the motion speed is greater than a first speed threshold, the target window is moved, collapsed, or hidden; and / or,

[0243] If the movement speed is less than a second speed threshold and the duration of the state where the movement speed is less than the second speed threshold is greater than a first time threshold, the target window is adjusted from the second state to the first state; if the movement speed is less than the second speed threshold and the duration of the state where the movement speed is less than the second speed threshold is less than the first time threshold, the target window is maintained in the second state.

[0244] The first state is the display state before the target window is moved, shrunk, or hidden, and the second state is the display state after the target window is moved, shrunk, or hidden.

[0245] Optionally, the target window includes a window located at the center of the display interface of the head-mounted display device; and / or,

[0246] The processor 1210 is specifically used to move the window located at the center of the display interface toward the edge of the display interface.

[0247] Optionally, the processor 1210 is specifically configured to: respond to a first input and adjust the target window from a second state to a first state; the first input includes at least one of gesture input, voice input, touch input, eye-tracking input, and posture input in response to head posture changes; and / or, determine that the user has stopped moving based on the motion state and adjust the target window from the second state to the first state;

[0248] The first state is the display state before the target window is moved, shrunk, or hidden, and the second state is the display state after the target window is moved, shrunk, or hidden.

[0249] Optionally, the motion state further includes the vertical motion acceleration of the user moving along the vertical direction and the horizontal motion acceleration of the user moving along the horizontal direction, and the processor 1210 is specifically used for:

[0250] The user is determined to stop moving if at least one of the following conditions is met:

[0251] The duration during which the horizontal motion acceleration is less than the third threshold is greater than the first duration;

[0252] The duration during which the change in the horizontal motion speed per unit time is less than the second threshold is greater than the second duration;

[0253] The duration during which the horizontal motion speed is less than the fifth threshold is greater than the third duration;

[0254] The duration during which the vertical velocity is less than the sixth threshold is greater than the fourth duration.

[0255] Optionally, the processor 1210 is specifically configured to: adjust the target window from a first state to a second state according to a preset movement rate and / or a preset size zoom rate. Further, the processor can also adjust the target window from a second state to a first state according to a preset movement rate and / or a preset size zoom rate.

[0256] Optionally, the triggering conditions for moving, shrinking, or hiding the target window include at least one of the following:

[0257] The head-mounted display device is in motion-linked mode;

[0258] The head-mounted display device has activated the function of adjusting the display status of the motion trigger window;

[0259] The target window belongs to a preset application, which is an application that disables motion linkage mode.

[0260] The motion linkage mode is a mode based on the adjustment of the motion status window display status.

[0261] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described window display method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0262] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0263] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above window display method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0264] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0265] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0266] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0267] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A window display method, applied to a head-mounted display device, characterized in that, include: Acquire the user's motion state, including the direction of motion; The target window in the display interface of the head-mounted display device can be moved, shrunk, or hidden according to the motion state to reduce obstruction of the user's field of vision.

2. The method according to claim 1, characterized in that, The direction of movement includes the vertical direction; As the user moves along the vertical direction, the target window can be moved, collapsed, or hidden.

3. The method according to claim 2, characterized in that, The movement direction includes a vertical direction and a horizontal direction; the vertical direction is perpendicular to the horizontal direction, and the target window is moved, collapsed, or hidden when the angle between the user's movement direction and the horizontal direction is greater than 25°.

4. The method according to claim 1, characterized in that, The direction of motion includes the vertical direction, and the motion state also includes the vertical acceleration of the user moving along the vertical direction. The user's motion state is obtained by the accelerometer of the inertial measurement module of the head-mounted display device. If the user is confirmed to be in at least one of the following scenarios through the motion state: going up or down stairs, standing up, or sitting down, the target window is moved, collapsed, or hidden.

5. The method according to claim 1, characterized in that, The direction of movement includes a horizontal direction; the method further includes: As the user moves along the horizontal direction, the relative displacement between the user and the target object in the real scene is confirmed by the camera of the head-mounted display device. When the user moves relative to the target object, the target window is moved, collapsed, or hidden. The target window continues to be displayed while the user remains stationary relative to the target object.

6. The method according to claim 5, characterized in that, The movement of the user relative to the target object includes: within a unit time range, the relative displacement between the user and the target object is greater than a first threshold. The condition that the user is stationary relative to the target object includes: within a unit time range, the relative displacement between the user and the target object is less than the first threshold.

7. The method according to claim 1, characterized in that, The direction of movement includes the horizontal direction; the state of movement also includes the user's horizontal movement speed and / or horizontal movement acceleration along the horizontal direction, and moving, shrinking, or hiding the target window in the display interface of the head-mounted display device according to the state of movement includes: The motion state includes the horizontal motion speed. If the change in the horizontal motion speed per unit time is greater than a second threshold, it is determined that the user is moving along the horizontal direction, and the target window is moved, collapsed, or hidden. If the change in the horizontal motion speed per unit time is less than the second threshold, it is determined that the user is not moving along the horizontal direction, and the target window remains in a first state; and / or, The motion state includes the horizontal motion acceleration. If the horizontal motion acceleration is greater than a third threshold, the user is determined to be moving along the horizontal direction, and the target window is moved, collapsed, or hidden. If the horizontal motion acceleration is less than the third threshold, the target window is maintained in a first state; and / or, The motion state includes the horizontal motion speed, which includes the speed in the direction of motion and the speed in the normal direction of motion. If the change in the speed in the normal direction of motion per unit time is greater than a fourth threshold, it is determined that the user is moving along the horizontal direction, and the target window is moved, collapsed, or hidden. If the change in the speed in the normal direction of motion per unit time is less than the fourth threshold and the change in the speed in the direction of motion per unit time is less than a fifth preset value, the target window is maintained in a first state. The first state refers to the display state of the target window before it is moved, collapsed, or hidden.

8. The method according to claim 5, characterized in that, The method further includes: When the user moves along the horizontal direction, the visual-assisted positioning function of the head-mounted display device is activated to confirm the relative displacement between the user and the target object.

9. The method according to any one of claims 1 to 8, characterized in that, The motion state also includes motion speed; if the motion speed is greater than a first speed threshold, the target window is moved, collapsed, or hidden; and / or, If the movement speed is less than a second speed threshold and the duration of the state where the movement speed is less than the second speed threshold is greater than a first time threshold, the target window is adjusted from the second state to the first state; if the movement speed is less than the second speed threshold and the duration of the state where the movement speed is less than the second speed threshold is less than the first time threshold, the target window is maintained in the second state. The first state is the display state before the target window is moved, shrunk, or hidden, and the second state is the display state after the target window is moved, shrunk, or hidden.

10. The method according to any one of claims 1 to 8, characterized in that, In response to a first input, the target window is adjusted from a second state to a first state; the first input includes at least one of gesture input, voice input, touch input, eye-tracking input, and posture input in response to head posture changes; and / or, Based on the motion state, the user is determined to stop moving, and the target window is adjusted from the second state to the first state. The first state is the display state before the target window is moved, shrunk, or hidden, and the second state is the display state after the target window is moved, shrunk, or hidden.

11. The method according to claim 10, characterized in that, The motion state also includes the vertical acceleration of the user moving in the vertical direction and the horizontal acceleration of the user moving in the horizontal direction. Determining that the user stops moving based on the motion state includes: The user is determined to stop moving if at least one of the following conditions is met: The duration during which the horizontal motion acceleration is less than the third threshold is greater than the first duration; The duration during which the change in the horizontal motion speed per unit time is less than the second threshold is greater than the second duration; The duration during which the horizontal motion speed is less than the fifth threshold is greater than the third duration; The duration during which the vertical velocity is less than the sixth threshold is greater than the fourth duration.

12. A head-mounted display device, characterized in that, include: An inertial measurement module is used to acquire the user's motion state, including the direction of motion; The display module is used to move, shrink, or hide the target window in the display interface of the head-mounted display device according to the motion state, so as to reduce obstruction of the user's field of vision.

13. A head-mounted display device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the window display method as described in any one of claims 1 to 11.

Images