Window display method and device, head-mounted display equipment and storage medium

Abstract

The embodiment of the invention provides a window display method and device, head-mounted display equipment and a storage medium, and the method comprises the steps: determining a currently running application program of the head-mounted display equipment when the head-mounted display equipment is in a multi-degree-of-freedom mode; if the application program is a target program, the multi-degree-of-freedom mode is switched to a main window fixed mode, in the main window fixed mode, a main display window is displayed in a field angle of the head-mounted display device, and the window displayed in the field angle does not change along with the posture change of the head-mounted display device, and in the multi-degree-of-freedom mode, the main display window is displayed in the field angle of the head-mounted display device. The display window in the field angle is changed along with the change of the posture of the head-mounted display equipment. The application program is associated with the operation mode (namely the multi-degree-of-freedom mode and the main window fixed mode) of the head-mounted display equipment, so that the display window displayed in the field angle is determined according to the operation mode of the head-mounted display equipment and the currently operated application program; and determining whether the displayed display window is changed along with the change of the posture of the head-mounted display equipment, so that the intelligence of displaying the plurality of display windows in the field angle of the head-mounted display equipment is improved.

Description

Technical Field

[0001] The present invention relates to the field of display technology, and in particular to a window display method, device, head-mounted display device and storage medium. Background Art

[0002] With the development of smart wearable technology, head-mounted display devices such as augmented reality devices and virtual reality devices have entered the lives of users. At present, head-mounted display devices can display multiple display windows at the same time, but due to the limited field of view of the head-mounted display device, these multiple display windows cannot be displayed at the same time within the field of view of the head-mounted display device. Therefore, how to control the intelligent display of multiple display windows within the field of view of the head-mounted display device is an urgent problem to be solved. Summary of the invention

[0003] Embodiments of the present invention provide a window display method, apparatus, head-mounted display device, and storage medium, aiming to improve the intelligence of displaying multiple display windows within the field of view of the head-mounted display device.

[0004] In a first aspect, an embodiment of the present invention provides a window display method, which is applied to a head-mounted display device, and the method includes:

[0005] When the head mounted display device is in a multi-degree-of-freedom mode, determining an application currently running on the head mounted display device;

[0006] If the application is a target program, it switches from the multi-degree-of-freedom mode to the main window fixed mode, wherein in the main window fixed mode, the main display window is displayed within the field of view of the head-mounted display device, and the window displayed within the field of view does not change with the change of the posture of the head-mounted display device. In the multi-degree-of-freedom mode, the window displayed within the field of view changes with the change of the posture of the head-mounted display device.

[0007] In a second aspect, an embodiment of the present invention provides a window display method, which is applied to a head-mounted display device, and the method includes:

[0008] A running program determination module, used for determining an application currently running on the head mounted display device when the head mounted display device is in a multi-degree-of-freedom mode;

[0009] A mode switching module is used to switch from the multi-degree-of-freedom mode to the main window fixed mode if the application is a target program, wherein in the main window fixed mode, the main display window is displayed within the field of view of the head-mounted display device, and the window displayed within the field of view does not change with the change of the posture of the head-mounted display device, and in the multi-degree-of-freedom mode, the window displayed within the field of view changes with the change of the posture of the head-mounted display device.

[0010] In the third aspect, an embodiment of the invention further provides a head-mounted display device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the window display method described in the first aspect is realized.

[0011] In a fourth aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the window display method as described in the first aspect.

[0012] The embodiment of the present invention provides a window display method, device, head-mounted display device and storage medium. In the embodiment of the present application, when the head-mounted display device is in a multi-degree-of-freedom mode, the application currently running on the head-mounted display device is determined; if the application is a target program, the multi-degree-of-freedom mode is switched to the main window fixed mode, wherein in the main window fixed mode, the main display window is displayed within the field of view of the head-mounted display device, and the window displayed within the field of view does not change with the change of the posture of the head-mounted display device, and in the multi-degree-of-freedom mode, the window displayed within the field of view changes with the change of the posture of the head-mounted display device. In this embodiment, by associating the application with the operating mode of the head-mounted display device (i.e., the multi-degree-of-freedom mode and the main window fixed mode), it is realized that according to the operating mode of the head-mounted display device and the currently running application, it is determined which display window is displayed within the field of view of the head-mounted display device, and whether the display window displayed within the field of view changes with the change of the posture of the head-mounted display device, thereby improving the intelligence of displaying multiple display windows within the field of view of the head-mounted display device. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 is a schematic structural block diagram of a head mounted display device provided by an embodiment of the present invention;

[0015] Figure 2 It is a flowchart of a window display method provided by an embodiment of the present invention;

[0016] Figure 3a is a reference schematic diagram of a multi-window display in an embodiment of the present invention;

[0017] Figure 3b is another reference schematic diagram of multi-window display in an embodiment of the present invention;

[0018] Figure 4 It is a schematic structural block diagram of a window display device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0021] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0022] The embodiments of the present invention provide a window display method, device, head mounted display device and storage medium. The embodiments of the present invention provide a technical solution.

[0023] In the embodiment of the present invention, the head-mounted display device may be AR (Augmented Reality) glasses, an AR helmet, VR (Virtual Reality) glasses, etc.

[0024] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0025] Figure 1 The present invention is a schematic structural block diagram of a head mounted display device provided by an embodiment of the present invention.

[0026] like Figure 1 As shown, the head mounted display device 100 includes a processor 101 and a memory 102, and the processor 101 and the memory 102 are connected via a bus 103, such as an I2C (Inter-integrated Circuit) bus.

[0027] Specifically, the processor 101 is used to provide computing and control capabilities to support the operation of the entire head-mounted display device 100. The processor 101 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0028] Specifically, the memory 102 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0029] Those skilled in the art will understand that Figure 1 The structure shown in the figure is merely a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the head-mounted display device to which the embodiment of the present invention is applied. The specific head-mounted display device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0030] The processor 101 is used to run a computer program stored in the memory 102. The computer program can be set in a head-mounted display device. When the head-mounted display device executes the computer program, it implements any one of the window display methods provided in the embodiments of the present invention.

[0031] In one embodiment, the processor is used to run a computer program stored in the memory, and implements the following steps when executing the computer program:

[0032] When the head mounted display device is in a multi-degree-of-freedom mode, determining an application currently running on the head mounted display device;

[0033] If the application is a target program, it switches from the multi-degree-of-freedom mode to the main window fixed mode, wherein in the main window fixed mode, the main display window is displayed within the field of view of the head-mounted display device, and the window displayed within the field of view does not change with the change of the posture of the head-mounted display device. In the multi-degree-of-freedom mode, the window displayed within the field of view changes with the change of the posture of the head-mounted display device.

[0034] In one embodiment, the processor is further configured to implement the following steps when executing the computer program:

[0035] In response to the head-mounted display device receiving a first control instruction in a main window fixed mode, the display content of the main display window within the field of view angle in the main window fixed mode is controlled according to the first control instruction, wherein the first control instruction includes an instruction triggered by a touch component on the head-mounted display device body or a control device connected to the head-mounted display device.

[0036] In one embodiment, if the application is a target program, after the step of switching from the multi-free mode to the main window fixed mode, the processor is further configured to implement the following steps when executing the computer program:

[0037] If it is detected that the application program exits the foreground running state, the head mounted display device is controlled to switch from the main window fixed mode to the multi-degree-of-freedom mode.

[0038] In one embodiment, the processor is further configured to implement the following steps when executing the computer program:

[0039] In response to the head-mounted display device receiving a second control instruction in the multi-degree-of-freedom mode, the display content within the field of view angle in the multi-degree-of-freedom mode is controlled according to the second control instruction, wherein the second control instruction includes an instruction triggered according to the user's gaze point data collected by the head-mounted display device.

[0040] In one embodiment, the processor is further configured to implement the following steps when executing the computer program:

[0041] In the multi-degree-of-freedom mode, if the window displayed within the field of view of the head-mounted display device is a secondary display window, when a third control instruction is received, the window displayed within the field of view is controlled to switch from the secondary display window to the main display window.

[0042] In one embodiment, the processor is further configured to implement the following steps when executing the computer program:

[0043] When switching the display window within the viewing angle, if the switched display window is completely displayed within the viewing angle, outputting a prompt sound;

[0044] and / or

[0045] During the process of switching the display window within the viewing angle, a preset animation is displayed at the edge of the viewing angle.

[0046] In one embodiment, if the application is a target program, the step of switching from the multi-degree-of-freedom mode to the main window fixed mode comprises:

[0047] If the application is a target program, the current display coordinates of each display window of the multi-degree-of-freedom mode are initialized, and a sensor for detecting the device posture in the head-mounted display device is turned off to switch the multi-degree-of-freedom mode to a main window fixed mode.

[0048] It should be noted that technicians in the relevant field can clearly understand that for the convenience and brevity of description, the specific working process of the head-mounted display device described above can refer to the corresponding process in the following window display method embodiment, and will not be repeated here.

[0049] The following will be combined Figure 1 The head mounted display device in the embodiment of the present invention is described in detail about the window display method provided by the embodiment of the present invention. It should be noted that Figure 1 The head mounted display device is only used to explain the window display method provided in the embodiment of the present invention, but does not constitute a limitation on the application scenario of the window display method provided in the embodiment of the present invention.

[0050] See also Figure 2 , Figure 2 It is a flowchart of a window display method provided by an embodiment of the present invention.

[0051] like Figure 2 As shown, the window display method is applied to a head mounted display device, and the window display method includes step S101 and step S102.

[0052] Step S101: When the head mounted display device is in a multi-degree-of-freedom mode, determine an application currently running on the head mounted display device.

[0053] In this embodiment, the head-mounted display device includes two operating modes, one is a multi-degree-of-freedom mode and the other is a main window fixed mode. These two operating modes are mainly used to describe the operating mode of the window displayed within the field of view of the head-mounted display device. Exemplarily, the multi-degree-of-freedom mode may be a 3DOF (Degree of Freedom) or 6DOF mode. Among them, the 3DOF head-mounted display device can track the movement of the user's head in three degrees of freedom, usually yaw, pitch and roll. Yaw is a rotation around the vertical axis (z-axis), just like the left and right rotation of the user's head; pitch is a rotation around the horizontal axis (y-axis), similar to the user's nodding action; roll is a rotation around the longitudinal axis (x-axis), such as the user's head tilting to one side. By tracking these three degrees of freedom, the head-mounted display device can adjust the display angle of the content displayed in the field of view according to the movement of the user's head, so that the displayed content maintains a relatively stable positional relationship with the user's perspective to a certain extent. In VR devices and AR devices, 3DOF tracking is mainly used for simple head posture tracking. Taking VR devices as an example, it allows users to change their perspective by rotating their heads, so that the virtual scene changes accordingly with the yaw, pitch and roll movements of the head, providing a basic immersive experience.

[0054] 6DOF head-mounted display devices can not only track the three rotational degrees of freedom (yaw, pitch, roll) of the head, but also track three translational degrees of freedom, namely the movement of front and back (x-axis), left and right (y-axis), and up and down (z-axis), which enables the head-mounted display device to more accurately locate the position and posture of the user's head in three-dimensional space. In VR devices and AR devices, 6DOF technology can achieve more accurate spatial positioning and tracking. Taking games in VR devices as an example, players can move, jump, dodge, etc. freely in the virtual space, because the VR device can perceive the position and posture changes of the player's head and the handle (if any) in real time. In AR devices, 6DOF can make virtual objects more accurately superimposed on real scenes. For example, in architectural design displays, users can freely shuttle inside and outside the building model, and observe from different angles. The position and display angle of the building model will be accurately adjusted according to the user's position and posture. It is understandable that when a user wears a head-mounted display device, the position and posture of the user's head will determine the posture of the head-mounted display device.

[0055] When the head mounted display device is in the multi-degree-of-freedom mode, the head mounted display device determines the application currently running, wherein the application currently running on the head mounted display device is an application running in the foreground of the head mounted display device.

[0056] In one embodiment, the head mounted display device can determine whether it is in the multi-degree-of-freedom mode by the current mode identifier, and the mode identifier of the multi-degree-of-freedom mode is different from the mode identifier of the main window fixed mode.

[0057] Step S102: If the application is a target program, switching from the multi-degree-of-freedom mode to the main window fixed mode, wherein in the main window fixed mode, the main display window is displayed within the field of view of the head-mounted display device, and the window displayed within the field of view does not change with the change of the posture of the head-mounted display device. In the multi-degree-of-freedom mode, the window displayed within the field of view changes with the change of the posture of the head-mounted display device.

[0058] After the head mounted device determines the currently running application, it determines whether the application is the target program. If the current application is determined to be the target program, the head mounted display device switches from the multi-degree-of-freedom mode to the main window fixed mode. Among them, the target program is an application running in the main window fixed mode, and the specific applications that move in the main window fixed mode are pre-set. Exemplarily, the program identifier of the target program and the mode identifier of the main window fixed mode can be associated to obtain the association relationship between the target program and the main window fixed mode, so that the head mounted display device can determine whether the currently running application is the target program through the association relationship, wherein the program identifier is a unique identity identifier that can be used to identify the application. In this embodiment, the expression form of the program identifier is not specifically limited. In the main window fixed mode, the display window in the field of view of the head mounted display device does not change with the change of the posture of the head mounted display device. In the main window fixed mode, the main display window is displayed in the field of view of the head mounted display device; in the multi-degree-of-freedom mode, the display window in the field of view changes with the change of the posture of the head mounted display device.

[0059] Field of View (FOV) refers to the angle measurement of the range that can be observed by the user in a head-mounted display device. It represents the angle range of the left and right and up and down directions that the user can see through the head-mounted display device, usually described in horizontal and vertical angles. For example, a head-mounted display device has a horizontal field of view of 90 degrees, which means that the angle range that the user can see to the left or right through the head-mounted display device is 90 degrees.

[0060] In one embodiment, the multi-degree-of-freedom mode can be set as the default operating mode, that is, the head-mounted display device will be in the main window fixed mode only when the head-mounted display device runs the target program, and in other cases, the head-mounted display device will be in the multi-degree-of-freedom mode.

[0061] In a head-mounted display device, there may be multiple display windows, each of which may display different content. However, due to the limited field of view of the head-mounted display device, only one display window may be displayed at a time. Among the multiple display windows, one is the main display window, and the rest are sub-display windows. In a head-mounted display device, there is no limitation on the arrangement of the multiple display windows. For example, the multiple display windows may be arranged horizontally or vertically, and so on. The specific display window that is the main display window may be set by the user, and no specific limitation is made here. For example, referring to FIG3 , in the head-mounted display device of FIG3 , there are three display windows, namely display window A, display window B, and display window C. In FIG3 , the three display windows are arranged horizontally, among which the middle display window A may be set as the main display window, and the remaining two display windows B and display window C may be set as sub-display windows. Figure 3a In the example, display window A is displayed within the field of view of the head mounted display device. Figure 3b , display window B is displayed within the field of view of the head-mounted display device. Exemplarily, when the three display windows are arranged horizontally and the head-mounted display device is currently in the multi-degree-of-freedom mode, if the head-mounted display device displays the main display window arranged in the middle in the field of view, when the head-mounted display device detects that the head of the user is turned to the left, it indicates that the posture of the head-mounted display device is also turned to the left. At this time, the secondary display window corresponding to the left turn will be displayed within the field of view.

[0062] It is understandable that the areas of multiple display windows in the head-mounted display device may be the same or different, and the area of ​​each display window may be less than or equal to the area corresponding to the field of view angle.

[0063] In one embodiment, the head-mounted display device can detect its device posture in the main window fixed mode, but does not respond to changes in the device posture, that is, the head-mounted display device can obtain sensor data from a sensor that detects the device posture, but does not process or respond to the sensor data, thereby maintaining the main window fixed mode.

[0064] This embodiment determines the application currently running on the head-mounted display device when the head-mounted display device is in the multi-degree-of-freedom mode; if the application is a target program, the multi-degree-of-freedom mode is switched to the main window fixed mode, wherein in the main window fixed mode, the main display window is displayed within the field of view of the head-mounted display device, and the window displayed within the field of view does not change with the change of the posture of the head-mounted display device, and in the multi-degree-of-freedom mode, the window displayed within the field of view changes with the change of the posture of the head-mounted display device. This embodiment associates the application with the operating mode of the head-mounted display device (i.e., the multi-degree-of-freedom mode and the main window fixed mode), thereby realizing the determination of which display window to display within the field of view of the head-mounted display device according to the operating mode of the head-mounted display device and the currently running application, and the determination of whether the display window displayed within the field of view changes with the change of the posture of the head-mounted display device, thereby improving the intelligence of displaying multiple display windows within the field of view of the head-mounted display device.

[0065] In one embodiment, the step S102 includes:

[0066] If the application is a target program, the current display coordinates of each display window of the multi-degree-of-freedom mode are initialized, and a sensor for detecting the device posture in the head-mounted display device is turned off to switch the multi-degree-of-freedom mode to a main window fixed mode.

[0067] In one embodiment, if the head-mounted display device determines that the application currently running is the target program, the head-mounted display device initializes the display coordinates of each display window in the multi-degree-of-freedom mode, and turns off the sensor for detecting the device posture to switch the multi-degree-of-freedom mode to the main window fixed mode. It can be understood that in the head-mounted display device, each display window has a corresponding initial display coordinate, and the initial display coordinate can be used to determine the display position of each display window in the initial state. For example, the initial display coordinate of the main display window can be used to determine that the initial state of the main display window is displayed in the field of view angle, and the initial display coordinate of the auxiliary display window can be determined according to the initial display coordinate of the main display window, the arrangement of multiple display windows, and the size of each display window. Each display window has at least one initial display coordinate. By initializing the current display coordinates of each display window in the multi-degree-of-freedom mode, the current display coordinates of the auxiliary display window and the main display window can be changed to the initial display coordinates, so that the main display window is displayed in the field of view angle of the head-mounted display device, which is convenient for the user wearing the head-mounted display device to view the content in the main display window.

[0068] A sensor for detecting the posture of the device is provided in the head-mounted display device, and the current posture of the head-mounted display device can be determined by the sensor for detecting the posture of the device. Among them, the sensor for detecting the posture of the device includes at least one of an IMU (Inertial Measurement Unit), a gyroscope and an accelerometer. It is understandable that when the sensor for detecting the posture of the device in the head-mounted display device is turned off, the head-mounted device cannot detect changes in its posture, so the main display window can be always displayed within the field of view, thereby being in the main window fixed mode. The main window fixed mode can be understood as a mode in which the main display window is displayed under 0DOF.

[0069] This embodiment initializes the current display coordinates of each display window in the multi-degree-of-freedom mode and turns off the sensor for detecting the device posture in the head-mounted display device, thereby switching the multi-degree-of-freedom mode to the main window fixed mode, thereby avoiding the sensor for detecting the device posture in the main window fixed mode from being in operation, thereby reducing the power consumption of the head-mounted display device.

[0070] In one embodiment, the method further includes: in response to the head-mounted display device receiving a first control instruction in a main window fixed mode, controlling the display content of the main display window within the field of view angle in the main window fixed mode according to the first control instruction, wherein the first control instruction includes an instruction triggered by a touch component on the head-mounted display device body or a control device connected to the head-mounted display device.

[0071] In one embodiment, after the head-mounted display device switches from the multi-degree-of-freedom mode to the main window fixed mode, the head-mounted display device detects whether a first control instruction is received, wherein the first control instruction includes an instruction triggered by the user through a touch component on the head-mounted display device body or a control device connected to the head-mounted display. The touch component can be a button and / or a touch bar on the head-mounted display device. For example, when the head-mounted display device is an AR glasses, the touch component can be a touch bar on the temple of the AR glasses. The control device can be a finger ring, a bracelet or a mobile phone, and the control device can establish a communication connection with the head-mounted display device through WiFi or Bluetooth.

[0072] When the head mounted display device receives the first control instruction in the main window fixed mode, the head mounted display device controls the display content of the main display window within the field of view in the main window fixed mode according to the first control instruction. The first control instruction may be a delete instruction, an edit instruction, a switch instruction, etc. for the display content.

[0073] It is understandable that when the head-mounted display device is in the main window fixed mode, since the head-mounted display device will not process the sensor data of the sensor used to detect the device posture (the head-mounted display device will not respond to the user's head movement data), or the sensor used to detect the device posture is in the off state (the head-mounted display device cannot obtain the user's head movement data), the head-mounted display cannot trigger control instructions through head movement control. This embodiment avoids the situation where the user triggers control instructions through head movement control in the main window fixed mode, resulting in the head-mounted display device being unable to respond to operation instructions, thereby improving the response success rate of the head-mounted display device to operation instructions in the main window fixed mode.

[0074] In one embodiment, after step S102, the method further includes:

[0075] If it is detected that the application program exits the foreground running state, the head mounted display device is controlled to switch from the main window fixed mode to the multi-degree-of-freedom mode.

[0076] In one embodiment, if the head-mounted display device detects that the currently running application exits the foreground running state, the head-mounted display device controls it to switch from the main window fixed mode to the multi-degree-of-freedom mode. Among them, the application exiting the foreground running state includes both the application switching from the foreground running state to the background running state and the application switching from the foreground running state to the closed state. It can be understood that the application exiting the foreground running state is the target program. When the head-mounted display device switches from the main window fixed mode to the multi-degree-of-freedom mode, the head-mounted display device will display the display window last displayed in the multi-degree-of-freedom mode within the field of view angle or display the main display window within the field of view angle.

[0077] For example, if the head mounted display device was in the multi-degree-of-freedom mode last time, and the B sub-display window was displayed in the field of view, and then the multi-degree-of-freedom mode was switched to the main window fixed mode. When the head mounted display device is switched from the main window fixed mode to the multi-degree-of-freedom mode, the head mounted display device will display the B sub-display window in its field of view.

[0078] In this embodiment, after the head-mounted display device detects that the application has exited the foreground, the head-mounted display device switches from the main window fixed mode back to the multi-degree-of-freedom mode, so that the user can control the head-mounted display device more flexibly.

[0079] In one embodiment, the method further comprises:

[0080] In response to the head-mounted display device receiving a second control instruction in the multi-degree-of-freedom mode, the display content within the field of view angle in the multi-degree-of-freedom mode is controlled according to the second control instruction, wherein the second control instruction includes an instruction triggered according to the user's gaze point data collected by the head-mounted display device.

[0081] When the head mounted display device receives the second operation instruction in the multi-degree-of-freedom mode, the head mounted display device controls the display content within the field of view in the multi-degree-of-freedom mode according to the second control instruction, wherein the second control instruction includes a command triggered according to the head movement data of the user collected by the head mounted display device, and the head mounted display device can collect the head movement data of the user through the IMU. Exemplarily, if the head mounted display device is in the multi-degree-of-freedom mode, if the virtual cursor falls on a certain icon in the display window displayed in the current field of view, the head mounted display device determines the distance and direction of the user's head movement according to the collected user's head movement data, generates a second control instruction according to the movement distance and direction, and controls the movement of the virtual cursor according to the second control instruction, thereby controlling the movement of the icon, the movement distance and direction of the virtual cursor are determined by the distance and direction of the user's head movement, and the movement trajectory of the virtual cursor is the same as the movement trajectory of the icon.

[0082] It is understandable that the first control instruction and the second control instruction may include instructions triggered by the head mounted display device through collecting user voice; when the head mounted display device is equipped with a camera, the first control instruction and the second control instruction may also include instructions triggered by the head mounted display device collecting user gestures through its camera. In another embodiment, the second control instruction may also include instructions triggered by a touch component on the head mounted display device body or a control device connected to the head mounted display device.

[0083] This embodiment enables the head-mounted display device to respond to control instructions triggered by user head movement data when the head-mounted display device is in a multi-degree-of-freedom mode, thereby providing a head movement control method for the head-mounted display device in the multi-degree-of-freedom mode, thereby achieving the head-mounted display device without the need for an external control device or the user manually touching the touch components on the head-mounted display device body to trigger control instructions, thereby improving the triggering efficiency of control instructions in the head-mounted display device.

[0084] In one embodiment, the method further comprises:

[0085] In the multi-degree-of-freedom mode, if the window displayed within the field of view of the head-mounted display device is a secondary display window, when a third control instruction is received, the window displayed within the field of view is controlled to switch from the secondary display window to the main display window.

[0086] When the head-mounted display device is in the multi-degree-of-freedom mode, if the window displayed by the head-mounted display device within the field of view is a secondary display window, the head-mounted display detects whether it has received a third control instruction, wherein the third control instruction is a switching instruction for controlling window switching. This embodiment does not limit the triggering method of the third control instruction, as long as the command triggering method that the head-mounted display device can respond to in the multi-degree-of-freedom mode can be used to trigger the third control instruction. When the head-mounted display device receives the third control instruction, the head-mounted display can control the window displayed within the field of view to switch from the secondary display window to the main display window, that is, reset the main display window to the center of the user's field of view.

[0087] In one embodiment, when the head mounted display device receives the third operation instruction, the head mounted display device outputs a prompt message to inform the user that the third operation instruction has been received. Exemplarily, the prompt message may be a vibration message or a flashing indicator light.

[0088] In this embodiment, in the multi-degree-of-freedom mode, if the head-mounted display device wants to switch the secondary display window displayed within the field of view to the main display window, the user can trigger the window switching instruction through head movement, voice, a control device connected to the head-mounted display device, or a touch component set on the head-mounted display device body, so as to reset the main display window to the center of the user's field of view, thereby improving the convenience of triggering the control instruction of window switching in the multi-degree-of-freedom mode.

[0089] In one embodiment, the method further comprises:

[0090] When switching the display window within the viewing angle, if the switched display window is completely displayed within the viewing angle, outputting a prompt sound;

[0091] and / or

[0092] During the process of switching the display window within the viewing angle, a preset animation is displayed at the edge of the viewing angle.

[0093] In one embodiment, when the head-mounted display device switches the display window within the field of view, when the switched display window is fully displayed within the field of view, the head-mounted display device outputs a prompt sound to prompt the user that the switched display window has been fully displayed within the field of view. Switching the display window within the field of view includes two scenarios. Scenario ①: Switching the main display window within the field of view to the secondary display window, at which time the switched display window is the secondary display window; Scenario ②: Switching the secondary display window within the field of view to the main display window, at which time the switched display window is the main display window. The type of prompt sound output and the output duration of the prompt sound can be set according to specific needs, and this embodiment does not impose specific restrictions on this. For example, when the switched display window is fully displayed within the field of view, a crisp "buzzing" sound is output.

[0094] In one embodiment, when the head-mounted display device switches the display window within the field of view, the head-mounted display device displays a preset animation at the edge of the field of view. The edge of the field of view may be all around the field of view, or may be a partial area around the field of view. The preset animation may be set according to specific needs, and this embodiment does not impose specific restrictions on this. Exemplarily, the preset animation may be an animation that fades in and out, or displays a faint light and shadow effect at the edge of the field of view. In one embodiment, the theme color of the display window before switching and the display window after switching within the field of view may be the same or different. The theme color of the display window after switching and the determination of the content to be displayed in the display window after switching.

[0095] In this embodiment, when switching windows within the field of view angle, if the switched display window is completely displayed within the field of view angle, the head-mounted display will output a prompt sound; and / or in the process of switching the display window within the field of view angle, a preset animation is displayed at the edge of the field of view angle of the head-mounted display device. In this embodiment, the user is informed of the switching of the display window within the field of view angle by outputting a prompt sound and / or displaying an animation, so that the user can intuitively understand the switching process of the display window within the field of view angle and whether the display window switching is completed.

[0096] See also Figure 4 , Figure 4 It is a schematic block diagram of the structure of a window display device provided by an embodiment of the present invention.

[0097] like Figure 4 As shown, the window display device 400 is applied to a head-mounted display device, and the window display device 400 includes:

[0098] A running program determination module 410, configured to determine an application currently running on the head mounted display device when the head mounted display device is in a multi-degree-of-freedom mode;

[0099] The mode switching module 420 is used to switch from the multi-degree-of-freedom mode to the main window fixed mode if the application is a target program, wherein in the main window fixed mode, the main display window is displayed within the field of view of the head-mounted display device, and the window displayed within the field of view does not change with the change of the posture of the head-mounted display device. In the multi-degree-of-freedom mode, the window displayed within the field of view changes with the change of the posture of the head-mounted display device.

[0100] In one embodiment, the window display device 400 further includes:

[0101] A first display content control module is used to control the display content of the main display window within the field of view angle in the main window fixed mode in response to the head-mounted display device receiving a first control instruction in the main window fixed mode, according to the first control instruction, wherein the first control instruction includes an instruction triggered by a touch component on the head-mounted display device body or a control device connected to the head-mounted display device.

[0102] In one embodiment, the mode switching module 420 is further configured to control the head mounted display device to switch from the main window fixed mode to the multi-degree-of-freedom mode if it is detected that the application program exits the foreground running state.

[0103] In one embodiment, the window display device 400 further includes:

[0104] A second display content control module is used to control the display content within the field of view angle in the multi-degree-of-freedom mode in response to the head-mounted display device receiving a second control instruction in the multi-degree-of-freedom mode according to the second control instruction, wherein the second control instruction includes an instruction triggered according to the user's gaze point data collected by the head-mounted display device.

[0105] In one embodiment, the window switching module 420 is also used to, in the multi-degree-of-freedom mode, if the window displayed within the field of view of the head-mounted display device is a secondary display window, then upon receiving a third control instruction, control the window displayed within the field of view to switch from the secondary display window to the main display window.

[0106] In one embodiment, the window display device 600 further includes:

[0107] A prompt sound output module, configured to output a prompt sound when switching a display window within the viewing angle, if the switched display window is completely displayed within the viewing angle;

[0108] and / or

[0109] The animation output module is used to display a preset animation at the edge of the field of view during the process of switching the display window within the field of view.

[0110] In one embodiment, the window switching module 420 includes:

[0111] A coordinate initialization unit, configured to initialize the current display coordinates of each display window in the multi-degree-of-freedom mode if the application is a target program;

[0112] A sensor shutoff unit is used to shut off a sensor in the head mounted display device for detecting the device posture, so as to switch the multi-degree-of-freedom mode to a main window fixed mode.

[0113] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the window display device described above can refer to the corresponding process in the aforementioned window display method embodiment, and will not be repeated here.

[0114] An embodiment of the present invention further provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any window display method provided in the description of the embodiment of the present invention.

[0115] The storage medium may be an internal storage unit of the head mounted display device described in the above embodiment, such as a hard disk or memory of the head mounted display device. The storage medium may also be an external storage device of the head mounted display device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., provided on the head mounted display device.

[0116] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0117] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0118] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A window display method, characterized in that: The method is applied to a head mounted display device, and the method comprises: When the head mounted display device is in a multi-degree-of-freedom mode, determining an application currently running on the head mounted display device; If the application is a target program, it switches from the multi-degree-of-freedom mode to the main window fixed mode, wherein in the main window fixed mode, the main display window is displayed within the field of view of the head-mounted display device, and the window displayed within the field of view does not change with the change of the posture of the head-mounted display device. In the multi-degree-of-freedom mode, the window displayed within the field of view changes with the change of the posture of the head-mounted display device.

2. The window display method according to claim 1, characterized in that: The method further comprises: In response to the head-mounted display device receiving a first control instruction in a main window fixed mode, the display content of the main display window within the field of view angle in the main window fixed mode is controlled according to the first control instruction, wherein the first control instruction includes an instruction triggered by a touch component on the head-mounted display device body or a control device connected to the head-mounted display device.

3. The window display method according to claim 1, characterized in that: If the application is a target program, after the step of switching from the multi-free mode to the main window fixed mode, the method further includes: If it is detected that the application program exits the foreground running state, the head mounted display device is controlled to switch from the main window fixed mode to the multi-degree-of-freedom mode.

4. The window display method according to claim 1, characterized in that: The method further comprises: In response to the head-mounted display device receiving a second control instruction in the multi-degree-of-freedom mode, the display content within the field of view angle in the multi-degree-of-freedom mode is controlled according to the second control instruction, wherein the second control instruction includes an instruction triggered according to the user's gaze point data collected by the head-mounted display device.

5. The window display method according to claim 1, characterized in that: The method further comprises: In the multi-degree-of-freedom mode, if the window displayed within the field of view of the head-mounted display device is a secondary display window, when a third control instruction is received, the window displayed within the field of view is controlled to switch from the secondary display window to the main display window.

6. The window display method according to claim 5, characterized in that: The method further comprises: When switching the display window within the viewing angle, if the switched display window is completely displayed within the viewing angle, outputting a prompt sound; and / or During the process of switching the display window within the viewing angle, a preset animation is displayed at the edge of the viewing angle.

7. The window display method according to any one of claims 1 to 6, characterized in that: If the application is a target program, the step of switching from the multi-degree-of-freedom mode to the main window fixed mode includes: If the application is a target program, the current display coordinates of each display window of the multi-degree-of-freedom mode are initialized, and a sensor for detecting the device posture in the head-mounted display device is turned off to switch the multi-degree-of-freedom mode to a main window fixed mode.

8. A window display device, characterized in that: The window display device is applied to a head-mounted display device, and the window display device comprises: A running program determination module, used for determining an application currently running on the head mounted display device when the head mounted display device is in a multi-degree-of-freedom mode; A mode switching module is used to switch from the multi-degree-of-freedom mode to the main window fixed mode if the application is a target program, wherein in the main window fixed mode, the main display window is displayed within the field of view of the head-mounted display device, and the window displayed within the field of view does not change with the change of the posture of the head-mounted display device, and in the multi-degree-of-freedom mode, the window displayed within the field of view changes with the change of the posture of the head-mounted display device.

9. A head mounted display device, characterized in that: The head-mounted display device includes a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the window display method as described in any one of claims 1 to 7 is realized.

10. A storage medium for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the window display method according to any one of claims 1 to 7.

Images