Method, apparatus, head-mounted display device and storage medium for controlling a head-mounted display device

Abstract

Embodiments of the present application provide a method and device for controlling a head-mounted display device, the head-mounted display device and a storage medium. The method comprises: in response to collecting a trigger signal indicating that at least one physical button is triggered, determining an operation instruction corresponding to the trigger signal, the operation instruction comprising a shortcut instruction and a screen menu adjustment instruction; if the operation instruction is the shortcut instruction, performing a shortcut operation corresponding to the shortcut instruction; and if the operation instruction is the screen menu adjustment instruction, performing an adjustment operation corresponding to the screen menu adjustment instruction on a to-be-adjusted item in a screen menu currently displayed by the head-mounted display device, so that the head-mounted display device adjusts information display mode in response to the screen menu adjustment instruction.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, head-mounted display device, and storage medium for controlling a head-mounted display device. Background Technology

[0002] Head-mounted displays (HUDs) are devices worn on the head that send optical signals to the eyes to achieve virtual reality (VR), augmented reality (AR), or mixed reality (MR) effects. Unlike existing electronic devices such as smartphones, computers, and smartwatches, HUDs differ in their wearing methods and operating principles, making existing interaction methods unsuitable or not fully applicable to them. Therefore, developing interaction technologies suitable for HUDs has become a pressing issue in this field. Summary of the Invention

[0003] This application provides a method, apparatus, head-mounted display device, and storage medium for controlling a head-mounted display device.

[0004] On one hand, embodiments of this application provide a method for controlling a head-mounted display device. The method includes: in response to a trigger signal that at least one physical button has been triggered, determining an operation instruction corresponding to the trigger signal, the operation instruction including a shortcut instruction and a screen menu adjustment instruction; if the operation instruction is a shortcut instruction, then executing a shortcut operation corresponding to the shortcut instruction; if the operation instruction is a screen menu adjustment instruction, then performing an adjustment operation corresponding to the screen menu adjustment instruction for the option to be adjusted in the screen menu currently displayed on the head-mounted display device, so that the head-mounted display device adjusts the information display mode in response to the screen menu adjustment instruction.

[0005] On the other hand, embodiments of this application also provide an apparatus for controlling a head-mounted display device, comprising: a response module, configured to, in response to a trigger signal collected from at least one physical button being triggered, determine an operation instruction corresponding to the trigger signal, the operation instruction including a shortcut instruction and a screen menu adjustment instruction; a first execution module, configured to, if the operation instruction is a shortcut instruction, execute a shortcut operation corresponding to the shortcut instruction; and a second execution module, configured to, if the operation instruction is a screen menu adjustment instruction, execute an adjustment operation corresponding to the screen menu adjustment instruction for the option to be adjusted in the screen menu currently displayed on the head-mounted display device, so that the head-mounted display device adjusts the information display mode in response to the screen menu adjustment instruction.

[0006] In another aspect, embodiments of this application also provide a head-mounted display device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one computer software, and the computer software causes the processor to execute the method described in any of the preceding embodiments.

[0007] Finally, embodiments of this application also provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the embodiments.

[0008] In this embodiment, in response to a trigger signal indicating that at least one physical button has been triggered, an operation command corresponding to the trigger signal can be determined. The operation command includes shortcut commands and screen menu adjustment commands. If the operation command is a shortcut command, the corresponding shortcut operation can be executed. If the operation command is a screen menu adjustment command, the adjustment operation corresponding to the screen menu adjustment command can be executed for the adjustable option in the currently displayed screen menu of the head-mounted display device, so that the head-mounted display device adjusts the information display method in response to the screen menu adjustment command. This allows users to control the head-mounted display device to perform corresponding shortcut operations and screen menu adjustments by triggering physical buttons, without needing to run corresponding software to make the head-mounted display device perform the same operation, thus facilitating user control of the head-mounted display device. Furthermore, the embodiments of this application can centralize the adjustment of various information display modes into the screen menu. When a user wants to adjust a certain information display mode, they can find the corresponding menu option in the screen menu, select the menu option to be adjusted using a physical button, and then trigger the trigger signal corresponding to the screen menu adjustment command using the physical button. This allows the head-mounted display device to adjust the information display mode in response to the screen menu adjustment command, effectively reducing the time users spend adjusting the information display mode of the head-mounted display device and making it more convenient for users to adjust the information display mode of the head-mounted display device. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0010] Figure 1 An exemplary system architecture applicable to this application is shown;

[0011] Figure 2 A flowchart illustrating the steps of a method for controlling a head-mounted display device provided in some optional embodiments of this application;

[0012] Figure 3 Flowcharts of steps for a method of controlling a head-mounted display device provided in some alternative embodiments of this application;

[0013] Figure 4 A schematic diagram of the vertical field of view provided for some optional embodiments of this application;

[0014] Figure 5 This is a schematic diagram of the structure of a head-mounted display device provided in some optional embodiments of this application;

[0015] Figure 6 A schematic diagram of a virtual screen provided for some optional embodiments of this application;

[0016] Figure 7 Structural block diagrams of an apparatus for controlling a head-mounted display device provided in some optional embodiments of this application;

[0017] Figure 8 This is a schematic diagram of a head-mounted display device provided in some optional embodiments of this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0019] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in the embodiments of this application refers to and includes any or all possible combinations of one or more associated listed items.

[0020] It should be understood that in the description of the embodiments of this application, the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. They are only for the convenience of describing the scheme of the embodiments of this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0021] Furthermore, when an element or layer is referred to as being "on" another element or layer, "connected to," or "bonded to" another element or layer, the element or layer may be directly on the other element or layer, directly connected to, or directly bonded to the other element or layer, or there may be intermediate elements or layers. However, when an element or layer is referred to as being "directly on" another element or layer, "directly connected to," or "directly bonded to" another element or layer, there are no intermediate elements or layers.

[0022] The terms First, Second, etc., are used to describe various elements, components, regions, layers, and / or parts, but these elements, components, regions, layers, and / or parts should not be limited by these terms. These terms are used to distinguish one element, component, region, layer, and / or part from another element, component, region, layer, and / or part.

[0023] Unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. Figure 1 An exemplary system architecture 100 is shown, which can be applied to an embodiment of the method for controlling a head-mounted display device or an apparatus for controlling a head-mounted display device according to this application.

[0024] like Figure 1 As shown, the system architecture 100 may include a source device 101 and a head-mounted display device 102. The source device 101 and the head-mounted display device 102 can be connected via a network, which can include various methods such as wired, wireless communication links, or fiber optic cables. The source device 101 and the head-mounted display device 102 can interact via the network to send or receive data information, etc.

[0025] Users can use source device 101 to interact with head-mounted display device 102 via a network to receive or send data, etc. Various applications can be installed on source device 101, such as web browser applications, video playback software, and driver software for the head-mounted display device. The aforementioned head-mounted display device 102 can be an electronic device with image display capabilities, including but not limited to AR smart glasses and VR smart glasses.

[0026] In some optional embodiments, the head-mounted display device 102 and the source device 101 can be separate electronic devices. The source device 101 can acquire image data, etc., through local storage or cloud transmission, so that the source device 101 can send the image data as source data to the head-mounted display device 102. Optionally, the head-mounted display device can process the data acquired from the source device 101 or the locally stored data in response to the user's triggering of physical buttons, thereby enabling shortcut operations corresponding to the physical buttons or adjustments to the screen menu.

[0027] It should be noted that the method for controlling the head-mounted display device provided in this application embodiment is generally executed by the head-mounted display device 102, and correspondingly, the device for controlling the head-mounted display device is generally disposed in the head-mounted display device 102.

[0028] In some alternative embodiments, the head-mounted display device 102 may also be an integrated electronic device. In this case, the head-mounted display device 102 can acquire various data from the cloud or local storage and process and display the data accordingly.

[0029] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0030] See Figure 2 This application provides a method for controlling a head-mounted display device, including:

[0031] S210, in response to the acquisition of a trigger signal that at least one physical button has been triggered, determine the operation instruction corresponding to the trigger signal, the operation instruction including shortcut instructions and screen menu adjustment instructions.

[0032] Head-mounted display devices may have one, two, three, or other numbers of physical buttons. As one possible implementation, a head-mounted display device may include physical buttons such as "X," "+," and "-" that can be used to perform different functions. For example, the "X" button can be used for confirmation, while "+" and "-" can be used to switch between selected icons or menu options or adjust the parameters of the selected indicator. Of course, in addition to physical buttons such as "X," "+," and "-," head-mounted display devices may also include other physical buttons.

[0033] The trigger signal for a physical button can refer to the number of times the physical button is clicked within a predetermined time period, and the duration of each click. In some optional embodiments, a correspondence can be established between the number of times the physical button is clicked within the predetermined time period and the duration of each click, and the operation command. This allows for the determination of the correspondence between the trigger signal indicating the number of times the physical button is clicked within the predetermined time period and / or the duration of each click, and the operation command. Based on this correspondence, the operation command corresponding to the trigger signal can be determined in response to the acquisition of at least one trigger signal from a physical button.

[0034] For example, a first trigger signal can be set to indicate that the physical button "X" is clicked once within a predetermined time period, and the duration of the click is within a first time period range, corresponding to a first operation instruction; a second trigger signal can be set to indicate that the physical button "X" is clicked once within a predetermined time period, and the duration of the click is within a second time period range, corresponding to a second operation instruction. The maximum value in the first time period range can be less than the minimum value in the second time period range. Thus, when the first trigger signal is acquired, the corresponding operation instruction can be determined to be the first operation instruction, and when the second trigger signal is acquired, the corresponding operation instruction can be determined to be the second operation instruction.

[0035] In some optional embodiments, the operation instructions may include shortcut instructions and screen menu adjustment instructions. Shortcut instructions can be instructions used to instruct the execution of quick operations. Quick operations may include setting operations that adjust the functional settings of the head-mounted display device, such as adjusting the lens transparency of the head-mounted display device. Screen menu adjustment instructions can be instructions used to instruct the execution of adjustment operations on the options to be adjusted in the screen menu. The options to be adjusted in the screen menu can be the selected menu options in the screen menu, and performing an adjustment operation on the options to be adjusted can be understood as performing an operation on the options to be adjusted. For example, adjusting the parameters in the options to be adjusted, performing a confirmation operation on the options to be adjusted, and switching the currently adjusted options to another menu option, etc. It is understood that adjusting the options to be adjusted in the screen menu can change the way information is displayed on the head-mounted display device. For example, it can change the size of the virtual screen of the head-mounted display device, the shortcut instructions corresponding to the quick operations of the head-mounted display device, etc.

[0036] In some optional embodiments, a correspondence can be set between the display state of the screen menu when each physical button is triggered (e.g., the screen menu may be displayed or not displayed) and the instruction category to which the operation instruction corresponding to the trigger signal of the physical button belongs. Based on the trigger signal of the physical button, the time when the physical button is triggered and the display state of the screen menu at that time can be determined. Then, based on the correspondence and the display state of the screen menu at that time, the instruction category to which the operation instruction corresponding to the trigger signal of the physical button is triggered can be determined. It should be understood that the instruction category of the operation instruction can include shortcut instructions and screen menu adjustment instructions.

[0037] For example, if the screen menu is displayed when the physical button "X" is triggered, the operation command corresponding to the trigger signal of the physical button "X" is a screen menu adjustment command; if the screen menu is not displayed when the physical button "X" is triggered, the operation command corresponding to the trigger signal of the physical button "X" is a shortcut command. When the physical button "X" is triggered, the trigger time of the physical button "X" and the display state of the screen menu can be determined according to the trigger signal of the physical button "X". If the screen menu is displayed when the physical button "X" is triggered, it can be determined that the trigger signal of the button "X" is a screen menu adjustment command; if the screen menu is not displayed when the physical button "X" is triggered, it can be determined that the trigger signal of the button "X" is, for example, a shortcut command for calling the screen menu.

[0038] S220. If the operation instruction is a shortcut instruction, then the shortcut operation corresponding to the shortcut instruction will be executed.

[0039] S230. If the operation instruction is a screen menu adjustment instruction, then for the option to be adjusted in the screen menu currently displayed on the head-mounted display device, the adjustment operation corresponding to the screen menu adjustment instruction is executed, so that the head-mounted display device responds to the screen menu adjustment instruction to adjust the information display mode. In some optional embodiments, the screen menu adjustment instruction can be an instruction for directly or indirectly adjusting the options of the screen menu. The screen menu may include multiple display mode adjustment options for adjusting the information display mode of the head-mounted display device. When the option to be adjusted is a display mode adjustment option, the head-mounted display device can respond to the adjustment operation to adjust the information display mode by executing the adjustment operation corresponding to the screen menu adjustment instruction, so as to achieve the purpose of adjusting the information display mode in response to the screen menu adjustment instruction. When the option to be adjusted is a menu option other than a display mode adjustment option, the adjustment operation corresponding to the screen menu adjustment instruction may include multiple operations (such as option selection, option confirmation, and adjustment parameters, etc.). After executing the corresponding adjustment operation, the head-mounted display device can respond to the adjustment operation to adjust the information display mode.

[0040] Understandably, the screen menu can offer multiple adjustment options, such as display mode, display size, volume, and notification format. Each option adjusts the way information is displayed on the head-mounted display. Therefore, when the operation command is a screen menu adjustment command, first identify the option to be adjusted, and then execute the corresponding operation to achieve the goal of adjusting the information display format of the head-mounted display. In this solution, multiple information display formats of the head-mounted display can be adjusted through a unified logic; the method is simple and has a low learning curve.

[0041] As a feasible implementation, the screen menu can include multi-level menus. For example, the screen menu can include first-level menus, second-level menus, and third-level menus. The display mode adjustment option can be an option from any level of menu. Optionally, the menu option in the lowest level menu of the screen menu can be a display mode adjustment option.

[0042] In this embodiment, in response to the acquisition of a trigger signal indicating that at least one physical button has been triggered, an operation command corresponding to the trigger signal can be determined. The operation command may include shortcut commands and screen menu adjustment commands. If the operation command is a shortcut command, the corresponding shortcut operation can be executed. If the operation command is a screen menu adjustment command, the adjustment operation corresponding to the screen menu adjustment command can be executed for the adjustable option in the currently displayed screen menu of the head-mounted display device, so that the head-mounted display device adjusts the information display method in response to the screen menu adjustment command. This allows users to control the head-mounted display device to perform corresponding shortcut operations by triggering physical buttons, without having to use corresponding software to make the head-mounted display device perform the same operation, thus facilitating user control of the head-mounted display device. Furthermore, the embodiments of this application can centralize the adjustment of various information display modes into the screen menu. When a user wants to adjust a certain information display mode, they can find the corresponding menu option in the screen menu, select the menu option to be adjusted using a physical button, and then trigger the trigger signal corresponding to the screen menu adjustment command using the physical button. This allows the head-mounted display device to adjust the information display mode in response to the screen menu adjustment command, eliminating the need to search for software to adjust the information display mode among a large number of software programs. This effectively reduces the time users spend adjusting the information display mode of the head-mounted display device and makes it more convenient for users to adjust the information display mode of the head-mounted display device.

[0043] like Figure 3 As shown, in some optional embodiments, the screen menu may include a menu option display area and virtual buttons corresponding to each physical button, with a one-to-one correspondence between the physical buttons and the virtual buttons. The virtual buttons and the menu option display area are arranged adjacent to each other. The screen menu establishes a connection between the physical buttons and the menu options that the user needs to adjust via the physical buttons through the virtual buttons, allowing the user wearing the head-mounted display device to perceive the connection between the physical buttons and the options to be adjusted in the screen menu through the virtual buttons.

[0044] The method provided in this application embodiment also includes:

[0045] S310. Determine the virtual button corresponding to the triggered physical button.

[0046] Based on the trigger signal of the collected physical button, the triggered physical button can be identified, and then the virtual button corresponding to the triggered physical button can be identified.

[0047] S320. Set an identifier for the determined virtual key. The identifier is used to indicate the physical key that is triggered.

[0048] As a feasible implementation, the display identifier set for the determined virtual button can be a highlight identifier. This highlight identifier makes the virtual button determined in step S310 stand out from other virtual buttons, thus indicating the triggered physical button. Optionally, the identifier set for the determined virtual button can also be other than a symbol or text identifier, as long as it can distinguish the virtual button determined in step S310 from other virtual buttons.

[0049] S330, Render a screen menu including the identified virtual buttons to display the rendered screen menu within the field of view (FOV) of the head-mounted display device.

[0050] In some optional embodiments, a processor such as a CPU can render the screen menu, which includes the identified virtual buttons, and the rendered screen menu can be displayed within the field of view of the head-mounted display device after rendering is complete.

[0051] In this embodiment, the screen menu may include a menu option display area and virtual buttons corresponding to each physical button, with the virtual buttons and menu option display area arranged adjacent to each other. Optionally, the virtual buttons and menu option display area may be arranged adjacent to each other in a left-right display format. Furthermore, this solution can determine the virtual button corresponding to the triggered physical button, set an identifier for the determined virtual button, and render a screen menu including the identified virtual button to display the rendered screen menu within the field of view of the head-mounted display device, ensuring that the screen menu displayed by the head-mounted display device includes the identified virtual button. By indicating the triggered physical button with the identified virtual button, the tactile feedback of the user triggering the physical button can be combined with the visual feedback of the highlighted virtual button. This allows the user to intuitively understand whether the action of triggering the physical button is effective and the specific physical button triggered, even if they cannot see the physical button while wearing the head-mounted display device, thus facilitating the user's operation of the head-mounted display device via the physical button. It is understood that the wearing method of head-mounted display devices is significantly different from that of terminal devices such as mobile phones, computers, and smartwatches; the physical buttons of head-mounted display devices should not appear within the user's field of vision to avoid obstructing the user's view. Therefore, when users adjust the head-mounted display device using physical buttons, there is normally no visual feedback from the physical button adjustments. The solution disclosed in this embodiment can achieve visual feedback from the physical button adjustments of the head-mounted display device through virtual buttons and virtual button icons, thereby improving the accuracy of head-mounted display device adjustments and enhancing the user experience.

[0052] In some alternative embodiments, the head-mounted display device can be an eyeglass-style head-mounted display device comprising a frame, lenses, and temples. The lenses may be mounted in the frame, and the temples may be mounted at both ends of the frame.

[0053] Optionally, the temple of the head-mounted display device near the frame may have at least two physical buttons, which may be arranged sequentially along the length of the temple. Optionally, the temple may include an upper and a lower side facing away from each other, where the upper side is the side facing away from the ear when worn, and the lower side is the side facing away from the upper side. The at least two physical buttons may be located on the lower side of the temple for easy thumb activation. As a feasible implementation, the virtual buttons corresponding to the physical buttons may be located on the same side of the menu option display area as the physical buttons, allowing the user to associate the virtual buttons with the physical buttons.

[0054] In some optional embodiments, the field of view of the head-mounted display device may include a horizontal field of view (HFOV) and a vertical field of view (VFOV). For example... Figure 4 As shown, 400 represents the display area / virtual screen of the head-mounted display device, C represents the user's eyes, and the horizontal and vertical field of view are respectively... Figure 4 A and B in the example.

[0055] In some optional embodiments, the above method may further include: sequentially rendering virtual buttons corresponding to each physical button along the extension direction of the vertical field of view of the head-mounted display device. The arrangement order of the virtual buttons is consistent with the arrangement order of the corresponding physical buttons on the temples. The arrangement and display method of physical buttons and corresponding virtual buttons disclosed in this solution further improves the visual feedback effect of the corresponding virtual buttons when the physical buttons are triggered.

[0056] As a feasible implementation, the temple of the head-mounted display device near the frame can be equipped with three physical buttons: an "X" button, a "+" button, and a "-" button. These buttons can be sequentially arranged along the length of the temple. Alternatively, they can be positioned along the vertical field of view of the head-mounted display device, i.e., along... Figure 4 The virtual keys corresponding to the "X" key, the "+" key, and the "-" key are rendered sequentially along the extension direction of the straight line containing the middle arrow 410.

[0057] As an example, such as Figure 5As shown, in the head-mounted display device 500, in the direction from one end of the temple 510 near the frame 520 to the other end of the temple, the "X" key, "+" key, and "-" key can be arranged in the following order: X" key, "+" key, and "-" key. This corresponds to the arrangement order of the temples, as shown below. Figure 6 As shown, the virtual key 601 corresponding to the "X" key, the virtual key 602 corresponding to the "+" key, and the virtual key 603 corresponding to the "-" key are adjacent to the screen menu 610. In the direction of looking down from the top of the virtual screen 600 to the bottom of the virtual screen, the arrangement order of the virtual keys corresponding to the "X" key, the "+" key, and the "-" key can be as follows: the virtual key 601 corresponding to the "X" key, the virtual key 602 corresponding to the "+" key, and the virtual key 603 corresponding to the "-" key.

[0058] In this embodiment, at least two physical buttons are provided on the temple of the head-mounted display device near the frame, with each physical button arranged sequentially along the length of the temple. Virtual buttons corresponding to each physical button can be rendered sequentially along the vertical field of view of the head-mounted display device. The arrangement order of the virtual buttons is consistent with the arrangement order of the corresponding physical buttons on the temple, allowing the user to determine the arrangement order of the physical buttons on the temple based on the arrangement order of the virtual buttons, thereby distinguishing each physical button and improving the accuracy of the user clicking the desired physical button.

[0059] In some alternative embodiments, shortcuts may include screen menu invoke commands.

[0060] In some optional embodiments, if the operation instruction is a shortcut instruction, then the shortcut operation corresponding to the shortcut instruction is executed, which may include: if the operation instruction is a screen menu wake-up instruction, executing the operation corresponding to the screen menu wake-up instruction to display the screen menu within the field of view of the head-mounted display device.

[0061] As a feasible implementation, the operation command can be a screen menu invocation command. The trigger signal of the physical button "X" being triggered twice consecutively within a predetermined time period can be used to invoke the screen menu. Therefore, if the physical button "X" is triggered twice consecutively within a predetermined time period, the screen menu can be invoked.

[0062] In this embodiment, the shortcut command may include a screen menu activation command. When the operation command is a screen menu activation command, the operation corresponding to the screen menu activation command can be executed to display the screen menu within the field of view of the head-mounted display device. This allows the user to trigger physical buttons in a predetermined manner, generating a trigger signal corresponding to the screen menu activation command, thereby enabling the head-mounted display device to display the screen menu within the field of view without needing to enter the corresponding software or page for operation, thus simplifying the process required for the user to activate the screen menu.

[0063] In some optional embodiments, the operation corresponding to the screen menu invocation command is executed to display the screen menu within the field of view of the head-mounted display device. This includes: executing the operation corresponding to the screen menu invocation command, rendering the screen menu and the source image in parallel; overlaying the rendered screen menu onto the rendered source image, and performing a composite processing on the overlaid image; and sending the composite image to the virtual screen of the head-mounted display device to display the composite image within the field of view of the head-mounted display device. This solution can invoke the screen menu while displaying the source image without exiting the currently displayed source image, thus improving the efficiency of adjusting the information display method of the head-mounted display device.

[0064] Optionally, the source image can be understood as an image generated from the source image data of pictures, videos, etc., displayed by the head-mounted display device. The source image can be provided by a source device paired with the head-mounted display device. After obtaining the source image data from the source device, the head-mounted display device can process the source image data through the corresponding source image processing unit using the appropriate algorithm to obtain the source image.

[0065] In some optional embodiments, the head-mounted display device may further include a menu image processing unit, which can render the screen menu after running a corresponding algorithm. Optionally, the menu image processing unit and the source image processing unit can run in parallel to render the corresponding screen menu and source image. By using different processing units to render the screen menu and source image respectively, the processing unit for rendering the screen menu can focus on rendering the screen menu, and the processing unit for rendering the source image can focus on rendering the source image, thus improving the efficiency of rendering the screen menu and rendering the source image. In addition, by using different processing units to render the screen menu and source image respectively, the processing unit for rendering the screen menu and the processing unit for rendering the source image can be set specifically according to the different amounts of computation required to render the screen menu and the source image, thereby giving full play to the performance of the processing units. For example, the amount of computation required to render the screen menu is usually less than the amount of computation required to render the source image, so a higher-performance processing unit can be used to render the source image, and a lower-performance processing unit can be used to render the screen menu.

[0066] As a feasible approach, the background color of the screen menu can be dark to improve its readability and comfort.

[0067] In some optional embodiments, after the screen menu and source image are rendered, the rendered screen menu can be overlaid on the rendered source image. By compositing the overlaid image, display content containing the screen menu can be obtained. Optionally, the rendered screen menu can be sent to a first layer so that the cache of the first layer can store the screen menu. The source image can be sent to a second layer so that the cache of the second layer can store the source image. Then, the screen menu is retrieved from the cache of the first layer, and the source image is retrieved from the cache of the second layer. By performing image compositing on the two, a composite image displayed on the virtual screen of the head-mounted display device can be obtained.

[0068] In this embodiment, when executing the operation corresponding to the screen menu wake-up command, the screen menu and the source image can be rendered in parallel. The rendered screen menu is then superimposed on the rendered source image. The superimposed image is then composited and sent to the virtual screen of the head-mounted display device. This allows the composite image to be displayed within the field of view of the head-mounted display device, eliminating the need to display the rendered screen menu and the rendered source image separately. This avoids conflicts between displaying the rendered screen menu and the rendered source image, and improves the efficiency of adjusting the information display method of the head-mounted display device.

[0069] In some optional embodiments, rendering the screen menu and the source image in parallel may include: acquiring spatial distribution information of the head-mounted display device, wherein the spatial distribution information includes either pose information or attitude information; rendering the source image based on the acquired spatial distribution information; and rendering the screen menu in parallel to display the screen menu at a specified location within the field of view of the head-mounted display device.

[0070] The source image processing unit processes spatial distribution information, such as through spatial anchoring algorithms, to determine the spatial distribution of the rendered source image. As a possible implementation, the spatial distribution of the rendered source image can be fixed at a fixed position in space. Simultaneously, the menu processing unit, operating in parallel with the source image processing unit, can run algorithms such as binocular processing algorithms to ensure the rendered screen menu is located at a specified position within the field of view of the head-mounted display device. For example, the screen menu can be continuously displayed at the center of the field of view of the head-mounted display device. Therefore, in the composite image displayed by the head-mounted display device, the source image is anchored at a specified position in physical space, and the screen menu is located at a specified position within the field of view of the head-mounted display device. For example, the spatial distribution of the rendered screen menu can be kept directly in front of the user's eyes. When the user performs actions such as turning their head, looking down, or looking up, the spatial position of the rendered screen menu moves with the user's movements to maintain the relative position of the rendered screen menu with respect to the user's eyes. This ensures the display effect of the source image while preventing the screen menu from being lost from the field of view.

[0071] In some optional embodiments, the specified location of the display screen menu can be a default location or a location set by the user. This application does not limit the setting method of the specified location.

[0072] In this embodiment, spatial distribution information of the head-mounted display device can be obtained. Based on the obtained spatial distribution information, a source image is rendered; a screen menu is rendered in parallel to display the screen menu at a specified position within the field of view of the head-mounted display device. The scheme disclosed in this embodiment can use different spatial algorithms to render the source image and the screen menu in parallel, so that although both are displayed simultaneously in the virtual screen of the head-mounted display device, they have different display methods, improving the display effect of the head-mounted display device. Specifically, the screen menu currently displayed by the head-mounted display device can be continuously located at a specified position within the field of view of the head-mounted display device, while the source image is anchored at a specified position in physical space.

[0073] In some alternative embodiments, the frequency of use of the shortcut operation is greater than the frequency of use of at least one of the operations included in the adjustment operation.

[0074] The adjustment operations in this application embodiment may include multiple adjustment commands to adjust the information display method, such as adjusting the size of the virtual screen, adjusting the distance between the virtual screen and the viewer's eyes, etc. Optionally, the adjustment operations corresponding to the screen menu adjustment commands may also include shortcut operations corresponding to shortcut commands.

[0075] By setting the frequency of use of a shortcut operation to be greater than the frequency of use of at least one of the operations included in the adjustment operation, frequently used operations can be set as shortcut operations for easy access by users.

[0076] As a feasible approach, the order of menu options for each operation in the screen menu can be determined based on the frequency of use of the multiple operations included in the adjustment operation. This allows menu options for frequently used operations to be placed at the top of the screen menu, making it easier for users to access these operations through the menu options in the screen menu.

[0077] In some alternative embodiments, quick operations include at least one of the following: adjusting the virtual screen projection mode of the head-mounted display, displaying a menu, calibrating the virtual screen, and adjusting the transparency of the lenses of the head-mounted display.

[0078] The virtual screen projection modes of head-mounted displays can include hover mode (or spatial anchoring mode) and gimbal mode. In hover mode, the image displayed on the virtual screen can be fixed in a fixed position in space. In gimbal mode, the virtual screen can move with the head, and image stabilization can be applied to the virtual screen to prevent dizziness caused by screen movement. Virtual screen calibration can be used to calibrate the orientation of the virtual screen so that it corresponds to the direction of the user's eyes. The lenses of the head-mounted display can be electrochromic lenses. By adjusting the transparency of the lenses, the light transmission effect of the lenses can be changed, that is, the clarity of the external scene viewed by the user through the lenses of the head-mounted display.

[0079] In this embodiment, quick operations can include adjusting the virtual screen projection mode of the head-mounted display device, displaying the menu, calibrating the virtual screen, adjusting the transparency of the lenses of the head-mounted display device, and so on. Users can trigger the head-mounted display device to quickly perform various operations such as adjusting the virtual screen projection mode, displaying the menu, calibrating the virtual screen, or adjusting the transparency of the lenses of the head-mounted display device through the trigger signal triggered by the physical button, which can greatly facilitate the use of the head-mounted display device.

[0080] like Figure 7 As shown in the figure, this application embodiment also provides a device 600 for controlling a head-mounted display device, including:

[0081] The response module 710 is used to respond to the acquisition of a trigger signal that at least one physical button is triggered, and to determine the operation instruction corresponding to the trigger signal, the operation instruction including shortcut instructions and screen menu adjustment instructions;

[0082] The first execution module 720 is used to execute the shortcut operation corresponding to the shortcut instruction if the operation instruction is the shortcut instruction;

[0083] The second execution module 730 is configured to, if the operation instruction is the screen menu adjustment instruction, perform an adjustment operation corresponding to the screen menu adjustment instruction for the option to be adjusted in the screen menu currently displayed by the head-mounted display device, so that the head-mounted display device adjusts the information display mode in response to the screen menu adjustment instruction.

[0084] It should be understood that the device for controlling a head-mounted display device provided in this application embodiment is based on the same inventive concept as the aforementioned method embodiment and can achieve the same effect. The structure and operation of the device for controlling a head-mounted display device can be found in the description of the aforementioned display device embodiment, and will not be repeated here.

[0085] This application also provides a head-mounted display device. For example... Figure 8 As shown, the head-mounted display device may include: a processor 802, a communications interface 804, a memory 806, and a communications bus 808.

[0086] in:

[0087] The processor 802, communication interface 804, and memory 806 communicate with each other through the communication bus 808.

[0088] Communication interface 804 is used to communicate with other electronic devices or servers, such as the source device mentioned above.

[0089] The processor 802 is used to execute computer software 810, specifically to perform the relevant steps in the embodiments of the method for controlling the head-mounted display device described above.

[0090] Specifically, computer software 810 may include program code, which includes executable instructions, which may be computer operation instructions.

[0091] The processor 802 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0092] Memory 806 is used to store computer software 810. Memory 806 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0093] The computer software 810 may include multiple executable instructions. Specifically, the computer software 810 may use multiple executable instructions to cause the processor 802 to perform the operation corresponding to the method for controlling the head-mounted display device described in any of the foregoing method embodiments.

[0094] The specific implementation of each step in computer software 810 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0095] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.

[0096] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0097] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods described herein are implemented. Furthermore, when a general-purpose computer accesses the code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0098] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0099] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A method for controlling a head-mounted display device, characterized in that, The method includes: In response to the acquisition of a trigger signal that at least one physical button has been triggered, an operation instruction corresponding to the trigger signal is determined, the operation instruction including shortcut instructions and screen menu adjustment instructions; If the operation instruction is the shortcut instruction, then the shortcut operation corresponding to the shortcut instruction is executed; If the operation instruction is the screen menu adjustment instruction, then for the option to be adjusted in the screen menu currently displayed on the head-mounted display device, the adjustment operation corresponding to the screen menu adjustment instruction is executed, so that the head-mounted display device responds to the screen menu adjustment instruction to adjust the information display mode.

2. The method according to claim 1, characterized in that, The screen menu includes a menu option display area and virtual buttons corresponding to each physical button, wherein the virtual buttons and the menu option display area are arranged adjacent to each other; The method further includes: Determine the virtual button corresponding to the triggered physical button; Set an identifier for the identified virtual button, the identifier being used to indicate the triggered physical button; The rendering includes a screen menu of the identified virtual buttons, to be displayed within the field of view of the head-mounted display device.

3. The method according to claim 2, characterized in that, The temple of the head-mounted display device has at least two physical buttons near the frame, and the physical buttons are arranged sequentially along the length of the temple; the field of view includes a vertical field of view, and the method further includes: Along the extension direction of the vertical field of view of the head-mounted display device, virtual buttons corresponding to each of the physical buttons are rendered sequentially, wherein the arrangement order of the virtual buttons is consistent with the arrangement order of the corresponding physical buttons on the temples.

4. The method according to claim 1, characterized in that, The shortcut commands include screen menu activation commands; If the operation instruction is the shortcut instruction, then the shortcut operation corresponding to the shortcut instruction is executed, including: If the operation instruction is the screen menu activation instruction, the operation corresponding to the screen menu activation instruction is executed to display the screen menu within the field of view of the head-mounted display device.

5. The method according to claim 4, characterized in that, The step of performing the operation corresponding to the screen menu wake-up command to display the screen menu within the field of view of the head-mounted display device includes: Execute the operation corresponding to the screen menu invocation command, and render the screen menu and the source image in parallel respectively; The rendered screen menu is overlaid onto the rendered source image, and the overlaid image is then composited. The synthesized image is sent to the virtual screen of the head-mounted display device to be displayed within the field of view of the head-mounted display device.

6. The method according to claim 5, characterized in that, The parallel rendering of the screen menu and the source image includes: Acquire spatial distribution information of the head-mounted display device, wherein the spatial distribution information includes one of pose information and posture information; Based on the acquired spatial distribution information, the source image is rendered; The screen menu is rendered in parallel to display the screen menu at a specified location within the field of view of the head-mounted display device.

7. The method according to claim 1, characterized in that, The frequency of use of the shortcut operation is greater than the frequency of use of the at least one of the operations included in the adjustment operation.

8. The method according to claim 1, characterized in that, The quick operation includes at least one of the following: adjusting the virtual screen projection mode of the head-mounted display device, displaying the menu, calibrating the virtual screen, and adjusting the transparency of the lenses of the head-mounted display device.

9. A device for controlling a head-mounted display device, characterized in that, include: The response module is used to respond to the acquisition of a trigger signal that at least one physical button is triggered, and to determine the operation instruction corresponding to the trigger signal. The operation instruction includes shortcut instructions and screen menu adjustment instructions. The first execution module is configured to execute the shortcut operation corresponding to the shortcut instruction if the operation instruction is the shortcut instruction. The second execution module is configured to, if the operation instruction is the screen menu adjustment instruction, perform an adjustment operation corresponding to the screen menu adjustment instruction for the option to be adjusted in the screen menu currently displayed by the head-mounted display device, so that the head-mounted display device responds to the screen menu adjustment instruction to adjust the information display mode.

10. A head-mounted display device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one computer software that causes the processor to perform the method as described in any one of claims 1-8.

11. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-8.

Images