Control apparatus, method, device and storage medium for a head-mounted display device

Abstract

The embodiment of the present disclosure provides a control device and method for a head-mounted display device, the head-mounted display device, an electronic device and a storage medium, the control method comprising: obtaining a working mode set, wherein the working mode set is used to represent at least one of the following: a display mode of the head-mounted display device, an interaction mode of the head-mounted display device, and a preset category application in an activated state; determining, according to the working mode set, a control weight of each control component in a plurality of control components used to control a pointer in a virtual space rendered by the head-mounted display device, wherein the plurality of control components comprise output quantities corresponding to a plurality of control modes of the head-mounted display device respectively; and performing weighted fusion on the plurality of control components by using the control weight to obtain a control quantity used to control the pointer in the virtual space rendered by the head-mounted display device.

Description

Technical Field

[0001] This disclosure relates to mixed reality display technology, and in particular to a control device and method for a head-mounted display device, a head-mounted display device, an electronic device, and a storage medium. Background Technology

[0002] With the continuous development of mixed reality display technology, the control requirements for human-computer interaction in head-mounted display devices are becoming increasingly diverse. In related technologies, human-computer interaction control for head-mounted display devices mainly relies on independently controlling objects in the virtual space of the head-mounted display device using the head, eyes, or hands. Summary of the Invention

[0003] This disclosure provides a control device and method for a head-mounted display device, a head-mounted display device, an electronic device, and a storage medium, which enable users to achieve joint control based on the head, eyes, and hands during human-computer interaction while wearing the head-mounted display device, thereby improving the user experience.

[0004] One aspect of this disclosure provides a control method for a head-mounted display device, comprising: acquiring a set of operating modes, wherein the set of operating modes is used to characterize at least one of the following: a display mode of the head-mounted display device, an interaction mode of the head-mounted display device, and a preset category application being active; determining, based on the set of operating modes, a control weight for each of a plurality of control components used to control a pointer in a virtual space rendered by the head-mounted display device, wherein the plurality of control components include output quantities corresponding to a plurality of control methods of the head-mounted display device; and using the control weights to perform weighted fusion of the plurality of control components to obtain a control quantity used to control a pointer in a virtual space rendered by the head-mounted display device.

[0005] Another aspect of this disclosure provides a control device for a head-mounted display device, comprising: an acquisition module configured to: acquire a set of operating modes, wherein the set of operating modes is used to characterize at least one of the following: a display mode of the head-mounted display device, an interaction mode of the head-mounted display device, and a preset category application being active; a weight determination module configured to: determine, based on the set of operating modes, a control weight for each of a plurality of control components used to control a pointer in a virtual space rendered by the head-mounted display device, wherein the plurality of control components include output quantities corresponding to a plurality of control methods of the head-mounted display device; and a control execution module configured to: use the control weights to perform weighted fusion of the plurality of control components to obtain a control quantity used to control a pointer in a virtual space rendered by the head-mounted display device.

[0006] In another aspect of the present disclosure, a head-mounted display device is provided, the head-mounted display device including the control device of the present disclosure.

[0007] In another aspect of the present disclosure, a non-volatile computer program product is provided, which stores instructions that, when executed by at least one data processor of at least one computing system, implement the control method of the present disclosure.

[0008] In another aspect of this disclosure, an electronic device is provided, comprising: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, wherein when the computer program is executed, it implements the control method of this disclosure.

[0009] In another aspect, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the control method of the present disclosure.

[0010] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0011] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0012] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0013] Figure 1 This is a flowchart of one embodiment of the control method for a head-mounted display device disclosed herein;

[0014] Figure 2 This is a flowchart of another embodiment of the control method for a head-mounted display device disclosed herein;

[0015] Figure 3 This is a flowchart of yet another embodiment of the control method for a head-mounted display device disclosed herein;

[0016] Figure 4 This is a schematic diagram of one embodiment of the publicly available world coordinate system;

[0017] Figure 5 This is a flowchart of yet another embodiment of the control method for a head-mounted display device disclosed herein;

[0018] Figure 6 This is a schematic diagram of one embodiment of the pre-defined key points of the eye in this disclosure;

[0019] Figure 7 This is a schematic diagram of one embodiment of the multi-screen mode disclosed herein;

[0020] Figure 8 This is a flowchart of another embodiment of the control method for a head-mounted display device disclosed herein;

[0021] Figure 9 This is a schematic diagram of one embodiment of the control device for a head-mounted display device disclosed herein;

[0022] Figure 10 This is a schematic diagram of another embodiment of the control device for a head-mounted display device disclosed herein;

[0023] Figure 11 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0024] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0025] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0026] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0027] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0028] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0029] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0030] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0031] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0032] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0033] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0034] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0035] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0036] Exemplary methods

[0037] Figure 1 This is a flowchart of one embodiment of the control method for a head-mounted display device disclosed herein. Figure 1 As shown, the control method for a head-mounted display device includes steps S110, S120, and S130. Each step is described in detail below.

[0038] S110, Obtain the set of working modes.

[0039] As an alternative implementation, the set of working modes can be used to characterize at least one of the following: the display mode of the head-mounted display device, the interaction mode of the head-mounted display device, and the activation state of a preset category application.

[0040] It should be noted that this disclosure does not limit the "method of obtaining" the set of working modes. For example, the set of working modes of the head-mounted display device can be obtained through pattern recognition.

[0041] Head-mounted display devices can be all-in-one devices that integrate computing power and display, or separate devices that separate computing power and display.

[0042] In this paper, for a split-type head-mounted display device, computing power can be provided by a computing unit, and display is performed by the display of the head-mounted display device connected to the computing unit. An all-in-one head-mounted display device can be considered as a head-mounted display device integrating both a computing unit and a display; the computing unit connected to the head-mounted display device refers to the computing unit integrated within the head-mounted display device. In subsequent processing, for both all-in-one and split-type head-mounted display devices, the corresponding operations can be considered to be performed by the computing unit connected to the head-mounted display device.

[0043] As an alternative implementation, the display mode of the head-mounted display device can be determined based on the number of screens presented by the head-mounted display device. For example, single-screen mode or multi-screen mode. The number of screens presented by the head-mounted display device can be provided by the operating system of the head-mounted display device.

[0044] The operating mode of a head-mounted display device can be determined based on the user's interaction with the device. This includes, but is not limited to, gesture mode, no-gesture mode, and supplementary control mode. Supplementary control modes include, but are not limited to, voice control mode and peripheral control mode (handheld controllers, ring controllers, mobile phones, mice, keyboards, etc.). Whether a gesture mode is included can be determined by checking if the currently captured image contains a preset gesture. If the captured image contains the preset gesture, then a gesture mode is included. If the captured image does not contain the preset gesture, then a no-gesture mode is included. Whether a supplementary control mode is included can be determined by checking if there is a supplementary control method. If the head-mounted display device's microphone is on and the voice it captures matches a preset voice command, then a voice control mode is included. If the microphone is off or on but the captured voice does not match a preset voice command, then a voice control mode is not included. The head-mounted display device's operating system can provide information on whether peripheral devices are connected. If the operating system recognizes a connected peripheral device, then a peripheral control mode is included.

[0045] Preset application categories can include, but are not limited to, game applications. The corresponding working mode is game mode, and the activation state is, for example, a game application that is launched and in the foreground. The operating system of the head-mounted display device can provide the category of the currently launched application, such as music application, game application, shopping application, etc.

[0046] In addition, the various working modes in the above set can be combined according to the user's application of the head-mounted display device.

[0047] S120. Based on the set of working modes, determine the control weight of each control component among the various control components used to control the pointers in the virtual space rendered by the head-mounted display device.

[0048] As an optional implementation, the multiple control components may include the output quantities corresponding to various control methods of the head-mounted display device.

[0049] As an optional implementation, the control method may include coarse control, medium control, fine control, and supplementary control, with different control methods corresponding to different control components. Among them, coarse control may be head movement control, medium control may be eye movement control, fine control may be gesture control, and supplementary control may be voice control or peripheral device control (gamepad, ring controller, mobile phone, mouse, keyboard, etc.).

[0050] S130. Using control weights, multiple control components are weighted and fused to obtain the control quantity used to control the pointer in the virtual space rendered by the head-mounted display device.

[0051] As an alternative implementation, weighted fusion can be achieved by weighting the control weights using control weight coefficients to obtain a joint control quantity, which serves as the control quantity for controlling pointers in the virtual space rendered by the head-mounted display device. For example, by multiplying the control components of the coarse control method, medium control method, fine control method, and supplementary control method by their respective control weights, and then summing the results, a joint control quantity that integrates the four control methods can be obtained.

[0052] As described above, the control method for head-mounted display devices disclosed herein first determines the control weight of each of the various control components used to control pointers in the virtual space rendered by the head-mounted display device, based on a set of operating modes. Then, using these control weights, the various control components are weighted and fused to obtain the control quantity used to control the pointers in the virtual space rendered by the head-mounted display device. This enables joint control of the head-mounted display device based on different control methods. Specifically, it allows users to achieve joint control based on head, eyes, hands, voice, and peripherals during human-computer interaction while wearing the head-mounted display device, without needing to switch control methods, thereby improving the user experience.

[0053] Step S120 can be implemented in a variety of available ways.

[0054] As an optional implementation, step S120 may include: in response to finding the set of operating modes in the control weight mapping table, obtaining the initial control weight of each control component associated with the set of operating modes from the control weight mapping table.

[0055] It is understandable that by pre-configuring the set of commonly used working modes and the initial control weights of each corresponding control component into a control weight mapping table, and then looking up the table, the execution speed of this step can be improved, thereby improving the implementation efficiency of the entire control method.

[0056] As an optional implementation method, refer to Figure 2 Step S120 may include:

[0057] S1210. Obtain the mode weight of each working mode in the set of working modes, and the component weight of each control component corresponding to each working mode; S1220. Calculate the control weight of each control component based on the mode weight of each working mode and the component weight of each control component corresponding to each working mode.

[0058] It is understandable that by using mode weights and component weights to calculate the control weight of each control component, the control weight of each control component under different combinations of operating modes can be determined, thus preparing for subsequent steps to fuse multiple control components for different operating modes.

[0059] The step S1210 above, "obtaining the mode weight of each working mode in the set of working modes", can be achieved in a variety of ways.

[0060] As an optional implementation, step S1210, "obtaining the mode weight of each working mode in the working mode set," may include:

[0061] In response to finding the set of working modes in the mode weight mapping table, the mode weights corresponding to each working mode in the set of working modes are obtained from the mode weight mapping table.

[0062] It's understandable that the pattern weight mapping table stores the pattern weights corresponding to each working mode in a pre-configured set of working modes. If the set of working modes can be found in the pattern weight mapping table, the pattern weights corresponding to each working mode can be directly obtained by looking up the table, which can improve method execution speed and thus increase efficiency.

[0063] As another optional implementation, step S1210, "obtaining the mode weight of each working mode in the working mode set", may include:

[0064] First, based on the preset priority of each working mode, sort each working mode in the working mode set from high to low to obtain a working mode sequence. Second, according to the preset selection rules, select the corresponding mode weight for each working mode in the working mode sequence from the preset mode weight table. The preset mode weight table includes at least one mode weight value. The preset selection rules are: according to the sorting of each working mode in the working mode sequence, select the top N mode weights with the largest mode weight values ​​from the mode weight table in sequence, where N is the number of working modes in the working mode sequence.

[0065] After obtaining the set of working modes, the top N modes with the largest weight values ​​can be selected from the preset mode weight table based on the sorting of the working mode sequence.

[0066] It is understandable that if the set of working modes is not found in the mode weight mapping table of the above optional implementation, the method of sorting the working mode sequence and selecting the top N mode weights with the largest mode weight values ​​from the mode weight table in this implementation can also be used. Because this implementation has strong universality, it can ensure that the final solution has a reasonable output control amount while taking into account execution speed.

[0067] As an optional implementation, the step S1210 of "obtaining the component weights of each control component corresponding to each working mode" can be achieved as follows: obtain the component weights of each control component corresponding to each working mode from a preset component weight mapping table.

[0068] As an optional example, the preset component weight mapping table can be as shown in Table 1:

[0069]

[0070] Table 1

[0071] The preset component weight mapping table in Table 1 is only an example. Those skilled in the art can adjust the corresponding control components of each of the above operating modes according to actual needs, and can add or delete certain operating modes as needed. Those skilled in the art can also add or delete certain control component weights according to actual needs and the required control method.

[0072] It is understandable that the preset component weight mapping table pre-configured each operating mode and the corresponding component weights of each control component. By directly looking up the table, the execution speed of this step can be accelerated, which is beneficial to improving the execution efficiency of the entire solution.

[0073] Step S1220 can be achieved in several ways:

[0074] As an optional implementation, without considering normalization, the product of the mode weight and the corresponding component weight of each control component in each working mode can be calculated; the products of each control component in each working mode are added together to obtain the control weight of each control component.

[0075] As another optional implementation, when normalization needs to be considered, the mode weight of each working mode in the set of working modes can be normalized to obtain the normalized mode weight value corresponding to each working mode; calculate the product of the normalized mode weight value of each control component in each working mode and the corresponding component weight; add the products of each control component in each working mode to obtain the control weight corresponding to each control component.

[0076] As another alternative implementation, if normalization needs to be considered, the product of the mode weight of each control component in each working mode and the corresponding component weight can be added to obtain the sum of the weight products of the working mode set corresponding to each control component; the sum of the weight products of each control component can be normalized to obtain the control weight corresponding to each control component.

[0077] To facilitate understanding, the following examples illustrate several ways of determining the control weight of each control component.

[0078] Assume the preset mode weight table contains preset mode weight values ​​of 0.8, 0.7, 0.5, 0.3, 0.2, and 0.1; and assume the priority between working modes is: game mode > gesture mode / no gesture mode > multi-screen mode / single-screen mode > voice mode / peripheral mode.

[0079] If the currently obtained set of working modes includes three modes: game mode, gesture mode, and single-screen mode, then by querying the preset mode weight table, the first three mode weight values ​​are taken as the mode weights for these three working modes, which are 0.8 for game mode, 0.7 for gesture mode, and 0.5 for single-screen mode. By querying the preset component weight mapping table, assuming the component weights of each control component in game mode are: W_head = 0.1, W_eye = 0.1, W_hand = 0.8; the component weights of each control component in gesture mode are: W_head = 0, W_eye = 0, W_hand = 1; and the component weights of each control component in single-screen mode are: W_head = 0.3, W_eye = 0.3, W_hand = 0.4.

[0080] The control weight for each control component can be calculated as follows:

[0081] (1) For the case of non-normalization,

[0082] The weight of the header control component is W_head_init = 0.8*0.1 + 0.7*0 + 0.5*0.3 = 0.23

[0083] The component weights for the eye control component are W_eye_init = 0.8*0.1 + 0.7*0 + 0.5*0.3 = 0.23

[0084] The component weight of the gesture control component is W_hand_init = 0.8 * 0.8 + 0.7 * 1 + 0.5 * 0.4 = 1.54

[0085] (2) For cases where it is necessary to normalize the pattern weights

[0086] The normalized weights for game mode, gesture mode, and single-screen mode are as follows:

[0087] 0.8 / (0.8+0.7+0.5)=0.4

[0088] 0.7 / (0.8+0.7+0.5)=0.35

[0089] 0.5 / (0.8+0.7+0.5)=0.25

[0090] Then, it is calculated based on the normalized value of the pattern weight.

[0091] The weight of the header control component is W_head_init = 0.4 * 0.1 + 0.35 * 0 + 0.25 * 0.3 = 0.115

[0092] The component weights for the eye control component are W_eye_init = 0.4*0.1 + 0.35*0 + 0.25*0.3 = 0.115

[0093] The weight of the gesture control component is W_hand_init = 0.4 * 0.8 + 0.35 * 1 + 0.25 * 0.4 = 0.77

[0094] (3) For cases where the product of mode weights and component weights needs to be normalized,

[0095] First, calculate the sum of the products of the mode weight and the component weight for each control component in game mode, gesture mode, and single-screen mode.

[0096] W_head'=0.8*0.1+0.7*0+0.5*0.3=0.23

[0097] W_eye'=0.8*0.1+0.7*0+0.5*0.3=0.23

[0098] W_hand'=0.8*0.8+0.7*1+0.5*0.4=1.54

[0099] Then, the sum of the products is normalized to obtain the result.

[0100] The weight of the head control component is W_head_init = 0.23 / (0.23 + 0.23 + 1.54) = 0.115

[0101] The component weights for the eye control component are W_eye_init = 0.23 / (0.23 + 0.23 + 1.54) = 0.115

[0102] The component weight for the gesture control is W_hand_init = 1.54 / (0.23 + 0.23 + 1.54) = 0.77

[0103] It is understandable that in the above example, calculating the control weights for each control component without considering normalization can speed up processing, making each control component more sensitive to the control method. However, considering normalization allows for a more reasonable allocation of weight proportions, thereby improving the accuracy of the control weights.

[0104] The working mode set, mode weight values ​​in the preset mode weight table, component weight values ​​of each control component in the preset component weight mapping table, and priorities between working modes shown in the above examples are merely examples to facilitate a better understanding of the technical solutions of this disclosure by those skilled in the art, and do not constitute a limitation on the working mode set, preset mode weight table, preset component weight mapping table, or priorities between working modes of this disclosure. Any adjustments made by those skilled in the art to the working mode set, preset mode weight table, preset component weight mapping table, or priorities between working modes in terms of values ​​and order after understanding the technical solutions of this disclosure should be included within the protection scope of this disclosure.

[0105] As an optional embodiment, in Figure 1 Based on the embodiments, the control method further includes: determining the control weight coefficient of the corresponding control component based on the signal-to-noise ratio of each control component among multiple control components; and correcting the control component based on the control weight coefficient to obtain the corrected control component.

[0106] The aforementioned "determining the control weight coefficient of the corresponding control component based on the signal-to-noise ratio (SNR) of each control component among multiple control components" can be achieved in several ways. For example, a control mapping table can be used to look up the control weight coefficient for each control component at various SNRs. The SNR in the control mapping table can be data intervals, with each interval corresponding to a control weight coefficient. Alternatively, the control mapping table can represent a discrete correspondence between SNR values ​​and control weight coefficients. When the difference between the obtained SNR and the SNR value in the control mapping table is minimal, the control weight coefficient corresponding to the SNR value with the smallest difference can be determined as the corresponding control weight coefficient. If there are two SNR values ​​with the smallest difference from the control mapping table, the average of the control weight coefficients corresponding to these two SNR values ​​can be taken as the corresponding control weight coefficient. Alternatively, a function showing a positive correlation between the control weight coefficient and the SNR can be used to calculate the control weight coefficient for the corresponding SNR.

[0107] As an optional implementation, multiple control components can use the same control mapping table to configure the correspondence between control weight coefficients and signal-to-noise ratio (SNR); alternatively, each control component can use a separate control mapping table. Multiple control components can also use the same positive correlation function to characterize the correspondence between control weight coefficients and SNR; alternatively, each control component can use a separate positive correlation function. The specific configuration can be determined based on user requirements and the accuracy of the control method.

[0108] Furthermore, the aforementioned "correcting the control component based on the control weight coefficient to obtain the corrected control component" can be implemented as follows: The corresponding control component value is adjusted using the control weight coefficient, thereby correcting the control component. For example, if the head control component is V_head and the head control weight coefficient is k_head, the head control component is adjusted by multiplying it by the head control weight coefficient. The corrected head control component is then k_head * V_head. This achieves the scheme of "correcting the control component based on the control weight coefficient."

[0109] It is understandable that by adjusting the weighting coefficients of the control components, the influence of a control component can be reduced before weighted fusion when its signal-to-noise ratio (SNR) is low, and increased when its SNR is high. Thus, the influence of a control component in the final weighted fusion result is determined based on its SNR. Increasing the proportion of reliable control methods and decreasing the proportion of control methods that affect reliability ensures the effectiveness of the final weighted fusion result and improves user experience.

[0110] As another optional embodiment, in Figure 1 Based on the embodiments, the control method further includes: determining the smoothing intensity coefficient of the corresponding control component based on the signal-to-noise ratio of each control component among multiple control components; and smoothing the control component based on the smoothing intensity coefficient to obtain the smoothed control component.

[0111] The aforementioned "determining the smoothing intensity coefficient of the corresponding control component based on the signal-to-noise ratio (SNR) of each control component among multiple control components" can be achieved in several ways. For example, a smoothing mapping table can be used to look up the smoothing intensity coefficient for each control component at various SNRs. The SNR in the smoothing mapping table can be data intervals, with each interval corresponding to a smoothing intensity coefficient. Alternatively, the smoothing mapping table can represent the correspondence between discrete SNR values ​​and smoothing intensity coefficients. When the difference between the obtained SNR and the SNR value in the smoothing mapping table is minimal, the smoothing intensity coefficient corresponding to the SNR value with the smallest difference can be determined as the corresponding smoothing intensity coefficient. If there are two SNR values ​​with the smallest difference between the obtained SNR and the smoothing mapping table, the average of the smoothing intensity coefficients corresponding to these two SNR values ​​can be taken as the corresponding smoothing intensity coefficient. Alternatively, a function showing a negative correlation between the smoothing intensity coefficient and the SNR can be used to calculate the smoothing intensity coefficient for the corresponding SNR.

[0112] As an optional implementation, multiple control components can use the same smoothing mapping table to configure the correspondence between the smoothing intensity coefficient and the signal-to-noise ratio (SNR); alternatively, each control component can use a separate smoothing mapping table to configure the correspondence between the smoothing intensity coefficient and the SNR. Multiple control components can also use the same negative correlation function to characterize the correspondence between the smoothing intensity coefficient and the SNR; alternatively, each control component can use a separate negative correlation function to characterize the correspondence between the smoothing intensity coefficient and the SNR. The specific configuration can be determined based on user requirements and the performance of the control method.

[0113] As an alternative example, the above smoothing of control components may include: low-pass filtering of multiple control components using smoothing intensity coefficients corresponding to multiple control components.

[0114] It is understandable that smoothing each control component using a smoothing intensity coefficient can filter out jitter interference in each control component, obtain the control intent of each control mode, thereby improving the accuracy of the control components, which is conducive to improving control effectiveness and stability, and optimizing user experience.

[0115] As another optional embodiment, in Figure 1Based on the embodiments, the control method further includes: adjusting the target control component according to the received user adjustment instruction for the target control component to obtain the adjusted target control component.

[0116] As an optional example, the user's adjustment command for the target control component can be issued through at least one of the various control methods (e.g., coarse control, medium control, fine control, and supplementary control). The adjustment of the target control component can include adjusting the control weight coefficient and / or smoothing intensity coefficient of the target control component according to the adjustment command, thereby achieving the adjustment of the target control component.

[0117] In addition, when a user wearing a head-mounted display device switches or the same user is in different usage states, the control weight coefficients and smoothing intensity coefficients of each control component can be recalculated and updated to match user needs and achieve personalized or intelligent adjustments.

[0118] As an optional embodiment, in Figure 1 Based on the embodiments, the control method further includes: determining each of a plurality of control components, each control component being determined by at least one external input signal, the external input signal including at least one of the following: sensing information collected by a sensor associated with the head-mounted display device, and output information of an external device associated with the head-mounted display device.

[0119] As an optional implementation, the control component is used to control the amount of change of a pointer in the virtual space rendered by the head-mounted display device in both displacement and orientation dimensions. Sensors associated with the head-mounted display device may include, but are not limited to, inertial measurement units, visual sensors (e.g., cameras, visual odometry), eye-tracking sensors, etc. External devices associated with the head-mounted display device may include, but are not limited to, gamepads, ring controllers, mobile phones, mice, keyboards, etc.

[0120] It is understood that by determining the control components through the above implementation method, the final control quantity of the pointer can be determined by weighted fusion based on the aforementioned control weights. Therefore, the execution of this embodiment can prepare for the subsequent "fusion" step.

[0121] As an optional implementation method, refer to Figure 3 Determining each of the multiple control components involves the following steps:

[0122] S210, in response to the head control method, uses external input signals to determine the pose change of the head-mounted display device based on the world coordinate system.

[0123] As an optional example, step S210 can be implemented as follows:

[0124] First, image information of the scanned area of ​​the head-mounted display device (HMD) is acquired by a visual sensor associated with the HMD (e.g., visual odometry based on ORB, SIFT, or deep learning), and pose information of the HMD is acquired by an inertial measurement unit (IMU). Then, based on the image and pose information, an environmental map of the scanned area of ​​the HMD is built using Simultaneous Localization and Mapping (SLAM) in a world coordinate system, and the real-time location of the HMD in the environmental map is determined. Finally, based on the real-time location of the HMD in the environmental map, the rotation matrix between the pose of the HMD at the current head sampling moment and the pose at the previous head sampling moment is determined as the pose change.

[0125] In addition, to further improve the accuracy of pose change, backend optimization of the visual sensor and IMU fusion SLAM system can be performed based on loop closure detection.

[0126] Reference Figure 4 The positive directions of each coordinate axis in the world coordinate system can be: the positive direction of the X-axis is parallel to the horizontal plane to the right; the positive direction of the Y-axis is perpendicular to the horizontal plane and vertically downward; and the positive direction of the Z-axis is parallel to the horizontal plane forward. The X-axis, Y-axis, and Z-axis satisfy the orthogonal right-handed coordinate system.

[0127] S220. Determine the head control components based on the pose change.

[0128] As an optional example, step S220 can be implemented as follows: First, the rotation angle of the head-mounted display device is determined based on the rotation matrix; then, the position offset of the pointer in the virtual space is calculated as a head control component based on the rotation angle and the depth information of the pointer in the virtual space rendered by the head-mounted display device.

[0129] It is understandable that by using the above method, while realizing head localization and environmental mapping, a real-time 3-DOF head rotation matrix can be obtained as a pose change quantity, so it is convenient and efficient to implement. In addition, by using rotation angle and depth information, the positional offset of the user's head on the pointer can be determined by simple calculation. Because the computational complexity is low, the response speed of the calculation result is fast.

[0130] As an optional implementation method, refer to Figure 5 Determining each of the multiple control components involves the following steps:

[0131] S310, in response to eye control, uses an external input signal to determine the positional change of a preset eye key point of the user wearing the head-mounted display device.

[0132] As an optional example, step S310 can be implemented as follows:

[0133] First, acquire at least one consecutive frame of eye images of a user wearing a head-mounted display device. Then, using an eye keypoint extraction network, determine the position information of preset eye keypoints from the at least one consecutive frame of eye images. The position information of the preset eye keypoints includes at least one of the following: a set of position information for the left eye preset keypoints and a set of position information for the right eye preset keypoints; wherein the left eye preset keypoints include at least the left pupil, and the right eye preset keypoints include at least the right pupil. Finally, based on the position information of the preset eye keypoints, determine the position change of the corresponding preset eye keypoints. The position change of the preset eye keypoints includes at least one of the following: the angle change of the ray formed by the left pupil at the current eye sampling time relative to the previous eye sampling time, and the angle change of the ray formed by the right pupil at the current eye sampling time relative to the previous eye sampling time; wherein the ray formed by the left pupil is a ray perpendicular to and passing through the geometric center of the left pupil; the ray formed by the right pupil is a ray perpendicular to and passing through the geometric center of the right pupil. All rays can exit from the pupil and point towards the virtual space. In the case of binocular control, it can be the average of the changes in the angle between the rays formed by the pupils of both eyes.

[0134] Reference Figure 6 The black dots distributed around the eyes are the aforementioned preset key eye points. Assuming the user's working mode when using the head-mounted display device is as follows... Figure 7 In the multi-screen mode shown, the positional change of the user's preset eye key points can be determined using the above implementation method.

[0135] Furthermore, this disclosure does not limit the method of acquiring at least one consecutive frame of eye images of a user wearing a head-mounted display device. For example, eye images of the user can be acquired via an eye-tracking sensor or a camera. The eye keypoint extraction network may include, but is not limited to, a neural network.

[0136] S320. Determine the eye control components based on the positional changes of preset key eye points.

[0137] As an optional example, step S320 can be implemented as follows: based on the positional change of the eye key points and the depth information of the pointer in the virtual space rendered by the head-mounted display device, calculate the positional offset of the pointer in the virtual space as an eye control component.

[0138] It is understandable that, using the above method, the change in the angle between the pupil ray at the current eye sampling time and the previous eye sampling time can be determined by detecting changes in the position of key points in the user's eye. Since the detection method is simple, it is easy to implement. Then, using this angle change and depth information, the eye control components can be determined through simple geometric calculations. Because the computational complexity is low, the response speed of the calculation results is relatively fast.

[0139] As an optional implementation method, refer to Figure 8 Determining each of the multiple control components involves the following steps:

[0140] S410, in response to the control mode being hand control, based on an external input signal, when the gesture is identified as a preset gesture, the position offset of the preset gesture is determined.

[0141] Preset gestures may include, but are not limited to: extending the thumb and index finger with the web of the thumb pointing in the direction of the ray for easy clicking; extending only the index finger with the index finger pointing in the direction of the ray; and pinching the tips of the thumb and index finger together as a selection control gesture.

[0142] Step S410 can be achieved in several ways:

[0143] As an optional implementation, at least one consecutive frame of hand images is acquired; a gesture recognition network is used to identify the at least one consecutive frame of hand images to determine whether a preset gesture is included; from the hand images determined to contain the preset gesture, hand key points are extracted, including at least one point representing the fingertip, knuckles, wrist, and palm; based on the hand key points, the deflection angle of the preset gesture at the current hand sampling time relative to the previous hand sampling time is determined as the position offset of the preset gesture. This implementation can be used for interaction with far-field objects via gestures.

[0144] As another optional implementation, at least one consecutive frame of hand images is acquired; a gesture recognition network is used to recognize the at least one consecutive frame of hand images to determine whether a preset gesture is included; from the hand images determined to contain the preset gesture, hand key points are extracted, including at least one point representing the fingertip, knuckle, wrist, and palm; the movement distance of the preset hand key points is calculated, and the movement distance is used as the position offset of the preset gesture. The preset hand key points can be hand key points such as the fingertip of the index finger. This implementation can be used for interaction with near-field objects via gestures.

[0145] Regarding the two implementation methods described above, it should be noted that methods for acquiring at least one consecutive frame of hand images include, but are not limited to, using a visual sensor to track and capture the user's hand. Types of visual sensors include, but are not limited to, grayscale cameras, RGB cameras, depth cameras, and LiDAR. The gesture recognition network may include, but is not limited to, a pre-trained neural network-based recognition model. Methods for extracting hand key points from hand images containing a preset gesture include, but are not limited to, using a pre-trained hand key point extraction model to extract hand key points from hand images containing a preset gesture.

[0146] As an optional example, the deflection angle of the preset gesture may include at least one of the following: the deflection angle corresponding to the virtual ray rendered by the gesture, or the deflection angle in the direction the gesture is pointing. The direction the gesture is pointing may, for example, be the direction indicated by the ray drawn from the line connecting the wrist and the tip of the index finger.

[0147] In addition, depending on the type of visual sensor, different key point extraction models corresponding to binocular or monocular cameras can be used, and the specific model can be selected based on a trade-off between performance and power consumption.

[0148] S420. Determine the hand control component based on the position offset.

[0149] Step S420 can also be implemented in several ways:

[0150] Corresponding to one of the above optional implementations, for example, the position offset of the pointer in the virtual space is calculated based on the deflection angle and the depth information of the pointer in the virtual space rendered by the head-mounted display device, and is used as a hand control component.

[0151] Corresponding to the other optional implementation above, the position offset of the preset gesture can be directly used as the hand control component.

[0152] It is understandable that by combining gesture recognition networks and key point extraction models, the user's preset hand key points can be extracted, thereby determining the deflection angle or movement distance of the user's preset gesture. This implementation process is simple to operate, and the detection accuracy of preset hand key points can be improved by adjusting the model. In addition, using the position offset directly as the hand control component is more intuitive and concise, which is conducive to improving the user experience.

[0153] It should be noted that the head sampling frequency is higher than the eye sampling frequency, which in turn is higher than the hand sampling frequency. Since head sampling primarily uses an IMU (Integrated Measurement Unit), and IMUs can achieve sampling frequencies up to 1000Hz, the head sampling frequency is the highest. The eye sampling frequency uses a camera to capture images of the eyes, with a camera frequency of approximately 100Hz. The hand sampling frequency uses image recognition technology and a camera to capture images; because it requires complex calculations, its frequency is lower. Assuming the initial times are the head sampling time, eye sampling time, and hand sampling time, because the head sampling frequency is higher than the eye sampling frequency, which is higher than the hand sampling frequency, the head pose is updated at the next head sampling time, while the eye and hand sampling times have not yet arrived. At this point, the head control components are output. At the next eye sampling time, the eye position offset is updated, and the head control components are also updated; therefore, the head and eye control components are output at this point. When the next hand sampling time arrives and the hand pose is updated, the head control components (and possibly eye control components) are also updated. Therefore, the control components of the head (and possibly the eyes) and the hand are output at this time.

[0154] As an optional implementation, determining each of the multiple control components further includes: in response to the control mode being voice control, determining voice control information based on an external input signal; and determining the voice control component based on the voice control information.

[0155] As an optional example, in voice control mode, the external input signal is speech (e.g., the user's voice command), which can be identified from the user's speech using a speech recognition model. Then, it is determined whether the speech information matches a preset voice command. If they match, a control displacement command is generated based on the speech information. Based on the control displacement command, the position offset of the pointer in the virtual space rendered by the head-mounted display device is determined as the voice control component. For example, if the user says "left," the control pointer will shift 5 degrees to the left relative to the user's position; if the user says "move to the top left corner," the control pointer will move to the top left corner of the currently located virtual object (e.g., a virtual interface); if the user says "wow," there is no voice control component because it does not match the preset voice command.

[0156] It is understandable that voice control is more direct and convenient for users, thus improving the user's interactive experience when using head-mounted display devices.

[0157] As an optional implementation, determining each of the multiple control components further includes: in response to the control mode being peripheral control, determining output control information of the external device based on an external input signal; and determining the peripheral control component based on the output control information.

[0158] As an optional example, external devices may include, but are not limited to, gamepads, ring controllers, mobile phones, mice, and keyboards. The position offset of the pointer in the virtual space rendered by the head-mounted display device is determined by the control information output from different external devices, and this offset serves as the corresponding peripheral control component.

[0159] It is understandable that external devices can assist users in interacting with head-mounted displays, such as using a gamepad, keyboard, or mouse when playing games, thereby enhancing the gaming experience.

[0160] As another optional embodiment, the control method may further include: displaying interactive content corresponding to the response of the head-mounted display device to the control quantity. It is understood that displaying interactive content (e.g., displaying selected operable content) facilitates user interaction.

[0161] As an optional implementation, the set of working modes can be the set of working modes within the visible area of ​​the head-mounted display device. It is understood that selecting the set of working modes within the visible area ensures the visibility of the user's interactive operations, thus improving the user experience.

[0162] In summary, the control method for head-mounted display devices disclosed herein determines the control weight of each of the various control components used to control pointers in the virtual space rendered by the head-mounted display device, based on a set of working modes; using the control weights, the various control components are weighted and fused to obtain the control quantity used to control pointers in the virtual space rendered by the head-mounted display device; thereby enabling joint control of the head-mounted display device based on different control methods, that is, enabling users to achieve joint control based on head, eyes, hands, voice, and peripherals during human-computer interaction while wearing the head-mounted display device, thereby improving the user experience.

[0163] Exemplary device

[0164] It should be understood that the control methods for head-mounted display devices described in the foregoing embodiments herein can also be similarly applied to the control devices for head-mounted display devices described below for similar extensions. For simplicity, they are not described in detail.

[0165] Figure 9 This is a schematic diagram of one embodiment of the control device for a head-mounted display device disclosed herein. Figure 9As shown, the control device for a head-mounted display device includes: an acquisition module 110 configured to acquire a set of working modes, wherein the set of working modes represents at least one of the following: the display mode of the head-mounted display device, the interaction mode of the head-mounted display device, and a preset category application being active; a weight determination module 120 configured to determine, based on the set of working modes, the control weight of each of the multiple control components used to control pointers in the virtual space rendered by the head-mounted display device, wherein the multiple control components include output quantities corresponding to various control methods of the head-mounted display device; and a control execution module 130 configured to use the control weights to perform weighted fusion of the multiple control components to obtain a control quantity used to control pointers in the virtual space rendered by the head-mounted display device.

[0166] As an optional implementation, the weight determination module 120 is further configured to: in response to finding the set of operating modes in the control weight mapping table, obtain the initial control weight of each control component associated with the set of operating modes from the control weight mapping table.

[0167] As an optional implementation method, refer to Figure 10 The weight determination module 120 includes: an acquisition submodule 1210, configured to: acquire the mode weight of each working mode in the working mode set, and the component weight of each control component corresponding to each working mode; and a calculation module 1220, configured to: calculate the control weight of each control component based on the mode weight of each working mode and the component weight of each control component corresponding to each working mode.

[0168] As an optional implementation, the acquisition submodule 1210 is further configured to: in response to finding a set of working modes in the mode weight mapping table, obtain the mode weights corresponding to each working mode in the set of working modes from the mode weight mapping table.

[0169] As an optional implementation, the acquisition submodule 1210 is further configured to: sort each working mode in the working mode set from high to low according to the preset priority of each working mode to obtain a working mode sequence; select the corresponding mode weight for each working mode in the working mode sequence from the preset mode weight table according to the preset selection rule; wherein, the preset mode weight table includes at least one mode weight value; the preset selection rule is: according to the sorting of each working mode in the working mode sequence, select the top N mode weights with the largest mode weight values ​​from the mode weight table in sequence, where N is the number of working modes in the working mode sequence.

[0170] As an alternative implementation, the computing module 1220 is further configured to be implemented in any of the following ways:

[0171] Calculate the product of the mode weight and the corresponding component weight for each control component in each operating mode; sum the products of each control component in each operating mode to obtain the control weight for each control component; normalize the mode weight of each operating mode in the set of operating modes to obtain the normalized mode weight value for each operating mode; calculate the product of the normalized mode weight value and the corresponding component weight for each control component in each operating mode; sum the products of each control component in each operating mode to obtain the control weight for each control component; sum the products of each control component in each operating mode and the corresponding component weight to obtain the sum of the weight products of the set of operating modes corresponding to each control component; normalize the sum of the weight products of each control component to obtain the control weight for each control component.

[0172] As an optional implementation, the acquisition submodule 1210 is further configured to: acquire the component weights of each control component corresponding to each working mode from a preset component weight mapping table.

[0173] As an optional embodiment, the control device further includes: a control coefficient determination module, configured to: determine the control weight coefficient of the corresponding control component based on the signal-to-noise ratio of each control component among multiple control components; and a correction execution module, configured to: correct the control component based on the control weight coefficient to obtain the corrected control component.

[0174] As an optional embodiment, the control device further includes: a smoothing coefficient determination module, configured to: determine the smoothing intensity coefficient of the corresponding control component based on the signal-to-noise ratio of each control component among multiple control components; and a smoothing execution module, configured to: smooth the control component based on the smoothing intensity coefficient to obtain the smoothed control component.

[0175] As an optional embodiment, the control device further includes: an instruction adjustment module configured to: adjust the target control component according to the received user's adjustment instruction for the target control component, to obtain the adjusted target control component.

[0176] As an optional implementation, the smoothing execution module is further configured to perform low-pass filtering on the multiple control components using smoothing intensity coefficients corresponding to the multiple control components.

[0177] As an optional embodiment, the control device further includes a control component determination module configured to determine each of a plurality of control components, each control component being determined by at least one external input signal, the external input signal including at least one of the following: sensing information collected by a sensor associated with the head-mounted display device, and output information of an external device associated with the head-mounted display device.

[0178] As an optional implementation, the control component determination module includes a head control component module, configured to: in response to the control mode being head control, determine the pose change of the head-mounted display device based on the world coordinate system using an external input signal; and determine the head control component based on the pose change.

[0179] As an optional implementation, the head control component module is further configured to: acquire image information of the scanned area of ​​the head-mounted display device collected by the visual sensor associated with the head-mounted display device, and pose information of the head-mounted display device collected by the inertial sensor; based on the image information and pose information, establish an environmental map of the scanned area of ​​the head-mounted display device based on the world coordinate system in a real-time localization and mapping manner, and determine the real-time location of the head-mounted display device in the environmental map; based on the real-time location of the head-mounted display device in the environmental map, determine the rotation matrix between the pose of the head-mounted display device at the current head sampling moment and the pose at the previous head sampling moment, as the pose change quantity.

[0180] As an optional implementation, the control component determination module includes an eye control component module, configured to: determine the rotation angle of the head-mounted display device based on a rotation matrix; and calculate the position offset of the pointer in the virtual space as a head control component based on the rotation angle and the depth information of the pointer in the virtual space rendered by the head-mounted display device.

[0181] As an optional implementation, the eye control component module is further configured to: in response to the control mode being eye control, use an external input signal to determine the positional change of a preset eye key point of the user wearing the head-mounted display device; and determine the eye control component based on the positional change of the preset eye key point.

[0182] As an optional implementation, the eye control component module is further configured to: acquire at least one consecutive frame of eye images of a user wearing a head-mounted display device; utilize an eye keypoint extraction network to determine the position information of preset eye keypoints from the at least one consecutive frame of eye images, the position information of the preset eye keypoints including at least one of the following: a set of position information of preset keypoints for the left eye and a set of position information of preset keypoints for the right eye; wherein, the preset keypoints for the left eye include at least the left pupil, and the preset keypoints for the right eye include at least the right pupil; determine the position change amount of the corresponding preset eye keypoints based on the position information of the preset eye keypoints, the position change amount of the preset eye keypoints including at least one of the following: the angle change of the ray formed by the left pupil at the current eye sampling time relative to the previous eye sampling time, and the angle change of the ray formed by the right pupil at the current eye sampling time relative to the previous eye sampling time; wherein, the ray formed by the left pupil is a ray perpendicular to and passing through the geometric center of the left pupil; and the ray formed by the right pupil is a ray perpendicular to and passing through the geometric center of the right pupil.

[0183] As an optional implementation, the eye control component module is further configured to: calculate the position offset of the pointer in the virtual space based on the position change of the eye key points and the depth information of the pointer in the virtual space rendered by the head-mounted display device, as the eye control component.

[0184] As an optional implementation, the control component determination module includes a hand control component module, configured to: in response to the control mode being hand control, based on an external input signal, determine the position offset of a preset gesture when the gesture is identified as a preset gesture; and determine the hand control component based on the position offset.

[0185] As an optional implementation, the hand control component module is further configured as follows:

[0186] Acquire at least one consecutive frame of hand images; use a gesture recognition network to recognize at least one consecutive frame of hand images to determine whether a preset gesture is contained; extract hand key points from the hand images that contain the preset gesture, the hand key points include at least one point used to represent the fingertip, knuckle, wrist, and palm; based on the hand key points, determine the deflection angle of the preset gesture at the current hand sampling time relative to the previous hand sampling time, as the position offset of the preset gesture;

[0187] As an optional implementation, the hand control component module is further configured to: calculate the position offset of the pointer in the virtual space based on the deflection angle and the depth information of the pointer in the virtual space rendered by the head-mounted display device, as the hand control component.

[0188] As an optional implementation, the hand control component module is further configured to: acquire at least one consecutive frame of hand images; use a gesture recognition network to recognize the at least one consecutive frame of hand images to determine whether a preset gesture is included; extract hand key points from the hand images that are determined to contain the preset gesture, the hand key points including at least one point for representing the fingertip, knuckle, wrist, and palm; calculate the movement distance of the preset hand key points, and use the movement distance as the position offset of the preset gesture;

[0189] As an optional implementation, the hand control component module is further configured to directly use the position offset of a preset gesture as the hand control component.

[0190] As an optional implementation, the control component determination module includes a voice control component module, configured to: determine voice control information based on an external input signal in response to the control mode being voice control; and determine voice control components based on the voice control information.

[0191] As an optional implementation, the control component determination module includes a peripheral control component module, which is further configured to: in response to the control mode being peripheral control, determine the output control information of the external device based on the external input signal; and determine the peripheral control component based on the output control information.

[0192] As an optional embodiment, the control device further includes a display module configured to display interactive content corresponding to the response of the head-mounted display device to the control quantity.

[0193] As an optional implementation, the set of operating modes is the set of operating modes within the visible area of ​​the head-mounted display device.

[0194] In summary, the control device disclosed herein for a head-mounted display device determines the control weight of each of the multiple control components used to control pointers in the virtual space rendered by the head-mounted display device, based on a set of operating modes; using the control weights, the multiple control components are weighted and fused to obtain the control quantity used to control pointers in the virtual space rendered by the head-mounted display device; thereby enabling joint control of the head-mounted display device based on different control methods, that is, enabling users to achieve joint control based on head, eyes, hands, voice, and peripherals during human-computer interaction while wearing the head-mounted display device, thereby improving the user experience.

[0195] Exemplary electronic devices

[0196] In addition, this disclosure also provides an electronic device, including:

[0197] Memory, used to store computer programs;

[0198] A processor is configured to execute a computer program stored in the memory, wherein, when the computer program is executed, it implements the control method for a head-mounted display device described in any of the above embodiments of the present disclosure.

[0199] Figure 11 This is a schematic diagram of an application embodiment of the electronic device disclosed herein. Below, reference is made to… Figure 11 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0200] like Figure 11 As shown, the electronic device includes one or more processors and memory.

[0201] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0202] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and a processor may execute the program instructions to implement the control methods for head-mounted display devices and / or other desired functions described in the various embodiments of this disclosure above.

[0203] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0204] In addition, the input device may include, for example, a keyboard, a mouse, etc.

[0205] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0206] Of course, for the sake of simplicity, Figure 11Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0207] This disclosure also provides a head-mounted display device, which includes the control device for the head-mounted display device disclosed herein.

[0208] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the control methods for head-mounted display devices according to various embodiments of this disclosure as described in the foregoing portions of this specification.

[0209] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0210] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the control method for a head-mounted display device according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0211] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0212] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0213] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0214] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0215] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0216] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0217] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0218] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0219] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A control method for a head-mounted display device, comprising: Obtain a set of working modes, wherein the set of working modes is used to characterize at least one of the following: the display mode of the head-mounted display device, the interaction mode of the head-mounted display device, and the preset category of applications being active; Based on the set of working modes, the control weight of each control component among the multiple control components used to control the pointer in the virtual space rendered by the head-mounted display device is determined, wherein the multiple control components include the output quantities corresponding to the multiple control modes of the head-mounted display device respectively; Using the control weights, the multiple control components are weighted and fused to obtain a control quantity used to control the pointers in the virtual space rendered by the head-mounted display device; The step of determining the control weight of each control component among multiple control components used to control pointers in the virtual space rendered by the head-mounted display device, based on the set of working modes, includes: Obtain the mode weight of each working mode in the set of working modes, and the component weight of each control component corresponding to each working mode; Based on the mode weight of each working mode and the component weight of each control component corresponding to each working mode, the control weight of each control component is calculated.

2. The control method according to claim 1, wherein, The step of determining the control weight of each control component among multiple control components used to control pointers in the virtual space rendered by the head-mounted display device, based on the set of operating modes, includes: In response to finding the set of operating modes in the control weight mapping table, the initial control weight of each control component associated with the set of operating modes is obtained from the control weight mapping table.

3. The control method according to claim 1, wherein, The step of obtaining the mode weight of each working mode in the set of working modes includes: In response to finding the set of working modes in the mode weight mapping table, the mode weights corresponding to each working mode in the set of working modes are obtained from the mode weight mapping table.

4. The control method according to claim 1, wherein, The step of obtaining the mode weight of each working mode in the set of working modes includes: Based on the preset priority of each working mode, each working mode in the set of working modes is sorted from high to low to obtain a working mode sequence; According to the preset selection rules, select the corresponding mode weight for each working mode in the working mode sequence from the preset mode weight table; The preset mode weight table includes at least one mode weight value; The preset selection rule is as follows: according to the sorting of each working mode in the working mode sequence, select the top N mode weights with the largest mode weight values ​​from the mode weight table in sequence, where N is the number of working modes in the working mode sequence.

5. The control method according to claim 1, wherein, The calculation of the control weight for each control component based on the mode weight of each operating mode and the component weights of each control component corresponding to each operating mode includes any one of the following: Calculate the product of the mode weight and the corresponding component weight for each control component in each operating mode; sum the products of each control component in each operating mode to obtain the control weight for each control component. The mode weight of each working mode in the set of working modes is normalized to obtain the normalized mode weight value corresponding to each working mode; Calculate the product of the normalized mode weight value and the corresponding component weight for each control component in each operating mode; The products of each control component in each operating mode are added together to obtain the control weight corresponding to each control component; Add the product of the mode weight and the corresponding component weight of each control component in each working mode to obtain the sum of the weight products of each control component corresponding to the set of working modes. The sum of the weight products of each control component is normalized to obtain the control weight corresponding to each control component.

6. The control method according to claim 1, wherein, Obtaining the component weights of each control component corresponding to each operating mode includes: Obtain the component weights of each control component corresponding to each operating mode from the preset component weight mapping table.

7. The control method according to claim 2, wherein, The control method further includes: Based on the signal-to-noise ratio of each of the multiple control components, the control weight coefficient corresponding to the control component is determined; Based on the control weight coefficient, the control component is corrected to obtain the corrected control component.

8. The control method according to claim 2, wherein, The control method further includes: Based on the signal-to-noise ratio of each of the multiple control components, the smoothing intensity coefficient corresponding to the control component is determined; Based on the smoothing intensity coefficient, the control component is smoothed to obtain the smoothed control component.

9. The control method according to claim 2, wherein, The control method further includes: Based on the received user's adjustment instruction for the target control component, the target control component is adjusted to obtain the adjusted target control component.

10. The control method according to claim 9, wherein, The smoothing of the control components includes: using the smoothing intensity coefficient corresponding to the multiple control components to perform low-pass filtering on the multiple control components.

11. The control method according to claim 1, wherein, The control method further includes: Each of a plurality of control components is determined by at least one external input signal, the external input signal including at least one of the following: sensing information acquired by a sensor associated with the head-mounted display device, and output information of an external device associated with the head-mounted display device.

12. The control method according to claim 11, wherein, Determining each of the multiple control components includes: In response to the control method being head control, the pose change of the head-mounted display device based on the world coordinate system is determined using the external input signal; Based on the change in pose, the head control components are determined.

13. The control method according to claim 12, wherein, Determining the pose change of the head-mounted display device based on the world coordinate system using the external input signal includes: The image information of the scanned area of ​​the head-mounted display device collected by the visual sensor associated with the head-mounted display device, and the pose information of the head-mounted display device collected by the inertial sensor are acquired. Based on the image information and the pose information, an environmental map of the scanned area of ​​the head-mounted display device based on the world coordinate system is established using a real-time positioning and mapping method, and the real-time positioning of the head-mounted display device in the environmental map is determined. Based on the real-time positioning of the head-mounted display device in the environmental map, the rotation matrix between the pose of the head-mounted display device at the current head sampling moment and the pose at the previous head sampling moment is determined, and is used as the pose change amount. Determining the head control components based on the pose change includes: Based on the rotation matrix, the rotation angle of the head-mounted display device is determined; Based on the rotation angle and the depth information of the pointer in the virtual space rendered by the head-mounted display device, the position offset of the pointer in the virtual space is calculated and used as the head control component.

14. The control method according to claim 11, wherein, Determining each of the multiple control components includes: In response to the control method being eye control, the positional change of a preset eye key point of the user wearing the head-mounted display device is determined using the external input signal; Based on the positional changes of the preset key eye points, the eye control components are determined.

15. The control method according to claim 14, wherein, The step of determining the positional change of preset eye key points of a user wearing the head-mounted display device using the external input signal includes: Acquire at least one consecutive frame of eye images of a user wearing the head-mounted display device; Using an eye key point extraction network, the position information of preset eye key points is determined from the at least one consecutive frame of eye images. The position information of the preset eye key points includes at least one of the following: a set of position information of preset key points of the left eye and a set of position information of preset key points of the right eye; wherein, the preset key points of the left eye include at least the pupil of the left eye and the preset key points of the right eye include at least the pupil of the right eye. Based on the position information of the preset eye key points, determine the position change amount of the corresponding preset eye key points. The position change amount of the preset eye key points includes at least one of the following: the angle change of the ray formed by the left pupil at the current eye sampling time relative to the previous eye sampling time, and the angle change of the ray formed by the right pupil at the current eye sampling time relative to the previous eye sampling time. Wherein, the ray formed by the left pupil is a ray perpendicular to and passing through the geometric center of the left pupil; the ray formed by the right pupil is a ray perpendicular to and passing through the geometric center of the right pupil; The determination of eye control components based on the positional changes of the key eye points includes: Based on the positional change of the key eye points and the depth information of the pointer in the virtual space rendered by the head-mounted display device, the positional offset of the pointer in the virtual space is calculated as the eye control component.

16. The control method according to claim 11, wherein, Determining each of the multiple control components includes: In response to the control method being hand control, based on the external input signal, when the gesture is identified as a preset gesture, the position offset of the preset gesture is determined; The hand control component is determined based on the position offset.

17. The control method according to claim 16, wherein, The step of determining the position offset of the preset gesture based on the external input signal, when the gesture is identified as a preset gesture, includes: Acquire at least one consecutive frame of hand images; Using a gesture recognition network, the hand image of at least one consecutive frame is identified to determine whether it contains a preset gesture; From the determined hand image containing the preset gesture, key points of the hand are extracted, including at least one point for representing the fingertip, knuckle, wrist, and palm. Based on the key hand points, the deflection angle of the preset gesture at the current hand sampling time relative to the previous hand sampling time is determined, and used as the position offset of the preset gesture. Determining the control component based on the position offset includes: Based on the deflection angle and the depth information of the pointer in the virtual space rendered by the head-mounted display device, the position offset of the pointer in the virtual space is calculated as a hand control component.

18. The control method according to claim 16, wherein, The step of determining the position offset of the preset gesture based on the external input signal, when the gesture is identified as a preset gesture, includes: Acquire at least one consecutive frame of hand images; Using a gesture recognition network, the hand image of at least one consecutive frame is identified to determine whether it contains a preset gesture; From the determined hand image containing the preset gesture, key points of the hand are extracted, including at least one point for representing the fingertip, knuckle, wrist, and palm. Calculate the movement distance of preset hand key points, and use the movement distance as the position offset of the preset gesture; Determining the control component based on the position offset includes: The position offset of the preset gesture is directly used as the hand control component.

19. The control method according to claim 11, wherein, Determining each of the multiple control components includes: In response to the control method being voice control, voice control information is determined based on the external input signal; Based on the voice control information, determine the voice control components.

20. The control method according to claim 11, wherein, Determining each of the multiple control components includes: In response to the control mode being peripheral control, the output control information of the external device is determined based on the external input signal; Based on the output control information, determine the peripheral control components.

21. The control method according to any one of claims 1-20, wherein, The control method further includes: Based on the head-mounted display device's response to the control quantity, interactive content corresponding to the response is displayed.

22. The control method according to any one of claims 1-20, wherein, The set of working modes is the set of working modes within the visible area of ​​the head-mounted display device.

23. A control device for a head-mounted display device, comprising: The acquisition module is configured to acquire a set of working modes, wherein the set of working modes is used to represent at least one of the following: the display mode of the head-mounted display device, the interaction mode of the head-mounted display device, and the preset category of applications being active. The weight determination module is configured to: determine the control weight of each of the multiple control components used to control the pointer in the virtual space rendered by the head-mounted display device according to the set of working modes, wherein the multiple control components include the output quantities corresponding to the multiple control modes of the head-mounted display device respectively; The control execution module is configured to: use the control weights to perform weighted fusion of the multiple control components to obtain a control quantity for controlling the pointers in the virtual space rendered by the head-mounted display device; The weight determination module includes: The acquisition submodule is configured to: acquire the mode weight of each working mode in the set of working modes, and the component weight of each control component corresponding to each working mode; The calculation module is configured to calculate the control weight of each control component based on the mode weight of each working mode and the component weight of each control component corresponding to each working mode.

24. A head-mounted display device, the head-mounted display device comprising the control device for the head-mounted display device as described in claim 23.

25. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the control method according to any one of claims 1-22.

26. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the control method according to any one of claims 1-22.

27. A non-volatile computer program product storing instructions that, when executed by at least one data processor of at least one computing system, implement the control method according to any one of claims 1-22.

Images