Method, apparatus, and interaction system for interacting with a head-mounted display device

Abstract

Embodiments of the present disclosure disclose a method, an apparatus, and an interaction system for interacting with a head-mounted display device. The method includes: activating a joystick mode in response to detecting that a user wearing the head-mounted display device has performed a preset joystick mode gesture action; in a state where the joystick mode is activated, determining an origin position of a virtual joystick coordinate system based on at least one of a hand position of the user and a position of a display screen of the head-mounted display device, and determining the virtual joystick coordinate system based on the origin position of the virtual joystick coordinate system; determining an input key value of the virtual joystick based on a projection of a target hand of the user in the virtual joystick coordinate system. Embodiments of the present disclosure can perform joystick operations on the head-mounted display device without being equipped with a real joystick, improving the user experience.

Description

Technical Field

[0001] The present disclosure relates to the field of virtual reality technology, and in particular to a method, device and interactive system for interacting with a head-mounted display device. Background Art

[0002] In scenarios such as virtual reality (VR), augmented reality (AR), mixed reality (MR), or extended display, the terminal provides users with an interactive and immersive experience by building a virtual environment.

[0003] In some scenarios of interacting with a head mounted display device, it is usually necessary to interact with the head mounted display device through a joystick, but some head mounted display devices are not equipped with a joystick. How to implement joystick operation of the head mounted display device without a joystick is an urgent problem to be solved. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure is proposed. Embodiments of the present disclosure provide a method, an apparatus and an interactive system for interacting with a head mounted display device.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for interacting with a head mounted display device is provided, comprising:

[0006] In response to detecting that a user wearing the head mounted display device performs a preset gesture action, activating a joystick mode;

[0007] In a state where the joystick mode is activated, determining the origin position of the virtual joystick coordinate system based on at least one of the user's hand position and the position of the display screen of the head mounted display device, and determining the virtual joystick coordinate system based on the origin position of the virtual joystick coordinate system;

[0008] Based on the projection of the user's target hand on the virtual joystick coordinate system, the input key value of the virtual joystick is determined.

[0009] According to a second aspect of an embodiment of the present disclosure, there is provided an apparatus for interacting with a head mounted display device, comprising:

[0010] A joystick mode activation module, used to activate the joystick mode when detecting that the user has performed a preset gesture action;

[0011] A virtual joystick coordinate system determination module, configured to determine the origin position of the virtual joystick coordinate system based on at least one of the user's hand position and the position of the display screen of the head-mounted display device in a state where the joystick mode is activated, and determine the virtual joystick coordinate system based on the origin position of the virtual joystick coordinate system;

[0012] An input key value determination module, configured to determine the input key value of the virtual joystick based on the projection of the user's target hand on the virtual joystick coordinate system.

[0013] According to a third aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium storing a computer program for executing the method for interacting with a head-mounted display device in the first aspect above.

[0014] According to a fourth aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0015] A processor;

[0016] A memory for storing executable instructions of the processor;

[0017] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for interacting with the head-mounted display device in the first aspect above.

[0018] The technical solutions of the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Description of the Drawings

[0019] By describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, the above and other objects, features, and advantages of the present disclosure will become more apparent. The drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation to the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 It is an exemplary system architecture diagram that can be applied to embodiments of the method or apparatus for interacting with a head-mounted display device of the present disclosure;

[0021] Figure 2 It is a flowchart of a method for interacting with a head-mounted display device in an embodiment of the present disclosure;

[0022] Figure 3 It is a flowchart of step S4 in an embodiment of the present disclosure;

[0023] Figure 4 It is a flowchart of step S4 in another embodiment of the present disclosure;

[0024] Figure 5 It is a schematic flowchart of step S6 in an embodiment of the present disclosure;

[0025] Figure 6 It is a schematic diagram of normalization processing based on an input value in an example of the present disclosure;

[0026] Figure 7 It is a schematic diagram of a gesture action for controlling a virtual handle or a virtual joystick in an example of the present disclosure;

[0027] Figure 8 It is a structural block diagram of a device for interacting with a head-mounted display device in an embodiment of the present disclosure;

[0028] Figure 9 It is a structural block diagram of an electronic device in an embodiment of the present disclosure. Detailed implementation manners

[0029] Next, example embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments of the present disclosure. It should be understood that the present disclosure is not limited by the example embodiments described herein.

[0030] It should be noted that: Unless otherwise specifically stated, the relative arrangements, numerical expressions, and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0031] Those skilled in the art can understand that terms such as "first", "second", etc. in the embodiments of the present disclosure are only used to distinguish different steps, devices, or modules, etc., and neither represent any specific technical meaning nor indicate an inevitable logical order between them.

[0032] It should also be understood that in the embodiments of the present disclosure, "a plurality of" may refer to two or more, and "at least one" may refer to one, two, or more.

[0033] It should also be understood that for any component, data, or structure mentioned in the embodiments of the present disclosure, without clear definition or contrary indication in the context, it can generally be understood as one or more.

[0034] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present disclosure generally represents an "or" relationship between the associated objects before and after.

[0035] It should also be understood that the descriptions of the various embodiments in this disclosure emphasize the differences between the various embodiments, and the similarities or resemblances among them can be referred to each other. For the sake of brevity, they will not be elaborated one by one.

[0036] The following description of at least one exemplary embodiment is actually merely illustrative and in no way limits the present disclosure or its application or use.

[0037] Techniques, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the said techniques, methods, and devices should be regarded as part of the specification.

[0038] It should be noted that like reference numerals and letters denote like items in the following figures, and thus, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0039] Embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate together with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, and so on.

[0040] Terminal devices, computer systems, servers and other electronic devices can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, object programs, components, logics, data structures, etc., which perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment where tasks are executed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0041] Figure 1 is an exemplary system architecture diagram of an embodiment of a method or apparatus for interacting with a head-mounted display device that can be applied to the present disclosure.

[0042] As Figure 1As shown in the figure, the system architecture may include a head-mounted display device 1, a network 2, and a server 3. The network 2 may serve as a medium for the communication link between the head-mounted display device 1 and the server 3. The network 2 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0043] The above-mentioned head-mounted display device 1 may be an electronic device with an image display function, including but not limited to AR smart glasses, VR smart glasses, XR smart glasses, etc. Since head-mounted display devices such as AR smart glasses, VR smart glasses, and XR smart glasses move with the user, calculations related to the positioning of the head-mounted display device are usually required during the image display process. The head-mounted display device 1 may be an all-in-one device, that is, the head-mounted display device 1 can independently undertake the computing function, can install various client applications, and can display its own provided image and video data, etc. Alternatively, the head-mounted display device 1 may also be a split device that cooperates with a terminal device to implement the image display function. In this case, the terminal device can undertake the computing function of the head-mounted display device, can install various client applications, and can provide the head-mounted display device with the images and video data to be displayed, etc. Here, for the convenience of representation, the all-in-one device and the split device can be collectively referred to as the head-mounted display device.

[0044] The above-mentioned head-mounted display device 1 may have sensors for collecting its own motion data and environmental data. The own motion data may be, for example, the angular velocity, acceleration, and magnetic field data of the head-mounted display device 1, etc. Correspondingly, the sensors may be IMUs and magnetometers, etc. The environmental data may be the image data of the environment where the head-mounted display device is located, the distance data from surrounding objects, the magnetic field data of the surrounding environment, etc. Correspondingly, the sensors may be cameras, distance sensors, magnetometers, etc. The data measured by these sensors can be used for the positioning calculation of the head-mounted display device 1 itself, the user's body parts, and the surrounding environment, etc.

[0045] The above-mentioned head-mounted display device 1 may provide various services. For example, after the head-mounted display device is started, the display screen of the head-mounted display device can be operated in six degrees of freedom according to the gesture actions of the user wearing the head-mounted display device.

[0046] It should be noted that the method for interacting with the head-mounted display device provided in the embodiments of the present disclosure is generally executed by the head-mounted electronic device 1. Correspondingly, the device for interacting with the head-mounted display device is generally set in the head-mounted electronic device 1.

[0047] Optionally, the method for interacting with the head-mounted display device provided in the embodiments of the present disclosure may also be executed by a terminal device connected to the head-mounted electronic device 1. Correspondingly, the device for interacting with the head-mounted display device may also be set in the terminal device.

[0048] Optionally, the method for interacting with a head-mounted display device provided by an embodiment of the present disclosure can be jointly executed by a head-mounted electronic device 1 and a terminal device connected to the head-mounted electronic device 1. For example, the head-mounted display device can use its own sensors to collect data for subsequent calculations. For example, it can collect data for calculations such as gesture action detection and recognition, hand position calculation, and projection of the target head. The terminal device can undertake the calculation work. Correspondingly, for example, the device for interacting with the head-mounted display device can also be partially arranged in the head-mounted display device and partially arranged in the terminal device.

[0049] The method for interacting with a head-mounted display device provided by an embodiment of the present disclosure can establish a virtual handle and a virtual joystick for a user wearing the head-mounted display device. The user can operate the virtual handle and the virtual joystick through gesture actions, thereby realizing interaction with the display screen of the head-mounted display device. Among them, the virtual joystick can be set on the virtual handle, so that the virtual handle and the virtual joystick form an integral whole. Or the virtual joystick and the virtual handle can be separately set.

[0050] When the user performs a preset joystick mode gesture action, the joystick mode can be activated. In the state where the joystick mode is activated, the input key value of the joystick can be determined by detecting the user's hand position. Optionally, a virtual joystick coordinate system can be established based on at least one of the user's handle position and the position of the display screen of the head-mounted display device. Optionally, the input key value of the virtual joystick can be determined based on the projection of the user's target hand in the virtual joystick coordinate system.

[0051] When the user performs a preset handle mode gesture action, the handle mode can be activated. In the state of the handle mode, the input key value of the virtual handle is determined by detecting the user's handle mode gesture action.

[0052] It can be understood that in an embodiment of the present disclosure, at least one of the joystick mode and the handle mode can be executed.

[0053] Optionally, the position of the virtual handle in the world coordinate system can also be determined based on the specified hand joint position of the user, and a virtual handle coordinate system is established for the user to determine the rotation matrix of the virtual handle coordinate system relative to the world coordinate system. It can be understood that in an embodiment of the present disclosure, the activation operations of the above joystick mode and handle mode do not need to be executed, and the position and rotation of the virtual handle can be determined by collecting images of the user's hand.

[0054] It should be noted that the "hand position" in the above text can refer to the position of the hand when the user performs virtual joystick operations, which may include the palm position and finger positions of the hand. The "target hand" in the above text can refer to the hand that performs virtual joystick operations or handle operations. The "virtual joystick" in the above text can refer to a virtual object formed in the virtual world for the user to perform joystick operations on the display screen of the head-mounted display device. The "virtual handle" in the above text can refer to a virtual object formed in the virtual world for the user to perform handle operations on the display screen of the head-mounted display device. The above "virtual joystick" and "virtual handle" can be displayed on the display screen or not, as long as the corresponding operations can be triggered in response to the user's actions.

[0055] It should also be pointed out that although the solution of the present disclosure can be applied to a head-mounted display device, it does not exclude that the solution can also be applied to the server 3, which can be a background server. In the case where the solution of the present disclosure is applied to the server 3, the server activates the joystick mode in response to detecting that a user wearing a head-mounted display device has performed a preset joystick mode gesture action; in the state where the joystick mode is activated, the origin position of the virtual joystick coordinate system is determined based on at least one of the user's hand position and the display screen of the head-mounted display device, and the virtual joystick coordinate system is determined based on the origin position of the virtual joystick coordinate system; the input key value of the virtual joystick is determined based on the projection of the user's target hand in the virtual joystick coordinate system. In this case, the method for interacting with the head-mounted display device can be executed by the server 3, and correspondingly, the device for interacting with the head-mounted display device can also be set in the server 3.

[0056] Exemplary method

[0057] Figure 2 It is a schematic flowchart of a method for interacting with a head-mounted display device in an embodiment of the present disclosure. This embodiment can be applied to an electronic device, such as Figure 2 shown, and includes the following steps:

[0058] S0: Map a virtual handle and a virtual joystick that can perform six-degree-of-freedom control on the head-mounted display device, and map the operations on the virtual handle or virtual joystick to the user's specified gesture actions. For any operation on the virtual handle or virtual joystick, such as a button operation, a trigger operation, or a joystick operation, it is mapped to a corresponding gesture action.

[0059] S2: Activate the joystick mode in response to detecting that a user wearing a head-mounted display device has performed a preset joystick mode gesture action.

[0060] A camera can be set on the head-mounted display device. An image including the user's hand is captured by the camera. When it is recognized through the image that the user has performed a preset joystick mode gesture action for activating the joystick mode, the joystick mode is activated. Exemplarily, the preset gesture action for activating the joystick mode can be a single fixed gesture action, such as the gesture action representing "six", or the gesture action representing "making a phone call". The preset gesture action for activating the joystick mode can also be a set of continuous actions, such as the user's left thumb rotating counterclockwise or clockwise for one circle.

[0061] S4: In the state where the joystick mode is activated, determine the origin position of the virtual joystick coordinate system based on at least one of the user's hand position and the position of the display screen of the head-mounted display device, and determine the virtual joystick coordinate system based on the origin position of the virtual joystick.

[0062] The origin position of the virtual joystick coordinate system can be determined based on the specified position of the user's hand. For example, the joint position of a specified finger of the user can be used as the origin position of the virtual joystick coordinate system, or the average position of the joint positions of multiple fingers can be used as the origin position of the virtual joystick coordinate system. The origin position of the virtual joystick can also be determined based on the position of the display screen of the head-mounted display device. For example, for a head-mounted display device providing a two-dimensional display screen, the center point position of the display screen, or a preset position in the lower left corner of the display screen, or a preset position in the lower right corner of the display screen can be used as the origin position of the virtual joystick coordinate system. For another example, for a head-mounted display device providing a three-dimensional display screen, a two-dimensional display plane can be selected from the three-dimensional display screen, and the center point position of the display plane, or a preset position in the lower left corner of the display plane, or a preset position in the lower right corner of the display plane can be used as the origin position of the virtual joystick coordinate system.

[0063] After determining the origin position of the virtual joystick, the virtual joystick coordinate system can be established based on the origin position. Exemplarily, the X-axis of the virtual joystick coordinate system and the Y-axis perpendicular to the X-axis can be defined according to the determined origin position of the virtual joystick coordinate system.

[0064] S6: Determine the input key value of the virtual joystick based on the projection of the user's target hand in the virtual joystick coordinate system. The "target hand" in this embodiment can refer to the hand that manipulates the virtual joystick.

[0065] The projection direction can be used as the manipulation direction of the virtual joystick, the operation key value of the joystick is determined based on the end position of the projection, and then the input key value of the virtual joystick is determined according to the manipulation direction and the operation key value of the virtual joystick.

[0066] In this embodiment, when a user wearing a head-mounted display device activates the joystick mode through a preset gesture, a virtual joystick coordinate system is determined based on at least one of the user's hand position and the display screen of the head-mounted display device. Then, according to the projection of the user's target hand in the virtual joystick coordinate system, the input key value of the virtual joystick is determined, so that the head-mounted display device can be operated in a joystick manner without being equipped with a real joystick, improving the user experience.

[0067] Figure 3 It is a schematic flowchart of step S4 in an embodiment of the present disclosure. As Figure 3 shown, step S4 may include:

[0068] S4-A-2: Determine the origin position of the virtual joystick coordinate system based on the finger joint positions of the user when the joystick mode is activated.

[0069] The joint position of the root of the user's left index finger at the moment when the joystick mode is activated can be determined as the origin position of the virtual joystick coordinate system. The joint position of the root of the user's left thumb at the moment when the joystick mode is activated can also be determined as the origin position of the virtual joystick coordinate system. A position associated with the joint position of the designated finger part of the user at the moment when the joystick mode is activated can also be determined as the origin position of the virtual joystick coordinate system. The finger joint positions of the user and the origin position of the virtual joystick coordinate system are pre-mapped, and the mapping relationship can be adjusted. For example, for a user who is used to controlling the joystick with the right hand, the above-mentioned joint position of the root of the left index finger can be replaced with the joint position of the root of the right index finger, or the above-mentioned joint position of the root of the left thumb can be replaced with the joint position of the root of the right thumb.

[0070] S4-A-4: Determine the direction parallel to the first coordinate axis of the coordinate system of the head-mounted display device passing through the origin position of the virtual joystick coordinate system as the first coordinate axis direction of the virtual joystick coordinate system, and determine the direction parallel to the second coordinate axis of the coordinate system of the head-mounted display device passing through the origin position of the virtual joystick coordinate system as the second coordinate axis direction of the virtual joystick coordinate system.

[0071] The direction parallel to the X-axis of the coordinate system of the head-mounted display device passing through the origin position of the virtual joystick coordinate system can be used as the X-axis direction of the virtual joystick coordinate system, and the direction parallel to the Z-axis of the coordinate system of the head-mounted display device passing through the origin position of the virtual joystick coordinate system can be used as the Y-axis direction of the virtual joystick coordinate system.

[0072] In this embodiment, a mapping relationship can be established between the finger joint positions of the user and the virtual joystick, so as to determine the virtual joystick coordinate system, enabling the user to control the virtual joystick through finger joint movements, providing a good user experience. By using the finger joint position at the activation moment as the origin of the virtual joystick coordinate system, the subsequent hand movements of the user after activating the joystick mode can be directly mapped to the key values of the joystick, with a continuous operation process and a fast response speed.

[0073] Figure 4 It is a schematic flowchart of step S4 in another embodiment of the present disclosure. As Figure 4 shown, step S4 may include:

[0074] S4-B-2: Determine the origin position of the virtual joystick coordinate system based on a preset position in a preset display plane of the display screen.

[0075] When the display screen of the head-mounted display device is a three-dimensional display screen, the preset display plane may be a two-dimensional display plane in the three-dimensional display screen. When the display screen of the head-mounted display device is a two-dimensional display screen, the preset display plane is the two-dimensional display screen itself.

[0076] The preset display plane may include a rectangular screen, a circular screen, or other regularly shaped screens, such as a centrally symmetric screen. Based on the data of the preset display plane, the positions of all pixel points included in the preset display plane can be determined, and the position of any pixel point in the preset display plane can be determined as the origin position of the virtual joystick coordinate system. For example, the center position of the preset display plane can be determined as the origin position of the virtual joystick coordinate system, or the position of a corner point in the preset display plane can be determined as the origin position of the virtual joystick coordinate system.

[0077] S4-B-4: Determine the first preset direction of the preset display plane as the first coordinate axis direction of the virtual joystick coordinate system, and determine the second preset direction of the preset display plane as the second coordinate axis direction of the virtual joystick coordinate system.

[0078] When the preset display plane is a rectangular screen, the direction passing through the origin position of the virtual joystick coordinate system and parallel to one side of the rectangular screen can be used as the first preset direction, that is, the X-axis direction of the virtual joystick coordinate system. The direction passing through the origin position of the virtual joystick coordinate system and parallel to the other side of the rectangular screen can be used as the second preset direction, that is, the Y-axis direction of the virtual joystick coordinate system. Among them, one side of the rectangular screen is perpendicular to the other side of the rectangular screen.

[0079] When the display screen is a circular screen, a direction passing through the origin position and along a diameter direction of the circular screen can be used as the first preset direction, that is, the X-axis direction of the virtual joystick coordinate system. A direction passing through the origin position and perpendicular to the X-axis of the virtual joystick coordinate system and along a diameter direction can be used as the second preset direction, that is, the Y-axis direction of the virtual joystick coordinate system.

[0080] When the display screen is a trapezoid, a direction passing through the origin position and parallel to the bottom side of the trapezoid screen can be used as the first preset direction, that is, the X-axis direction of the virtual joystick coordinate system. A direction passing through the origin position and perpendicular to the bottom side of the trapezoid screen can be used as the second preset direction, that is, the Y-axis direction of the virtual joystick coordinate system.

[0081] In this embodiment, determining the virtual joystick coordinate system based on the display screen of the head-mounted display device can bring a feeling similar to touch operation when the user operates using the virtual joystick, and the user experience is good.

[0082] Figure 5 It is a schematic flowchart of step S6 in an embodiment of the present disclosure. As Figure 5 shown, step S6 may include:

[0083] S6-2: Obtain the orthogonal projection coordinates of the target finger of the target hand in the virtual joystick coordinate system. The orthogonal projection coordinates of the target finger in the virtual joystick coordinate system may include the orthogonal projection coordinates of the fingertip of the target finger in the virtual joystick coordinate system.

[0084] S6-4: Perform normalization processing on the orthogonal projection coordinates to determine the input key value of the virtual joystick.

[0085] Based on the orthogonal projection coordinates of the fingertip of the target finger in the virtual joystick coordinate system, determine the manipulation direction and manipulation distance of the virtual joystick. The manipulation direction can be represented by an angle. For example, using 90° represents that the user moves the virtual joystick upward, and using 270° represents that the user moves the virtual joystick downward, and so on. After performing normalization processing on the manipulation distance, the input key value of the virtual joystick can be determined according to the manipulation direction and the normalized manipulation distance.

[0086] In this embodiment, for the orthogonal projection coordinates of the target finger of the user in the virtual joystick coordinate system and performing normalization processing, the input key value of the virtual joystick can be obtained quickly and accurately.

[0087] In an embodiment of the present disclosure, step S6-4 may include: Based on the preset minimum absolute value of the coordinates, the preset numerical processing width, and the preset overshoot coefficient of the virtual joystick coordinate system, perform normalization processing on the orthogonal projection coordinates to determine the input key value of the virtual joystick.

[0088] The preset minimum absolute coordinate value can limit the minimum value of the absolute value of the orthographic projection coordinates. For example, when the minimum absolute coordinate value is V, when the coordinate value of one coordinate axis (such as the x-axis or y-axis) of the orthographic projection coordinates is within [-V, V], it is very likely due to the slight shaking of the user's finger. The user may not want the head-mounted display device to respond to the manipulation caused by the slight shaking of the finger. Therefore, by presetting the minimum absolute coordinate value, the probability of misoperation can be effectively reduced.

[0089] The preset numerical processing width can limit the value range of the input key value. For example, when the preset numerical processing width is 1, the value range of the input key value can be [-1, 1].

[0090] The preset overshoot coefficient can limit that when the coordinate value of the orthographic projection coordinates slightly exceeds the standard coordinate value range, the input key value corresponding to the coordinate value of the orthographic projection coordinates is still determined as the input key value corresponding to the boundary value of the standard coordinate value range. For example, when the standard coordinate value range is [-K, K], when the coordinate value of a certain coordinate axis (such as the x-axis or y-axis) of the orthographic projection coordinates is within [-K - L, -K), the input key value corresponding to this coordinate value is equal to the input key value corresponding to the coordinate value -K. When the coordinate value of a certain coordinate axis (such as the x-axis or y-axis) of the orthographic projection coordinates is within (K, K + L], the input key value corresponding to this coordinate value is equal to the input key value corresponding to the coordinate value K. Both K and L can be integers greater than 0. Optionally, L can be less than K, which can enable processing even when slightly exceeding the standard coordinate value range. It can be understood that the size relationship between L and K can also not be specifically limited. It can be seen that by presetting the overcharge coefficient, the user's slightly out-of-range operation can be regarded as a normal operation, improving the response probability of the user's intended operation.

[0091] In this embodiment, determining the input key value of the virtual joystick based on the preset minimum absolute coordinate value, preset numerical processing width, and preset overshoot coefficient of the virtual joystick coordinate system can reduce the probability of user misoperation and improve the response probability of the user's intended operation, providing a good user experience.

[0092] In an embodiment of the present disclosure, based on the preset minimum absolute coordinate value, preset numerical processing width, and preset overshoot coefficient of the virtual joystick coordinate system, normalizing the orthographic projection coordinates and determining the input key value of the virtual joystick may include:

[0093] S6-4-2: Determine the input value based on the orthographic projection coordinates. Denote the orthographic projection coordinate value of the fingers on the x-axis as x i . x i can be used as the input value.

[0094] S6-4-4: Perform normalization processing based on the input value to determine the input key value of the virtual joystick.

[0095] In one embodiment of the present disclosure, step S6-4-4 may include:

[0096] Determine a first numerical processing range based on the preset minimum absolute value of coordinates, determine a second numerical processing range based on the preset minimum absolute value of coordinates and the preset numerical processing width, and determine a third numerical processing range based on the preset minimum absolute value of coordinates, the preset numerical processing width, and the preset overcharge coefficient. Wherein, the maximum numerical boundary of the first numerical range is less than the maximum numerical boundary of the second numerical range, and the minimum numerical boundary of the first numerical range is greater than the minimum numerical boundary of the second numerical range, the maximum numerical boundary of the second numerical range is less than the maximum numerical boundary of the third numerical range, and the minimum numerical boundary of the second numerical range is greater than the minimum numerical boundary of the third numerical range.

[0097] If the input value is within the first numerical processing range, determine that the input key value is 0.

[0098] If the input value is outside the first numerical processing range and within the second numerical processing range, determine the input key value based on the input value, the preset minimum absolute value of coordinates, and the preset numerical processing width.

[0099] If the input value is outside the second numerical processing range and within the third numerical processing range, determine that the input key value is one of 1 and -1 based on the size relationship between the input value and 0.

[0100] If the input value is outside the third numerical processing range, determine that the input key value is 0.

[0101] Figure 6 It is a schematic diagram of normalization processing based on the input value in an example of the present disclosure. As Figure 6 shown, in this example, the preset minimum absolute value of coordinates is taken as 0.2, the preset numerical processing width is taken as 1, and the preset overcharge coefficient is taken as 1.2.

[0102] When x i is within the rectangle filled with vertical bars (i.e., the first preset numerical range), that is, -0.2 < x i ≤ 0.2, then the input key value = 0.

[0103] When -1.2 < x i ≤ -0.2 or 0.2 < x i ≤ 1.2 (i.e., the input value is outside the first numerical processing range and within the second numerical processing range. At this time, x i is outside the rectangle filled with vertical bars and within the rectangle filled with horizontal bars), the specific method for determining the value of the input key value is as follows: When -1.2 < x i ≤ -0.2, the value of the input key value = xi +0.2; when 0.2 < x i ≤ 1.2, the value of the input key = x i -0.2. This can make the input key value within [-1, 1].

[0104] When -1.4 < x i ≤ -1.2 or 1.2 < x i ≤ 1.4 (that is, the input value is outside the second numerical processing range and within the third numerical processing range. At this time, x i is outside the rectangle filled with horizontal bars and within the rectangle filled with diagonal squares), the method for determining the value of the input key is as follows: When -1.4 < x i ≤ -1.2, the value of the input key = -1; when 1.2 < x i ≤ 1.4, the value of the input key = 1.

[0105] When x i < -1.4 or x i > 1.4 (that is, the input value is outside the third numerical processing range. At this time, x i is outside the rectangle filled with diagonal squares), the method for determining the value of the input key is as follows: When x i < -1.4, the value of the input key = 0; when x i > 1.4, the value of the input key = 0.

[0106] It should be noted that the processing method for the orthogonal projection coordinate y i is similar to the processing method for the orthogonal projection coordinate x i .

[0107] In this embodiment, the first numerical processing range, the second numerical processing range, and the third numerical processing range are determined based on the preset minimum absolute value of the coordinate, the preset numerical processing width, and the preset overshoot coefficient of the virtual joystick coordinate system. Through the first numerical processing range, when the user's finger shakes slightly without manipulating the virtual joystick, the probability of misoperation can be reduced. Through the second numerical processing range, when the user performs basic manipulation on the virtual joystick, a response can be made. Through the third numerical processing range, the user's small-scale out-of-range operation can be regarded as a normal operation.

[0108] When the user manipulates the virtual joystick with a large amplitude, a response can be made according to the maximum allowable manipulation range of the virtual joystick. When the input value is greater than the third numerical processing range, it can be characterized that the user is not operating the virtual joystick. At this time, no response is made to the operation of the virtual joystick, and misoperation can be effectively avoided.

[0109] The method for interacting with the head-mounted display device in the embodiment of the present disclosure may further include:

[0110] S8: Activate the handle mode in response to detecting that the user has performed a preset handle mode gesture action.

[0111] S10: In the state where the handle mode is activated, determine the input key values of the virtual handle based on the handle mode gesture action. Among them, the handle mode gesture action includes at least one of a key pressing action and a grasping action. The input key values of the virtual handle include the input key values of the key pressing action and the input key values of the grasping action.

[0112] Before turning on the handle mode, the mapping relationship between controlling the virtual handle or the virtual joystick through gesture actions can be preset in advance. Among them, the gesture actions in the mapping relationship can include handle mode gesture actions and joystick mode gesture actions. Through the handle mode gesture action, the key operation and trigger operation of the virtual handle can be realized. Through the joystick mode gesture action, the joystick operation of the virtual joystick can be realized.

[0113] Figure 7 It is a schematic diagram of the gesture actions for controlling the virtual handle or the virtual joystick in an example of the present disclosure. As Figure 7 shown, the gesture action in (A) in Figure 7 can be mapped to the first key operation of the virtual handle (for example, the key pressing operation of the A key); the gesture action in (B) in Figure 7 can be mapped to the grasping key for grasping the virtual handle; the gesture action in (C) in Figure 7 can be mapped to pressing the trigger of the virtual handle; the gesture action in (D) in Figure 7 can be mapped to the second key operation of the virtual handle (for example, the key pressing operation of the Home key); the gesture action in (E) in Figure 7 can be mapped to activating the joystick mode; the gesture action in (F) in Figure 7 can be mapped to the third key operation of the virtual handle (for example, the key pressing operation of the B key).

[0114] After providing the display screen through the head-mounted display device, if it is detected that the user has performed a preset handle mode gesture action, the handle mode can be activated.

[0115] In the state where the handle mode is turned on, if it is detected that the user performs one of the handle mode gesture actions in (A), (B), (C), (D), and (F) in Figure 7 , the input key values of the virtual handle can be determined based on the handle mode gesture action.

[0116] In this embodiment, when the user uses the wearable device, the handle mode can be activated when it is detected that the user performs a preset handle mode gesture action. In the state where the handle mode is activated, the input key value of the virtual handle can be determined according to the user's gesture action, so that the wearable device can be operated with a handle without a real handle.

[0117] In an embodiment of the present disclosure, determining the input key value of the virtual handle based on the handle mode gesture action may specifically include:

[0118] S10-2: Determine the input distance value for the virtual handle based on the handle mode gesture action.

[0119] The distance value between two finger joints when the user performs the handle mode gesture action can be recognized by image recognition as the input distance value. For example, the distance value between two finger joints during the trigger operation of the virtual handle, or the distance value between the positions of the two hands during the grasping operation, is used as the input distance value.

[0120] S10-4: Normalize the input distance value based on the input distance value, a preset critical coefficient, a maximum processing distance threshold, and a minimum processing distance threshold to determine the input key value of the virtual handle.

[0121] The critical coefficient threshold can be defined as a decimal between 0 and 1, the maximum processing distance threshold is represented as max, and the minimum processing distance threshold is represented as min. The difference between max and min can be used as the processing distance, represented by length, and length is not equal to 0.

[0122] If length is not greater than 0, then output 0 (no operation is triggered), indicating that at least one of the maximum processing distance threshold and the minimum processing distance threshold is set incorrectly, and abort; if the input distance value ≤ max – length * threshold, indicating that the user presses the button of the virtual handle and reaches the maximum button depth, then output 1 and abort; if the input distance value < max, then output (max – input distance value) / (threshold * length), indicating that the user presses the button of the virtual handle but does not reach the maximum button depth, and abort; if the input distance value ≥ max, then output 0, indicating that the maximum processing distance threshold is set incorrectly, and abort. In an example of the present disclosure, min = 5mm, max = 15mm, and threshold = 0.5.

[0123] In this embodiment, based on the gesture actions in the handle mode, the input distance value of the virtual handle can be accurately obtained. Based on the input distance value, the preset critical coefficient, the maximum processing distance threshold, and the minimum processing distance threshold, the input distance value can be normalized, enabling fuzzy matching between the gesture actions in the handle mode and the input of the virtual handle, and making the input key value of the virtual handle within a reasonable range, thereby improving the operation success rate of the virtual handle and providing a good user experience.

[0124] The method for interacting with a head-mounted display device according to an embodiment of the present disclosure may further include:

[0125] S12: Determine the position of the virtual handle based on the specified hand joint positions of the user wearing the head-mounted display device.

[0126] The palm position, the root joint position of the index finger, or the root joint position of the middle finger can be determined as the position of the virtual handle. Alternatively, the average value of the joint positions of multiple fingers can be determined as the position of the virtual handle.

[0127] S14: Establish a virtual handle coordinate system for the user and determine the rotation matrix of the virtual handle coordinate system relative to the world coordinate system.

[0128] First, determine the forward vector and the secondary vector in the world coordinate system, where the forward vector and the secondary vector are not parallel. Then calculate the cross product of the secondary vector and the forward vector to obtain a vector that is orthogonal to both the forward vector and the secondary vector, denoted as the left vector. Next, calculate the cross product of the forward vector and the left vector to obtain a vector that is orthogonal to the forward vector, the secondary vector, and the left vector pairwise, denoted as the upward vector. Then, based on the forward vector, the left vector, and the upward vector, establish a 3×3 matrix. The first row elements of the matrix are the forward vector, the second row elements are the left vector, and the third row elements are the upward vector. This matrix can be used as the rotation matrix of the handle coordinate system relative to the world coordinate system.

[0129] In this embodiment, the position of the virtual handle can be determined based on the specified hand joint positions of the user, and the rotation matrix of the handle coordinate system of the virtual handle relative to the world coordinate system (i.e., the attitude of the virtual handle) can be determined. Furthermore, based on the position and attitude of the virtual handle, the specified functions of the virtual handle can be realized. For example, based on the position of the virtual handle, a ray pointing to the display screen of the head-mounted display device can be emitted from the set ray emission position in the virtual handle, and the function of selecting or manipulating the target object in the display screen through the ray can be achieved.

[0130] In an embodiment of the present disclosure, step S14 may include:

[0131] S14-A: Determine the rotation matrix of the handle coordinate system relative to the world coordinate system based on the virtual shoulder position, hand position of the user, and the coordinate system of the head-mounted display device.

[0132] The position from the right shoulder (or left shoulder) of the user to the palm center in the coordinate system of the head-mounted display device can be used as the forward vector, and the y-axis direction of the coordinate system of the head-mounted display device can be used as the secondary vector, so as to determine the rotation matrix of the handle coordinate system relative to the world coordinate system.

[0133] S14-B: Determine the rotation matrix of the handle coordinate system relative to the world coordinate system based on the virtual shoulder position, finger joint positions of the user, and the coordinate system of the head-mounted display device.

[0134] The position from the right shoulder (or left shoulder) of the user to the palm center in the coordinate system of the head-mounted display device can be used as the forward vector, and the position from the base of the little finger to the base of the index finger can be used as the secondary vector, so as to determine the rotation matrix of the handle coordinate system relative to the world coordinate system.

[0135] S14-C: Determine the rotation matrix of the handle coordinate system relative to the world coordinate system based on the hand position of the user.

[0136] The position from the center of the user's wrist to the base of the middle finger can be used as the forward vector, and the position from the base of the little finger to the base of the index finger can be used as the secondary vector, so as to determine the rotation matrix of the handle coordinate system relative to the world coordinate system.

[0137] S14-D: Determine the rotation matrix of the handle coordinate system relative to the world coordinate system based on the palm center and finger joint positions of the user.

[0138] The rotation of the user's palm relative to the base of the index finger (or the base of the middle finger) can be used to determine the rotation matrix of the handle coordinate system relative to the world coordinate system.

[0139] In this embodiment, the handle coordinate system can be selected personalized according to the user's needs, and then the degree of freedom of the handle coordinate system relative to the world coordinate system can be determined according to the handle coordinate system.

[0140] The method for interacting with a head-mounted display device according to the embodiments of the present disclosure may further include:

[0141] S16: Obtain the operation input value of the user on the virtual handle.

[0142] The gesture action of the user can be recognized by image recognition. When the gesture action of the user includes the gesture action for operating the virtual handle that has been pre-mapped, the operation input value for the virtual handle is obtained based on the recognized gesture action.

[0143] S18: If the operation input value is less than the minimum boundary threshold of the operation input value, update the minimum boundary threshold to the input value.

[0144] S20: If the operation input value is greater than the maximum boundary threshold of the operation input value, update the maximum boundary threshold to the input value.

[0145] In this embodiment, the operation input value of the virtual handle can be compared with the minimum boundary threshold and the maximum boundary threshold respectively, and whether to update the boundary threshold can be determined according to the comparison result, so that the setting of the minimum boundary threshold and the maximum boundary threshold can be more in line with the user's operation habits.

[0146] In an embodiment of the present disclosure, step S10 may include: in response to detecting that the user performs a trigger operation through the virtual handle, determining the input value of the trigger operation based on the distance between the fingertip positions of the first preset finger and the second preset finger when the user performs the trigger operation, such as a floating-point key value. The first preset finger may be the thumb of the hand performing the trigger operation, and the second preset finger may be the index finger or middle finger of the hand performing the trigger operation.

[0147] In this embodiment, when the user performs a trigger operation on the virtual handle, it is reasonable to determine the distance between the two fingers of the hand performing the trigger operation as the operation input value of the trigger operation.

[0148] In another embodiment of the present disclosure, step S14 may include: in response to detecting that the user performs a grasping operation on the virtual handle, determining the sum of the distances from the fingertips of multiple preset fingers of the hand performing the grasping operation to the palm as the input value of the grasping operation.

[0149] In this embodiment, when the user performs a grasping operation on the virtual handle, the sum of the distances from the fingertips of multiple preset fingers of the hand performing the grasping operation to the palm can be reasonably determined as the input value of the grasping operation.

[0150] The method for interacting with the head-mounted display device according to the embodiment of the present disclosure may further include: filtering the operation input value. For example, when determining the input distance value based on the finger joint position, the finger joint position can be used for filtering; when determining the input distance value based on the average position of multiple finger joints, multiple finger joints can be used for filtering.

[0151] In this embodiment, by filtering the operation input value, the accuracy of the operation input value can be improved, and further the accuracy of the input key value of the virtual handle or the virtual joystick can be improved, so that the accuracy between the user's gesture action and the response of the head-mounted display device can be improved.

[0152] Any method for interacting with a head-mounted display device provided in an embodiment of the present disclosure may be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices, servers, etc. Alternatively, any method for interacting with a head-mounted display device provided in an embodiment of the present disclosure may be executed by a processor. For example, the processor executes any method for interacting with a head-mounted display device mentioned in an embodiment of the present disclosure by calling corresponding instructions stored in a memory. This will not be elaborated further below.

[0153] Exemplary device

[0154] Figure 8 is a structural block diagram of a device for interacting with a head-mounted display device in an embodiment of the present disclosure. As Figure 8 shown, the device for interacting with a head-mounted display device includes:

[0155] A joystick mode activation module 100, configured to activate the joystick mode when detecting that the user has performed a preset gesture action;

[0156] A virtual joystick coordinate system determination module 200, configured to determine the origin position of the virtual joystick coordinate system based on at least one of the user's hand position and the display screen of the head-mounted display device in the state where the joystick mode is activated, and determine the virtual joystick coordinate system based on the origin position of the virtual joystick coordinate system;

[0157] An input key value determination module 300, configured to determine the input key value of the virtual joystick based on the projection of the user's target hand in the virtual joystick coordinate system.

[0158] In an embodiment of the present disclosure, the virtual joystick coordinate system determination module 200 is configured to determine the origin position of the virtual joystick coordinate system based on the finger joint position of the user when the joystick mode is activated; the virtual joystick coordinate system determination module 200 is further configured to determine the direction parallel to the first coordinate axis of the coordinate system of the head-mounted display device passing through the origin position as the first coordinate axis direction of the virtual joystick coordinate system, and determine the direction parallel to the second coordinate axis of the coordinate system of the head-mounted display device passing through the origin position as the second coordinate axis direction of the virtual joystick coordinate system.

[0159] In another embodiment of the present disclosure, the virtual joystick coordinate system determination module 200 is configured to determine, based on a preset position in a preset display plane of the display screen, the origin position of the virtual joystick coordinate system; the virtual joystick coordinate system determination module 200 is further configured to determine a first preset direction of the preset display plane as the first coordinate axis direction of the virtual joystick coordinate system, and determine a second preset direction of the preset display plane as the second coordinate axis direction of the virtual joystick coordinate system.

[0160] In an embodiment of the present disclosure, the input key value determination module 300 is configured to obtain the orthogonal projection coordinates of the target finger of the target hand in the virtual joystick coordinate system; the input key value determination module 300 is further configured to perform a normalization process on the orthogonal projection coordinates to determine the input key value of the virtual joystick.

[0161] In an embodiment of the present disclosure, the input key value determination module 300 is configured to perform a normalization process on the orthogonal projection coordinates based on a preset minimum absolute coordinate value, a preset numerical processing width, and a preset overshoot coefficient of the virtual joystick coordinate system to determine the input key value of the virtual joystick, where the preset minimum absolute coordinate value is the coordinate value of the orthogonal projection coordinates.

[0162] In one embodiment of the present disclosure, the input key value determination module 300 is configured to determine an input value based on the orthogonal projection coordinates; the input key value determination module 300 is further configured to determine a first numerical processing range based on the preset minimum absolute value of the coordinates, and determine a second numerical processing range based on the preset minimum absolute value of the coordinates and the preset numerical processing width, and determine a third numerical processing range based on the preset minimum absolute value of the coordinates, the preset numerical processing width, and the preset overcharge coefficient, wherein the maximum numerical boundary of the first numerical range is less than the maximum numerical boundary of the second numerical range, and the minimum numerical boundary of the first numerical range is greater than the minimum numerical boundary of the second numerical range, the maximum numerical boundary of the second numerical range is less than the maximum numerical boundary of the third numerical range, and the minimum numerical boundary of the second numerical range is greater than the minimum numerical boundary of the third numerical range; the input key value determination module 300 is further configured to determine the input key value to be 0 if the input value is within the first numerical processing range; the input key value determination module 300 is further configured to determine the input key value based on the input value, the preset minimum absolute value of the coordinates, and the preset numerical processing width if the input value is outside the first numerical processing range and within the second numerical processing range; the input key value determination module 300 is further configured to determine the input key value to be one of 1 and -1 based on the magnitude relationship between the input value and 0 if the input value is outside the second numerical processing range and within the third numerical processing range; the input key value determination module 300 is further configured to determine the input key value to be 0 if the input value is outside the third numerical processing range.

[0163] In one embodiment of the present disclosure, the device for interacting with the head-mounted display device may further include a virtual handle processing module, and the virtual handle processing module is configured to activate the handle mode in response to detecting that the user has performed a preset handle mode gesture action; the virtual handle processing module is further configured to determine the input key value of the virtual handle based on the handle mode gesture action in the state where the handle mode is activated, wherein the handle mode gesture action includes at least one of a key pressing action and a grasping action, and the input key value of the virtual handle includes the input key value of the key pressing action and the input key value of the grasping action.

[0164] In one embodiment of the present disclosure, the virtual handle processing module is configured to determine an input distance value for the virtual handle based on the handle mode gesture action; the virtual handle processing module is further configured to perform a normalization process on the input distance value based on the input distance value, a preset critical coefficient, a maximum processing distance threshold, and a minimum processing distance threshold to determine the input key value of the virtual handle.

[0165] In one embodiment of the present disclosure, the virtual handle processing module is configured to obtain the operation input value of the user on the virtual handle; the virtual handle processing module is further configured to update the minimum boundary threshold to the input value if the operation input value is less than the minimum boundary threshold of the operation input value; the virtual handle processing module is further configured to update the maximum boundary threshold to the input value if the operation input value is greater than the maximum boundary threshold of the operation input value.

[0166] In one embodiment of the present disclosure, when the virtual handle processing module detects that the user performs a trigger operation through the virtual handle, it determines the input value of the trigger operation based on the distance between the fingertip positions of a first preset finger and a second preset finger when the user performs the trigger operation.

[0167] In another embodiment of the present disclosure, when the virtual handle processing module detects that the user performs a grasping operation on the virtual handle, it determines the sum of the distances from the fingertips of multiple preset fingers of the hand performing the grasping operation to the palm as the input value of the grasping operation.

[0168] It should be noted that the specific implementation of the device for interacting with the head-mounted display device in the embodiments of the present disclosure is similar to the specific implementation of the method for interacting with the head-mounted display device in the embodiments of the present disclosure. For details, refer to the part of the method for interacting with the head-mounted display device. To avoid redundancy, it will not be elaborated here.

[0169] Exemplary electronic device

[0170] Next, refer to Figure 9 to describe the electronic device according to the embodiments of the present disclosure. As Figure 9 shown, the electronic device includes one or more processors 10 and a memory 20.

[0171] The processor 10 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0172] The memory 20 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage media, and the processor 10 may run the program instructions to implement the methods for interacting with the head-mounted display device in the various embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage media.

[0173] In one example, the electronic device may further include: an input device 30 and an output device 40, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown). The input device 30 may be, for example, a keyboard, a mouse, etc. The output device 40 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0174] Of course, for simplicity, Figure 9 only some of the components related to the present disclosure in the electronic device are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application scenarios, the electronic device may further include any other appropriate components.

[0175] Exemplary computer-readable storage medium

[0176] The computer-readable storage medium may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0177] In addition, the embodiments of the present disclosure further provide a head-mounted display device, including the device for interacting with the head-mounted display device in the above embodiments.

[0178] The basic principles of the present disclosure have been described in conjunction with specific embodiments. However, it should be noted that the advantages, benefits, effects, etc. mentioned in the present disclosure are only examples and not limitations. It cannot be considered that these advantages, benefits, effects, etc. are essential for each embodiment of the present disclosure. Additionally, the specific details disclosed above are only for illustrative and facilitating understanding purposes, rather than limitations. The above details do not limit the present disclosure to necessarily adopt the above specific details for implementation.

[0179] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple. For relevant parts, reference can be made to the partial description of the method embodiments.

[0180] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present disclosure are only illustrative examples and do not intend to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any way. Words such as "including", "comprising", "having", etc. are open-ended terms, meaning "including but not limited to", and can be used interchangeably with each other. The word "or" and "and" used herein refer to the word "and / or", and can be used interchangeably with each other, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to", and can be used interchangeably with each other.

[0181] The methods and apparatuses of the present disclosure can be implemented in many ways. For example, the methods and apparatuses of the present disclosure can be implemented through software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of the steps for the method is only for illustration purposes. The steps of the method of the present disclosure are not limited to the above specifically described order, unless otherwise specifically stated in other ways. Additionally, in some embodiments, the present disclosure can also be implemented as a program recorded in a recording medium, and these programs include machine-readable instructions for implementing the methods according to the present disclosure. Therefore, the present disclosure also covers the recording medium storing the programs for executing the methods according to the present disclosure.

[0182] It also needs to be pointed out that in the apparatuses, equipment, and methods of the present disclosure, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of the present disclosure.

[0183] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0184] The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present disclosure to the form disclosed herein. Although several example aspects and embodiments have been discussed above, those skilled in the art will recognize some of their variations, modifications, alterations, additions, and subcombinations.

Claims

1. A method for interacting with a head-mounted display device, comprising: activating a joystick mode in response to detecting that a user wearing the head-mounted display device has performed a preset joystick mode gesture action; in a state where the joystick mode is activated, determining an origin position of a virtual joystick coordinate system based on at least one of the user's hand position and the position of the display screen of the head-mounted display device, and determining the virtual joystick coordinate system based on the origin position of the virtual joystick coordinate system; determining an input key value of the virtual joystick based on the projection of the user's target hand in the virtual joystick coordinate system, including: determining a first numerical processing range, a second numerical processing range, and a third numerical processing range based on a preset absolute minimum coordinate value, a preset numerical processing width, and a preset overshoot coefficient of the virtual joystick coordinate system respectively, wherein the maximum numerical boundary of the first numerical processing range is less than the maximum numerical boundary of the second numerical processing range, and the minimum numerical boundary of the first numerical processing range is greater than the minimum numerical boundary of the second numerical processing range, the maximum numerical boundary of the second numerical processing range is less than the maximum numerical boundary of the third numerical processing range, and the minimum numerical boundary of the second numerical processing range is greater than the minimum numerical boundary of the third numerical processing range; determining the input key value based on the numerical processing range where the input value is located, wherein the input value is obtained based on the orthogonal projection coordinates of the target finger of the target hand in the virtual joystick coordinate system.

2. The method according to claim 1, wherein, the determining an origin position of a virtual joystick coordinate system based on at least one of the user's hand position and the position of the display screen, and determining the virtual joystick coordinate system based on the origin position of the virtual joystick coordinate system, includes: determining the origin position of the virtual joystick coordinate system based on the finger joint position of the user when the joystick mode is activated; determining the first axis direction of the virtual joystick coordinate system as the direction passing through the origin position and parallel to the first axis of the coordinate system of the head-mounted display device, and determining the second axis direction of the virtual joystick coordinate system as the direction passing through the origin position and parallel to the second axis of the coordinate system of the head-mounted display device.

3. The method according to claim 1, wherein, the determining an origin position of a virtual joystick coordinate system based on at least one of the user's hand position and the position of the display screen, and determining the virtual joystick coordinate system based on the origin position of the virtual joystick coordinate system, includes: determining a preset position in a preset display plane of the display screen as the origin position of the virtual joystick coordinate system; determining the first preset direction of the preset display plane as the first axis direction of the virtual joystick coordinate system, and determining the second preset direction of the preset display plane as the second axis direction of the virtual joystick coordinate system.

4. The method according to any one of claims 1-3, wherein, Determining the input key value of the virtual joystick based on the projection of the target hand of the user in the virtual joystick coordinate system includes: Obtaining the orthogonal projection coordinates of the target finger of the target hand in the virtual joystick coordinate system; Performing normalization processing on the orthogonal projection coordinates to determine the input key value of the virtual joystick.

5. The method according to claim 4, wherein, The performing normalization processing on the orthogonal projection coordinates to determine the input key value of the virtual joystick includes: Based on the preset absolute minimum coordinate value, preset numerical processing width, and preset overshoot coefficient of the virtual joystick coordinate system, performing normalization processing on the orthogonal projection coordinates to determine the input key value of the virtual joystick, and the preset absolute minimum coordinate value is the coordinate value of the orthogonal projection coordinates.

6. The method according to claim 5, wherein, The performing normalization processing on the orthogonal projection coordinates based on the preset absolute minimum coordinate value, preset numerical processing width, and preset overshoot coefficient of the virtual joystick coordinate system to determine the input key value of the virtual joystick includes: determining an input value based on the orthogonal projection coordinates; Determining a first numerical processing range based on the preset absolute minimum coordinate value, determining a second numerical processing range based on the preset absolute minimum coordinate value and the preset numerical processing width, and determining a third numerical processing range based on the preset absolute minimum coordinate value, the preset numerical processing width, and the preset overshoot coefficient; If the input value is within the first numerical processing range, determining the input key value to be 0; If the input value is outside the first numerical processing range and within the second numerical processing range, then determining the input key value based on the input value, the preset absolute minimum coordinate value, and the preset numerical processing width; If the input value is outside the second numerical processing range and within the third numerical processing range, determining the input key value to be one of 1 and -1 based on the magnitude relationship between the input value and 0; If the input value is outside the third numerical processing range, determining the input key value to be 0.

7. The method according to claim 1, wherein, further includes: In response to detecting that the user has performed a preset handle mode gesture action, activating the handle mode; In the state of activating the handle mode, determining the input key value of the virtual handle based on the handle mode gesture action, wherein the handle mode gesture action includes at least one of a key pressing action and a grasping action, and the input key value of the virtual handle includes the input key value of the key pressing action and the input key value of the grasping action.

8. The method according to claim 7, wherein, The determining the input key value of the virtual handle based on the handle mode gesture action includes: Based on the handle mode gesture action, determining the input distance value for the virtual handle; Based on the input distance value, preset critical coefficient, maximum processing distance threshold, and minimum processing distance threshold, performing normalization processing on the input distance value to determine the input key value of the virtual handle.

9. The method according to claim 8, wherein, further includes: Obtain the operation input value of the user for the virtual handle; If the operation input value is less than the minimum boundary threshold of the operation input value, update the minimum boundary threshold to the input value; If the operation input value is greater than the maximum boundary threshold of the operation input value, update the maximum boundary threshold to the input value.

10. The method according to claim 8 or 9, wherein, the obtaining the operation input value of the user for the virtual handle includes: When detecting that the user performs a trigger operation through the virtual handle, based on the distance between the fingertip positions of the first preset finger and the second preset finger when the user performs the trigger operation, determine the input value of the trigger operation.

11. The method according to claim 8 or 9, wherein, the obtaining the operation input value of the user for the virtual handle includes: When detecting that the user performs a grasping operation on the virtual handle, determine the sum of the distances from the fingertips of multiple preset fingers of the hand performing the grasping operation to the palm as the input value of the grasping operation.

12. A device for interacting with a head-mounted display device, comprising: A joystick mode activation module, configured to activate the joystick mode when detecting that the user performs a preset joystick mode gesture action; A virtual joystick coordinate system determination module, configured to, in a state where the joystick mode is activated, determine the origin position of the virtual joystick coordinate system based on at least one of the user's hand position and the position of the display screen of the head-mounted display device, and determine the virtual joystick coordinate system based on the origin position of the virtual joystick coordinate system; An input key value determination module, configured to determine the input key value of the virtual joystick based on the projection of the user's target hand in the virtual joystick coordinate system, including: determining a first numerical processing range, a second numerical processing range, and a third numerical processing range respectively based on the preset absolute minimum coordinate value, the preset numerical processing width, and the preset overshoot coefficient of the virtual joystick coordinate system, wherein the maximum numerical boundary of the first numerical processing range is less than the maximum numerical boundary of the second numerical processing range, and the minimum numerical boundary of the first numerical processing range is greater than the minimum numerical boundary of the second numerical processing range, the maximum numerical boundary of the second numerical processing range is less than the maximum numerical boundary of the third numerical processing range, and the minimum numerical boundary of the second numerical processing range is greater than the minimum numerical boundary of the third numerical processing range; determining the input key value based on the numerical processing range where the input value is located, wherein the input value is obtained based on the orthogonal projection coordinates of the target finger of the target hand in the virtual joystick coordinate system.

13. An electronic device, the electronic device comprising: A processor; A memory for storing executable instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for interacting with a head-mounted display device according to any one of claims 1-11 above.

14. A computer-readable storage medium storing a computer program for performing the method for interacting with a head-mounted display device according to any one of claims 1-11 above.

15. An interaction system comprising: a head-mounted display device, and the device for interacting with a head-mounted display device according to claim 12.

Images