Interaction method and apparatus, device and medium

Abstract

The present disclosure provides an interaction method and apparatus, a device and a medium. The method comprises: displaying at least one interaction object in a virtual space, wherein each interaction object comprises at least one virtual control, and each virtual control comprises at least two response layers; on the basis of a touch operation for any virtual control, determining a real-time operation position of the touch operation; and when it is determined that the real-time operation position of the touch operation is located in any response layer of the virtual control, controlling the virtual control to execute an interaction event associated with the response layer. The present disclosure can improve the interaction accuracy between users and virtual controls, so that the interaction between the users and the virtual controls is more natural and vivid, thereby improving the interaction effect between the users and virtual scenes.

Description

Interaction method, device, equipment and medium

[0001] This disclosure claims priority to Chinese invention patent application No. 202410177972.1, filed on February 8, 2024, entitled “Human-computer interaction method, apparatus, device, and medium.” This disclosure also claims priority to Chinese invention patent application No. 202410166722.8, filed on February 5, 2024, entitled “Interaction method, apparatus, storage medium, and device.” The entire contents of the aforementioned applications are incorporated herein by reference. Technical Field

[0002] The embodiments of the present disclosure relate to the field of human-computer interaction technology, and in particular to interaction methods, devices, equipment, and media. Background Art

[0003] Currently, electronic devices can provide users with a variety of virtual scenes, allowing users to immerse themselves in the virtual scenes. Among them, the virtual scene is a three-dimensional virtual world simulated by computer technology and digital simulation technology.

[0004] In order to facilitate user interaction with the virtual scene, virtual controls are usually set up in the virtual scene, such as setting virtual controls on virtual objects in the virtual scene, so as to meet the various interaction needs of users with the virtual scene. When traditionally using virtual controls to interact with the virtual scene, the user usually presses the virtual control and then releases it to trigger the execution of the interaction event associated with the virtual control. However, this interaction method relies on user gesture data or interactor data such as handles. If there are errors in the acquired user gesture data or interactor data, it will lead to misjudgment of the interaction with the virtual control, resulting in poor interaction with the virtual scene.

[0005] With the rise of the metaverse and the widespread adoption of XR devices, users are placing higher demands on the interactive experience and input efficiency of XR devices. However, traditional XR device input methods have many shortcomings, such as their limited understanding of XR device input interactions and the lack of physical feedback from virtual keyboards. These issues result in a poor user experience and low input efficiency when using XR devices. Summary of the Invention

[0006] The embodiments of the present disclosure provide a human-computer interaction method, apparatus, device, and medium, which can improve the accuracy of interaction between a user and a virtual control, making the interaction between the user and the virtual control more natural and realistic, and improving the interactive effect between the user and the virtual scene.

[0007] In a first aspect, an embodiment of the present disclosure provides a human-computer interaction method, comprising: displaying at least one interactive object in a virtual space, each of the interactive objects comprising at least one virtual control, and each of the virtual controls comprising at least two response layers; determining a real-time operation position of the touch operation based on a touch operation performed on any of the virtual controls; and when determining that the real-time operation position of the touch operation is located in any response layer of the virtual control, controlling the virtual control to execute an interactive event associated with the response layer.

[0008] In a second aspect, an embodiment of the present disclosure provides a human-computer interaction device, comprising: a display module for displaying at least one interactive object in a virtual space, each of the interactive objects comprising at least one virtual control, and each of the virtual controls comprising at least two response layers; a determination module for determining a real-time operation position of a touch operation based on a touch operation performed on any of the virtual controls; and a control module for controlling the virtual control to execute an interactive event associated with the response layer when determining that the real-time operation position of the touch operation is located in any response layer of the virtual control.

[0009] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising: a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the human-computer interaction method as described in the embodiment of the first aspect or its various implementation methods.

[0010] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the human-computer interaction method as described in the embodiment of the first aspect or its various implementations.

[0011] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising program instructions, which, when executed on an electronic device, enables the electronic device to execute the human-computer interaction method as described in the embodiment of the first aspect or its various implementations.

[0012] The embodiments of the present disclosure also provide an interaction method, apparatus, storage medium, and device, which can output interaction feedback information, improve the interaction convenience and input efficiency of the virtual keyboard, and improve the intuitiveness and naturalness of the interaction.

[0013] On the one hand, an embodiment of the present disclosure provides an interaction method, which includes: displaying a three-dimensional environment generated by an extended reality device; identifying a poke gesture of a current object; determining an interaction event between the current object and the virtual keyboard based on the poke gesture, the poke gesture collision body corresponding to the poke gesture, and the hover collision body corresponding to the virtual key in the virtual keyboard; displaying a three-dimensional virtual keyboard in the three-dimensional environment, wherein a plurality of virtual keys are displayed on the virtual keyboard; and outputting interaction feedback information in response to the interaction event, wherein the interaction feedback information includes at least a visual indication.

[0014] On the other hand, an embodiment of the present disclosure provides an interaction device, which includes: a first display unit for displaying a three-dimensional environment generated by an extended reality device; a second display unit for displaying a virtual keyboard in the three-dimensional environment, wherein a plurality of virtual keys are displayed on the virtual keyboard; an identification unit for identifying a poke gesture of a current object; a determination unit for determining an interaction event between the current object and the virtual keyboard based on the poke gesture, a poke gesture collision body corresponding to the poke gesture, and a hover collision body corresponding to the virtual key in the virtual keyboard; and an interaction unit for outputting interaction feedback information for the interaction event, wherein the interaction feedback information includes at least a visual indication.

[0015] On the other hand, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the interaction method described in any of the above embodiments.

[0016] On the other hand, an embodiment of the present disclosure provides a terminal device, which includes a processor and a memory, wherein a computer program is stored in the memory, and the processor is used to execute the interaction method described in any of the above embodiments by calling the computer program stored in the memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] FIG1A is a flow chart of a human-computer interaction method provided by an embodiment of the present disclosure;

[0019] FIG1B is a schematic diagram of a preset interaction gesture provided by an embodiment of the present disclosure;

[0020] FIG1C is a schematic diagram of a forward input touch operation provided by an embodiment of the present disclosure;

[0021] FIG1D is a schematic diagram of a side-input touch operation provided by an embodiment of the present disclosure;

[0022] FIG1E is a schematic diagram of deploying a response layer within a three-dimensional shell of a virtual control according to an embodiment of the present disclosure;

[0023] FIG1F is a schematic diagram of deploying a response layer on a virtual control of a two-dimensional planar structure according to an embodiment of the present disclosure;

[0024] FIG1G is a flowchart of another human-computer interaction method provided by an embodiment of the present disclosure;

[0025] FIG1H is a schematic diagram of another embodiment of the present disclosure for deploying a response layer within a three-dimensional shell of a virtual control;

[0026] FIG1I is a schematic diagram of another embodiment of the present disclosure for deploying a response layer on a virtual control with a two-dimensional planar structure;

[0027] FIG1J is a flowchart of another human-computer interaction method provided by an embodiment of the present disclosure;

[0028] FIG1K is a schematic diagram of displaying a touch operation layer on a touch start layer according to an embodiment of the present disclosure;

[0029] FIG1L is a schematic diagram of another embodiment of the present disclosure for deploying a response layer within a three-dimensional shell of a virtual control;

[0030] FIG1M is a schematic diagram of another embodiment of the present disclosure for deploying a response layer on a virtual control with a two-dimensional planar structure;

[0031] FIG1N is a flowchart of another human-computer interaction method provided by an embodiment of the present disclosure;

[0032] FIG10 is a front view of a method for setting a touch cancellation area according to an embodiment of the present disclosure;

[0033] FIG1P is a schematic diagram of another embodiment of the present disclosure for deploying a response layer within a three-dimensional shell of a virtual control;

[0034] FIG1Q is a schematic diagram of another embodiment of the present disclosure for deploying a response layer on a virtual control with a two-dimensional planar structure;

[0035] FIG1R is a flowchart of another human-computer interaction method provided by an embodiment of the present disclosure;

[0036] FIG1S is a schematic diagram of a structure in which multiple response layers are provided in a virtual control according to an embodiment of the present disclosure;

[0037] FIG1T is a schematic block diagram of a human-computer interaction device provided by an embodiment of the present disclosure;

[0038] FIG1U is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure;

[0039] FIG2A is a flow chart of an interactive method according to an embodiment of the present disclosure;

[0040] FIG2B is a schematic diagram of a first application scenario of the interaction method provided by an embodiment of the present disclosure;

[0041] FIG2C is a schematic diagram of a second application scenario of the interaction method provided by an embodiment of the present disclosure;

[0042] FIG2D is a schematic diagram of a third application scenario of the interaction method provided in an embodiment of the present disclosure;

[0043] FIG2E is a schematic diagram of a fourth application scenario of the interaction method provided in an embodiment of the present disclosure;

[0044] FIG2F is a schematic diagram of a fifth application scenario of the interaction method provided in an embodiment of the present disclosure;

[0045] FIG2G is a schematic diagram of a sixth application scenario of the interaction method provided in an embodiment of the present disclosure;

[0046] FIG2H is a schematic diagram of a seventh application scenario of the interaction method provided in an embodiment of the present disclosure;

[0047] FIG2I is a schematic diagram of an eighth application scenario of the interaction method provided in an embodiment of the present disclosure;

[0048] FIG2J is a schematic diagram of a ninth application scenario of the interaction method provided in an embodiment of the present disclosure;

[0049] FIG2K is a schematic diagram of a tenth application scenario of the interaction method provided in an embodiment of the present disclosure;

[0050] FIG2L is a schematic diagram of an eleventh application scenario of the interaction method provided by an embodiment of the present disclosure;

[0051] FIG2M is a schematic diagram of a twelfth application scenario of the interaction method provided in an embodiment of the present disclosure;

[0052] FIG2N is a schematic diagram of a thirteenth application scenario of the interaction method provided in an embodiment of the present disclosure;

[0053] FIG2O is a schematic diagram of a fourteenth application scenario of the interaction method provided in an embodiment of the present disclosure;

[0054] FIG2P is a schematic diagram of a fifteenth application scenario of the interaction method provided in an embodiment of the present disclosure;

[0055] FIG2Q is a schematic diagram of the structure of an interactive device provided in an embodiment of the present disclosure;

[0056] FIG2R is a first structural diagram of a terminal device provided in an embodiment of the present disclosure; and

[0057] FIG2S is a second structural diagram of the terminal device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0059] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0060] In the embodiments of the present disclosure, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or solution described as "exemplary" or "for example" in the embodiments of the present disclosure should not be interpreted as being more preferred or more advantageous than other embodiments or solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0061] In the description of the embodiments of the present disclosure, unless otherwise specified, "a plurality of" refers to two or more than two, that is, at least two. "At least one" refers to one or more than one.

[0062] To facilitate understanding of the embodiments of the present disclosure, before describing each embodiment of the present disclosure, some concepts involved in all embodiments of the present disclosure are first appropriately explained, as follows.

[0063] 1) Virtual Reality (VR) is a technology for creating and experiencing virtual worlds. It generates a virtual environment using multi-source information (VR, as mentioned in this article, includes at least visual perception, but can also include auditory perception, tactile perception, motion perception, and even taste and olfactory perception). It achieves the fusion of interactive three-dimensional dynamic scenes and the simulation of physical behavior in the virtual environment, immersing users in a simulated VR environment. This technology is used in a variety of virtual environments, including mapping, gaming, video, education, healthcare, simulation, collaborative training, sales, assisted manufacturing, maintenance, and repair.

[0064] 2) VR devices, terminals that achieve virtual reality effects, can usually be provided in the form of glasses, head-mounted displays (HMDs), or contact lenses to achieve visual perception and other forms of perception. Of course, the form of VR devices is not limited to this and can be further miniaturized or enlarged according to actual needs.

[0065] Optionally, the VR devices described in the embodiments of the present disclosure may include but are not limited to the following types.

[0066] 2.1) PC-based virtual reality (PCVR) devices use the PC to perform calculations and output data related to virtual reality functions. External PC-based virtual reality devices use the data output by the PC to achieve virtual reality effects.

[0067] 2.2) Mobile VR devices support the configuration of mobile terminals (such as smartphones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations for VR functions and outputs data to the mobile VR device, such as viewing VR videos through the mobile terminal's app.

[0068] 2.3) All-in-one virtual reality devices have a processor for performing relevant calculations for virtual functions, and thus have independent virtual reality input and output functions. They do not need to be connected to a PC or mobile terminal and have a high degree of freedom of use.

[0069] 3) Augmented Reality (AR): A technology that uses real-time calculations of the camera's pose parameters in the real world (also known as the 3D world or the real world) as the camera captures images. Based on these pose parameters, virtual elements are added to the captured images. Virtual elements include, but are not limited to, images, videos, and 3D models. The goal of AR technology is to overlay the virtual world on the real world for interactive viewing.

[0070] 4) Mixed Reality (MR): By presenting virtual scene information in a real scene, an interactive feedback information loop is established between the real world, the virtual world, and the user to enhance the realism of the user experience. For example, a simulated scene that integrates computer-generated sensory input (e.g., virtual objects) with sensory input from a physical scene or its representation. In some MR scenes, the computer-generated sensory input can adapt to changes in sensory input from the physical scene. In addition, some electronic systems used to present MR scenes can monitor the orientation and / or position relative to the physical scene so that virtual objects can interact with real objects (i.e., physical elements from the physical scene or their representations). For example, the system can monitor movement so that virtual plants appear stationary relative to physical buildings.

[0071] 5) Extended Reality (XR): This technology combines the real and virtual worlds through computers to create a virtual environment where humans and machines can interact. XR is a general term for multiple technologies, including VR, AR, and MR. By integrating the visual interaction technologies of these three, users can experience an immersive experience, seamlessly transitioning between the virtual and real worlds.

[0072] 6) A virtual scene is a virtual scene displayed (or provided) when an application is running on an electronic device. The virtual scene can be a simulation of the real world, a semi-simulation and semi-fictitious virtual scene, or a purely fictitious virtual scene. The virtual scene can be any of a two-dimensional virtual scene, a 2.5-dimensional virtual scene, or a three-dimensional virtual scene. The embodiments of the present disclosure do not limit the dimensions of the virtual scene. For example, a virtual scene can include the sky, land, ocean, etc., and the land can include environmental elements such as deserts and cities. Users can control virtual objects to move in the virtual scene. It should be understood that the above-mentioned virtual scene can also be referred to as a virtual space.

[0073] 7) Virtual objects are objects that interact in virtual scenes and are controlled by users or robot programs (for example, robot programs based on artificial intelligence). They can be still, move, and perform various behaviors in the virtual scene, such as various characters in games.

[0074] Normally, users can interact with virtual scenes by utilizing virtual controls set within the virtual scene. For example, a user presses a virtual control and then releases it to trigger an interaction event associated with the virtual control. However, this interaction method relies on user gesture data or interactor data. When errors exist in the acquired user gesture data or interactor data, misjudgment of the interaction with the virtual control will occur, resulting in poor interaction with the virtual scene. For example, the user's finger has actually pressed the virtual control, but due to factors such as recognition algorithms or device hardware defects, it is recognized that the user's finger has not pressed the virtual control.

[0075] In order to solve the above technical problems, the inventive concept of the present disclosure is: by setting multiple response layers for each virtual control in the virtual space, wherein the virtual control is located on any interactive object. Thus, when the user's touch operation on any virtual control is obtained, the real-time operation position of the touch operation is determined. When it is determined that the real-time operation position of the touch operation is located in a certain response layer on the virtual control, the virtual control is controlled to execute the interactive event associated with the response layer. In this way, by setting the interaction recognition strategy on the virtual control side, it is achieved that when the touch operation is located in different response layers, the virtual control can be controlled to execute different response events, rather than relying solely on gesture data or interactor data to determine whether the user interacts with the virtual control. Thereby, the accuracy of the interaction between the user and the virtual control can be improved, the interaction between the user and the virtual control can be made more natural and realistic, and the interactive effect between the user and the virtual scene can be improved.

[0076] The technical solutions of the present disclosure are described in detail below through some embodiments. The embodiments described below can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0077] Figure 1A is a flow chart of a human-computer interaction method provided in an embodiment of the present disclosure. The human-computer interaction method provided in an embodiment of the present disclosure can be performed by a human-computer interaction device. The human-computer interaction device can be composed of hardware and / or software and can be integrated into an electronic device. In the present disclosure, the electronic device can be any terminal device that can provide a virtual scene function to the user, such as a tablet computer, a personal desktop computer, a laptop computer, an XR device, and other wearable devices. The present disclosure does not impose any restrictions on the type of electronic device.

[0078] As shown in FIG1A , the method includes the following steps.

[0079] S101, displaying at least one interactive object in a virtual space, each interactive object including at least one virtual control, and each virtual control including at least two response layers.

[0080] The above-mentioned virtual space can be understood as any virtual scene provided by an electronic device to a user. The virtual scene is a three-dimensional virtual world simulated by computer technology and digital simulation technology.

[0081] For example, if a user wants to create a virtual space that suits their personal preferences, they can use the virtual scene creation function provided by the electronic device to build a personalized virtual space. Alternatively, the user can select any virtual space from multiple virtual spaces already created by the electronic device as their favorite virtual space, and this disclosure does not impose any restrictions on this.

[0082] To facilitate user interaction with the virtual control during the immersive experience, the present disclosure may set at least one interactive object within the virtual space, allowing the user to interact with the virtual space by interacting with the interactive object. The interactive object can be understood as a virtual object displayed within the virtual space.

[0083] In the present disclosure, the above-mentioned virtual objects may be selected as objects including virtual controls, such as virtual interfaces, virtual panels, virtual windows, etc.

[0084] The virtual control can be understood as a virtual key or a virtual button. Furthermore, the virtual control can be a three-dimensional control or a two-dimensional control, and the present disclosure does not impose any restrictions on this.

[0085] It should be understood that each virtual control on each interactive object in this disclosure is provided with multiple response layers, and each response layer is associated with a corresponding response event. A response event can be understood as an interaction event. This allows users to interact with the virtual space through a virtual control by determining which response layer of the virtual control the user-triggered interaction is acting on. The user-triggered interaction can then be responded to based on the response event associated with that response layer, thereby enhancing the user's interaction with the virtual scene.

[0086] The setting positions of the multiple response layers included in each of the above-mentioned virtual controls can be flexibly set according to the structure of the virtual control, and the present disclosure does not impose any restrictions on this.

[0087] S102: Determine a real-time operation position of the touch operation according to the touch operation on any virtual control.

[0088] It should be understood that any of the above-mentioned virtual controls may be a virtual control on any interactive object in the virtual space.

[0089] In the present disclosure, the touch operation may include at least one of the following: operation through an input device and gesture operation.

[0090] The input device may be, but is not limited to, a mouse, keyboard, handle, touchpad, or other interactive devices. For example, the user may use the mouse to click, double-click, or long-press any virtual control.

[0091] Optionally, when it is necessary to interact with the virtual space through virtual controls, the user can achieve this by triggering a touch operation on a virtual control on any interactive object.

[0092] Considering that when a user triggers a touch operation on a virtual control, the operation position corresponding to the touch operation will change from one position to another. For example, when a user presses a virtual control through a gesture, the operation position of the gesture pressing operation triggered by the user will change from the pressing start position to the pressing end position. Therefore, when the present disclosure detects a user's touch operation on any virtual control, it can detect the real-time operation position of the touch operation, so as to subsequently respond to the user's touch operation based on the real-time operation position of the touch operation.

[0093] In some optional embodiments, detecting the real-time operation position of the touch operation may include the following situations.

[0094] In the first scenario, when a user interacts with a virtual control through gestures, a gesture tracking camera on the electronic device can capture real-time images of the user's hand. Gesture recognition technology can then be used to identify the hand image and determine whether it corresponds to a preset interaction gesture. If the gesture is determined to be a preset interaction gesture, the gesture tracking camera can be used to capture real-time images of the user's hand performing the preset interaction gesture. Based on the real-time captured hand image, the position of the user's hand performing the preset interaction gesture can be determined, and this position can be used as the real-time operation position for the touch operation.

[0095] The preset interaction gestures can be set based on the compatibility between the user's gesture operation habits in the real environment and the interaction with the virtual control. For example, when pressing a virtual control, the preset interaction gesture can be selected as a single-handed pressing gesture, etc. This disclosure does not impose any restrictions on the preset interaction gestures.

[0096] In some optional implementations, the preset interaction gesture may be as shown in FIG1B , wherein one finger of the user's real hand is raised, and the remaining four fingers are bent toward the palm to form a single-hand pressing gesture.

[0097] That is, when identifying the collected user hand image, it is only necessary to identify whether there is a finger raised in the user hand image. When it is identified that there is a finger raised in the user hand image, the gesture recognition result can be determined to be a single-hand pressing gesture.

[0098] Considering that the tip of the raised finger in a single-handed press gesture typically contacts and presses a virtual control, the above method determines the position of the user's hand performing the preset interactive gesture based on the user's hand image captured in real time by the gesture tracking camera, and determines this user's hand position as the real-time operation position of the touch operation. Specifically, the position of the fingertip of the raised finger in the preset interactive gesture is obtained, and this fingertip position is determined as the real-time operation position of the touch operation.

[0099] In the second scenario, when a user interacts with a virtual control through an input device, the real-time position information of the input device or the touch movement on the input device can be obtained by acquiring the position information sent by the position acquisition device on the input device. The acquired real-time position information is then determined as the real-time operation position of the touch operation.

[0100] Considering that the input device in the present disclosure is a mouse, a handle, a touchpad or other controller, the present disclosure can adopt different methods according to the type of input device to obtain the real-time operation position of the touch operation.

[0101] As an optional embodiment, when the input device is a device such as a mouse or joystick that can move the operating position of a touch operation by moving its own position, the real-time position data of this device can be obtained by the position acquisition device in this type of device. Next, the real-time position data is obtained from the real-time position data. The obtained real-time position data is then determined as the real-time operating position of the touch operation.

[0102] In this disclosure, the aforementioned posture acquisition device may be any device or apparatus capable of acquiring posture information of devices such as a mouse and a controller, such as a posture sensor, an inertial measurement unit (IMU), and other devices including gyroscopes and accelerometers. This disclosure does not impose any restrictions on the posture acquisition device, as long as it can acquire real-time posture information of the aforementioned devices.

[0103] Furthermore, when the input device is a touch-operated device such as a touchpad, the present disclosure can obtain the real-time position corresponding to the touch operation on the touchpad and determine the real-time position corresponding to the touch operation as the real-time operation position of the touch operation. The method of obtaining the real-time position corresponding to the touch operation on the touchpad can be referred to in the prior art and will not be elaborated on here.

[0104] S103: When it is determined that the real-time operation position of the touch operation is located at any response layer of the virtual control, the virtual control is controlled to execute an interactive event associated with the response layer.

[0105] In this disclosure, each virtual control includes multiple response layers, and each response layer is associated with a different interaction event. For example, if a virtual control includes two response layers, a first response layer and a second response layer, the interaction event associated with the first response layer may be interaction event 1, and the interaction event associated with the second response layer may be interaction event 2. Interaction event 1 and interaction event 2 are different.

[0106] Optionally, after obtaining the real-time operation position of the touch operation, it can be determined whether the real-time operation position of the touch operation is located on a response layer among the multiple response layers of the virtual control. If it is determined that the real-time operation position of the touch operation is located on a response layer of the virtual control, the interactive event associated with the response layer is obtained, and the interactive event is executed on the virtual control to achieve a high-precision response to the user based on the operation position of the touch operation. The interaction between the user and the virtual control is made more natural and realistic, thereby improving the user's interactive experience of human-computer interaction. If it is determined that the real-time operation position of the touch operation is not located on a response layer of the virtual control, the real-time operation position of the touch operation is continued to be determined until the operation position of the touch operation is located on a response layer of the virtual control at a certain time, and the interactive event associated with the response layer is executed on the virtual control.

[0107] In some optional embodiments, when the present disclosure determines that the real-time operation position of a touch operation is located at any response layer of the virtual control, in addition to executing the interaction event associated with the response layer on the virtual control, vibration feedback and / or sound effect feedback corresponding to the response layer can also be output. This allows the user to understand the interaction status between the user and the virtual control in real time through the perception system such as touch and / or hearing, thereby reducing the interaction difficulty and improving the human-computer interaction experience.

[0108] In the present disclosure, each response layer of each virtual control corresponds to different vibration feedback and / or sound feedback, which makes it easier for users to distinguish which response layer of the virtual control the current touch operation acts on, thereby achieving better interaction effects.

[0109] Considering that when a user interacts with a virtual control, the input direction of the touch operation for the virtual control can be to input the touch operation directly to the virtual control, or to input the touch operation from the side of the virtual control. For example, Figure 1C shows that the touch operation input by the user for a certain virtual control is a forward input, that is, the touch operation is input directly to the virtual control. Figure 1D shows that the touch operation input by the user for a certain virtual control is a lateral input, that is, the touch operation is input from the side of the virtual control. Therefore, the present disclosure can also obtain the position information of the touch operation in the lateral direction, that is, the entry position, when it is detected that the input direction of the user's touch operation for any virtual control is input from the side (lateral direction) of the virtual control. And when the entry position is located in a certain response layer of the virtual control, the interactive event associated with the response layer is executed for the virtual control. In this way, it can avoid redeploying multiple response layers according to the operation position of the touch operation when the touch operation input by the user is input from the side, thereby improving the human-computer interaction response speed.

[0110] The technical solution disclosed in the embodiments of this disclosure sets multiple response layers for each virtual control on an interactive object displayed in a virtual space. This allows a user to interact with any virtual control through touch control, and the virtual control is controlled to execute the interaction event associated with the response layer based on the touch operation. This improves the accuracy of interaction between the user and the virtual control, making the interaction between the user and the virtual control more natural and realistic, and enhancing the user's interaction with the virtual scene.

[0111] In some optional embodiments, the virtual control in the present disclosure includes at least two response layers, which may include a hover collision layer and a touch end layer. Among them, the hover collision layer is used to indicate the effective position of interaction with the virtual control, that is, when the operation position of the touch operation is located in the hover collision layer, it can trigger the display adjustment of the virtual control, so that the user can interact with the virtual control more conveniently and quickly. The touch end layer is used to indicate the touch end position when the user touches the virtual control. For example, when the user presses the virtual control, the touch end layer is the position where the virtual control can no longer be pressed down, and the inability to be pressed down can be understood as the virtual control has been pressed to the bottom.

[0112] In the present disclosure, when a virtual control is a three-dimensional structure, the hover collision layer and touch end layer can be deployed within the three-dimensional shell of the virtual control, as shown in Figure 1E. Alternatively, when the virtual space is a two-dimensional planar structure, the hover collision layer and touch end layer can be deployed at corresponding positions with reference to the virtual control body, as shown in Figure 1F. This disclosure does not impose any restrictions on the deployment location of the response layer.

[0113] It should be understood that the deployment positions of the at least two response layers included in the virtual control in the present disclosure can be pre-set according to the virtual control structure, and each response layer is invisible to the user. This can meet the universal requirements of multiple response layers on virtual controls in different scenarios, and setting the response layer to be invisible can avoid touch interference for users.

[0114] On the basis that the virtual control shown in FIG. 1E and FIG. 1F includes at least two response layer structures, as shown in FIG. 1G , the above S103 may include the following steps: S103 - 1 and S103 - 2 .

[0115] At S103 - 1 , when the real-time operation position of the touch operation is located in the hovering collision layer, the virtual control is visually highlighted.

[0116] In some optional embodiments, when the real-time operation position of the acquired touch operation is determined to be located on the hover collision layer, it indicates that the user may need to select the virtual control corresponding to the hover collision layer. In this case, the present disclosure can adjust the visual display of the virtual control corresponding to the hover collision layer to remind the user that they can interact with the virtual control.

[0117] In the present disclosure, visually highlighting a virtual control may include at least one of the following: size enlargement and color adjustment.

[0118] Among them, the size enlargement process can enlarge the current size of the virtual control according to a preset enlargement ratio. The above preset enlargement ratio can be flexibly set according to actual needs, such as enlarging according to a 1:0.5 ratio, etc., and this disclosure does not impose any restrictions on this.

[0119] Color adjustment can be understood as adjusting the first display color of the virtual control to the second display color, such as adjusting white to gray.

[0120] At S103 - 2 , when the real-time operation position of the touch operation is located at the touch end layer, an interaction event associated with the virtual control is executed on the interaction object.

[0121] In some optional embodiments, when the real-time operation position of the acquired touch operation is determined to be on the touch end layer, it indicates that the user needs to select the virtual control corresponding to the touch end layer to execute the interaction event associated with the virtual control on the interactive object. In this case, the present disclosure can execute the interaction event associated with the virtual control on the interactive object to which the virtual control belongs.

[0122] For example, assuming that the interactive object where the virtual control is located is a document window, when the virtual control is a next page control, a page switching operation is performed on the document to switch the current page of the document to the next page.

[0123] For another example, assuming that the interactive object where the virtual control is located is a video playback interface, then when the virtual control is a control for closing the bullet screen, the bullet screen content in the video playback interface is closed.

[0124] It should be understood that the input direction of the user input touch operation for any virtual control includes: forward and lateral. Then, when the input direction of the touch operation is forward, the execution order of the above S103-1 and S103-2 is to execute S103-1 first and then execute S103-2. When the input direction of the touch operation is lateral, the execution order of the above S103-1 and S103-2 may be to execute S103-1 or S103-2 separately. In some embodiments, when a touch operation is input laterally, the real-time operation position of the touch operation is located in the hover collision layer, that is, S103-2 can also be executed after executing S103-1.

[0125] In some optional embodiments, the virtual control of the present disclosure includes at least two response layers and may also include a touch start layer. The touch start layer is used to indicate the position where the virtual control's shape changes when a user touches the virtual control. For example, when a user needs to press a virtual control, the touch start layer indicates the initial position of the virtual control before it is pressed.

[0126] In the present disclosure, when the virtual control is a three-dimensional structure, the touch-start layer can be deployed within the three-dimensional shell of the virtual control, as shown in Figure 1H . Alternatively, when the virtual space is a two-dimensional planar structure, the touch-start layer can be deployed at a corresponding position with reference to the virtual control body, as shown in Figure 1I . This disclosure does not impose any restrictions on the deployment location of the response layer.

[0127] On the basis that the virtual control shown in FIG1H and FIG1I includes at least two response layer structures, as shown in FIG1J , the above S103 may further include the following step: S103 - 3 .

[0128] At S103 - 3 , when the real-time operation position of the touch operation is located in the touch start layer, a touch operation layer supporting user touch is displayed.

[0129] In some optional embodiments, when it is determined that the real-time operation position of the acquired touch operation is located on the touch start layer, it indicates that the user wants to interact with the virtual control corresponding to the touch start layer. At this time, the present disclosure can display a touch operation layer that supports user touch at the touch start layer position, so that the user can act on the touch operation layer through touch operation. And when the operation position of the touch operation changes, the touch operation layer changes accordingly with the real-time operation position of the touch operation, so as to realize the visualization of the interaction process between the user and the virtual control, so that the user can understand the interaction status between himself and the virtual control through vision.

[0130] That is, the display position of the touch operation layer displayed at the touch start layer in the present disclosure will change accordingly with the real-time operation position of the touch operation. For example, in Figure 1K, when a user presses a control, if the operation position of the touch operation is on the touch start layer, the touch operation layer will be displayed at the touch start layer position. The user can then press the touch operation layer through the touch operation to achieve the effect of pressing the virtual control.

[0131] In the present disclosure, the shape and size of the touch operation layer are the same as those of the virtual controls, but the color of the touch operation layer can be different from the color of the virtual controls. This helps users clearly distinguish the location of the touch operation layer and interact with the virtual controls by manipulating the touch operation layer.

[0132] It should be understood that the input direction of the user input for the touch operation of any virtual control includes: forward and lateral. Then when the input direction of the touch operation is forward, the execution order of the above S103-1 to S103-3 is to execute S103-1 first, then execute S103-3, and finally execute S103-2. When the input direction of the touch operation is lateral, the execution order of the above S103-1, S103-2 and S103-3 can be to execute S103-1, S103-2 or S103-3 separately. Among them, when the touch operation is input laterally, the real-time operation position of the touch operation is in the hover collision layer, that is, S103-2 and S103-3 can also be executed after executing S103-1.

[0133] In some optional embodiments, a user may need to cancel their interaction with a virtual control due to other reasons while touching it. To this end, the virtual control disclosed herein includes at least two response layers and may also include a touch cancellation layer. The touch cancellation layer is used to control the touch operation to act on the touch cancellation layer when the user needs to cancel the current interaction operation.

[0134] In the present disclosure, when the virtual control is a three-dimensional structure, the touch cancellation layer can be deployed within the three-dimensional shell of the virtual control, as shown in Figure 1L. Alternatively, when the virtual space is a two-dimensional planar structure, the touch cancellation layer can be deployed at a corresponding position with reference to the virtual control body, as shown in Figure 1M. This disclosure does not impose any restrictions on the deployment location of the response layer.

[0135] It should be understood that the distance between the above-mentioned touch cancellation layer and the virtual control body should be greater than the distance between the aforementioned response layers, so as to avoid the touch operation penetrating the virtual control to reach the touch cancellation layer due to user hand shaking or other reasons, thereby causing the touch operation of the virtual control to be accidentally canceled.

[0136] On the basis that the virtual control shown in FIG1L and FIG1M includes at least two response layer structures, as shown in FIG1N , the above S103 may further include the following step: S103 - 4 .

[0137] At S103 - 4 , when the real-time operation position of the touch operation is located in the touch cancellation layer, the selection operation on the virtual control is canceled.

[0138] Optionally, when a user interacts with any virtual control by inputting a touch operation and needs to cancel the interaction, the user can control the touch operation to move toward the touch end layer and pass through the touch end layer until the operation position of the touch operation is located in the touch cancel layer. At this time, the electronic device determines that the user needs to cancel the interaction based on the operation position of the touch operation and ends the interaction process with the virtual control.

[0139] As an optional implementation, to avoid the user having to move the touch control position to the touch cancel layer each time they need to cancel an interactive operation, the present disclosure can determine the boundary area of ​​the virtual control and set any area larger than the boundary area as the touch cancel area. In this way, when the user needs to cancel an interactive operation, they can directly move the touch control position from the current position to the touch cancel area to cancel the interactive operation.

[0140] For example, as shown in Figure 1O, assuming that the front-view boundary area of ​​a virtual control is area 1, the area outside area 1 can be determined as the touch cancellation area. It should be understood that the shapes and sizes of the multiple response layers on the virtual control in this disclosure are the same as the shapes and sizes of the virtual control.

[0141] It should be understood that the input direction of the user input for the touch operation of any virtual control includes: forward and lateral. Then, when the input direction of the touch operation is forward, the execution order of the above S103-1 to S103-4 is to execute S103-1 first, then execute S103-3, then execute S103-2, and finally execute S103-4. When the input direction of the touch operation is lateral, the execution order of the above S103-1, S103-2, S103-3 and S103-4 can be to execute S103-1, S103-2, S103-3 or S103-4 separately. Among them, when the touch operation is input laterally, the real-time operation position of the touch operation is in the hover collision layer, that is, S103-2, S103-3 and S103-4 can also be executed after executing S103-1. Alternatively, when a touch operation is input from the side and the real-time operation position of the touch operation is located in the touch start layer, that is, after executing S103-3, S103-2 and S103-4 may be executed. Alternatively, when a touch operation is input from the side and the real-time operation position of the touch operation is located in the touch end layer, that is, after executing S103-2, S103-3 may be executed.

[0142] In some optional embodiments, given that virtual controls on interactive objects are not physical controls, it's difficult for a user to maintain the corresponding input device or real hand in a single, absolutely stationary position during touch interaction. Instead, the user's hand will experience slight jitter. Furthermore, this jitter can cause the touch operation's position to repeatedly affect the touch end layer, automatically triggering the execution of an interactive event associated with the virtual control on the interactive object each time the touch operation's position lands on the touch end layer. However, in reality, a user may only need to execute an interactive event associated with the virtual control on the interactive object once.

[0143] To this end, the virtual control in the present disclosure includes at least two response layers, and may also include a selection input layer and a selection exit layer. The selection input layer is used to determine the user's interaction intent with the virtual control. Specifically, when the operation position of a touch operation input by the user falls within the selection input layer, the present disclosure can determine that the user intends to interact with the virtual control. The selection exit layer is used to indicate the triggering position layer for executing the interaction event associated with the virtual control on the interactive object when the user interacts with the virtual control. Specifically, when the operation position of the touch operation moves from the touch end layer back to the selection exit layer, it indicates that the user intends to execute the interaction event associated with the virtual control on the interactive object, and the interaction event associated with the virtual control is triggered on the interactive object. Accordingly, if the operation position of the touch operation moves from the touch end layer back to the selection exit layer but does not move to the selection exit layer, and this repeats multiple times, the present disclosure determines that the above operations are virtual control selection operations, and does not execute the interaction event associated with the virtual control on the interactive object to which the virtual control belongs. Only when the operation position of the touch operation moves to the selection exit layer is it considered that the user intends to execute the interaction event associated with the virtual control on the interactive object. At this time, the interactive event associated with the virtual control is executed on the interactive object, which can achieve a de-jittering effect and avoid the problem of repeatedly triggering the interactive event associated with the virtual control on the interactive object due to hand shaking.

[0144] In the present disclosure, when the virtual control is a three-dimensional structure, the selection input layer and the selection exit layer can be deployed within the three-dimensional shell of the virtual control, as shown in Figure 1P. Alternatively, when the virtual space is a two-dimensional planar structure, the selection input layer and the selection output layer can be deployed at corresponding positions with reference to the virtual control body, as shown in Figure 1Q. This disclosure does not impose any restrictions on the deployment location of the response layer.

[0145] On the basis that the virtual control shown in FIG1P and FIG1Q includes at least two response layer structures, as shown in FIG1R , the above S103 may further include the following steps: S103 - 5 and S103 - 6 .

[0146] At S103 - 5 , when the real-time operation position of the touch operation is located in the selected input layer, the display mode of the virtual control is updated.

[0147] Optionally, when it is determined that the real-time operation position of the touch operation is located in the selected input layer, it indicates that the user intends to interact with the virtual control. In this case, the display state of the virtual control is adjusted to highlight that the user is about to interact with the virtual control, that is, the user is about to trigger an interaction event associated with the virtual control.

[0148] In the present disclosure, updating the display mode of the virtual control may be, but is not limited to, adjusting the display color of the virtual control from the second display color to a third display color. The third display color may be the same as or different from the first display color. Preferably, the present disclosure sets the third display color to a color different from the first display color, so that different interaction events associated with each response layer in the virtual control can be distinguished.

[0149] At S103 - 6 , when the real-time operation position of the touch operation is located at the selection and exit layer, an interaction event associated with the virtual control is executed on the interaction object.

[0150] In some optional embodiments, when it is determined that the real-time operation position of the acquired touch operation is located in the selection and exit layer, it indicates that the user needs to trigger the execution of an interaction event associated with the virtual control on the interactive object to which the virtual control belongs. In this case, the present disclosure obtains the interaction event associated with the virtual control and executes the acquired interaction event on the interactive object.

[0151] The technical solution disclosed in the embodiments of this disclosure sets multiple response layers for each virtual control on an interactive object displayed in a virtual space. When a user interacts with a virtual control through touch, the virtual control is controlled to execute the interaction event associated with the response layer based on the touch operation. This improves the accuracy of interaction between the user and the virtual control, making the interaction between the user and the virtual control more natural and realistic, and enhancing the user's interactive experience with the virtual scene.

[0152] To facilitate understanding of the human-computer interaction solution provided by the embodiment of the present disclosure, the human-computer interaction solution provided by the embodiment of the present disclosure is specifically described below with reference to a specific example. As shown in Figure 1S, it is assumed that a virtual control on an interactive object within a virtual control includes six response layers, namely, a hover collision layer, a touch start layer, a touch end layer, a selection input layer, a selection exit layer, and a touch cancel layer. Then, when the user uses a one-handed press gesture and interacts with the virtual control through a positive input method, the entire interaction process may include the following steps.

[0153] In step one, the user can move toward the direction of the virtual control through a one-hand pressing gesture. When the operation position of the one-hand pressing gesture moves to the hover collision layer of the virtual control, the virtual control is visually highlighted to prompt the user that the virtual control is currently being aimed at.

[0154] In step 2, the user can continue to use a single-handed press gesture to move in the direction of the virtual control. When the operation position of the single-handed press gesture moves from the hover collision layer to the touch start layer, the touch operation layer is displayed at the touch start layer position. The user can then press the touch operation layer, causing the touch operation layer to change accordingly based on the position of the user's press operation, thereby visually showing the user the effect of the virtual control being pressed.

[0155] In step three, when the user continues to move toward the direction of the virtual control through the one-hand press gesture, when the operation position of the one-hand press gesture moves from the touch start layer to the selection input layer, it is determined that the user has the intention to interact with the virtual control, that is, it is determined that the user wants to interact with the virtual control, then the display status of the virtual control can be updated to show the user that he can interact with the virtual control.

[0156] In step 4, the user continues to use a single-handed press gesture to move in the direction of the virtual control. When the operation position of the single-handed press gesture moves from the selection input layer to the touch end layer, it indicates that the user has pressed the virtual control to the lowest position. At this time, it is determined that it is possible to trigger an interaction event associated with the virtual control for the interactive object to which the virtual control belongs.

[0157] Among them, when the user determines that it is necessary to trigger the interaction event associated with the virtual control on the interactive object, the user can retract the one-hand press gesture to move the operation position of the one-hand press gesture from the touch end layer to the input exit layer, and then execute the interaction event associated with the virtual control on the interactive object to which the virtual control belongs.

[0158] In some optional embodiments, when the user needs to cancel the current interactive operation, the user can continue to move along the direction of the virtual control through a one-hand press gesture until the operation position of the one-hand press gesture moves to the touch cancellation layer, triggering the interaction cancellation event associated with the touch cancellation layer to cancel the current interactive operation.

[0159] In other words, the present disclosure sets multiple response layers for each virtual control on an interactive object displayed in a virtual space. This allows a user to interact with any virtual control through touch control, and the virtual control is controlled to execute the interaction event associated with that response layer based on the touch control's location. This improves the accuracy of interaction between the user and the virtual control, making the interaction more natural and realistic, and enhancing the user's interaction with the virtual scene.

[0160] A human-computer interaction device according to an embodiment of the present disclosure will be described below with reference to FIG1T FIG1T is a schematic block diagram of a human-computer interaction device according to an embodiment of the present disclosure.

[0161] As shown in FIG1T , the human-computer interaction device 400 includes a display module 410, a determination module 420, and a control module 430. Display module 410 is configured to display at least one interactive object in a virtual space, each of which includes at least one virtual control, and each of which includes at least two response layers. Determination module 420 is configured to determine the real-time operation location of a touch operation performed on any of the virtual controls. Control module 430 is configured to, upon determining that the real-time operation location of the touch operation is within any response layer of the virtual control, control the virtual control to execute an interactive event associated with the response layer.

[0162] In an optional implementation of the embodiment of the present disclosure, the at least two response layers include a hover-collision layer and a touch-end layer, and the control module 430 is specifically configured to: when the real-time operation position of the touch operation is located in the hover-collision layer, visually highlight the virtual control; and when the real-time operation position of the touch operation is located in the touch-end layer, execute an interaction event associated with the virtual control on the interactive object.

[0163] In an optional implementation of the embodiment of the present disclosure, the visually highlighting the virtual control includes at least one of the following: size enlargement and color adjustment.

[0164] In an optional implementation of the disclosed embodiment, the at least two response layers further include a touch-start layer, the touch-start layer being located between the hover-collision layer and the touch-end layer. Accordingly, the control module 430 is specifically configured to display a touch-operation layer that supports user touch when the real-time operation position of the touch operation is located within the touch-start layer.

[0165] In an optional implementation of the embodiment of the present disclosure, the display position of the touch operation layer changes with the real-time operation position of the touch operation.

[0166] In an optional implementation of the disclosed embodiment, the at least two response layers further include a touch cancel layer, located after the touch end layer. Accordingly, the control module is specifically configured to cancel the selection of the virtual control when the real-time operation position of the touch operation is within the touch cancel layer.

[0167] In an optional implementation of the embodiment of the present disclosure, the at least two response layers further include a selection input layer and a selection exit layer, wherein the selection input layer and the selection exit layer are located between the touch start layer and the touch end layer, and the selection input layer is located after the selection exit layer. Accordingly, the control module is specifically configured to: update the display mode of the virtual control when the real-time operation position of the touch operation is located in the selection input layer; and execute the interactive event associated with the virtual control on the interactive object when the real-time operation position of the touch operation is located in the selection exit layer.

[0168] In an optional implementation of the disclosed embodiment, the apparatus 400 further includes an output module configured to output vibration feedback and / or sound effect feedback corresponding to any response layer of the virtual control when the real-time operation position of the touch operation is determined to be within the response layer of the virtual control. The vibration feedback and / or sound effect feedback corresponding to each response layer of the virtual control is different.

[0169] In an optional implementation of the embodiment of the present disclosure, the input direction of the touch operation on any of the virtual controls includes a forward direction and a lateral direction. The control module is further configured to: when the input direction of the touch operation on any of the virtual controls is a lateral direction, control the virtual control to execute an interaction event corresponding to the position information of the touch operation in the lateral direction based on the position information of the touch operation in the lateral direction.

[0170] It should be understood that the device embodiment and the aforementioned method embodiment may correspond to each other, and similar descriptions can refer to the method embodiment. To avoid repetition, they are not described here in detail. Specifically, the device 400 shown in FIG1T can execute the method embodiment corresponding to FIG1, and the aforementioned and other operations and / or functions of each module in the device 400 are respectively for implementing the corresponding processes in each method in FIG1. ​​For the sake of brevity, they are not described here in detail.

[0171] The above text describes the device 400 of the embodiment of the present disclosure from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in the form of hardware, can be implemented by instructions in the form of software, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the first aspect of the method embodiment of the embodiment of the present disclosure can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the first aspect of the method disclosed in conjunction with the embodiment of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above-mentioned first aspect of the method embodiment in conjunction with its hardware.

[0172] The technical solution disclosed in the embodiments of this disclosure sets multiple response layers for each virtual control on an interactive object displayed in a virtual space. When a user interacts with a virtual control through touch, the virtual control is controlled to execute the interaction event associated with the response layer based on the touch operation. This improves the accuracy of interaction between the user and the virtual control, making the interaction between the user and the virtual control more natural and realistic, and enhancing the user's interactive experience with the virtual scene.

[0173] Figure 1U is a schematic block diagram of an electronic device provided in an embodiment of the present disclosure. As shown in Figure 1U, the electronic device 500 may include a memory 510 and a processor 520. The memory 510 is used to store computer programs and transmit the program code to the processor 520. In other words, the processor 520 can call and execute the computer program from the memory 510 to implement the human-computer interaction method in the embodiment of the present disclosure.

[0174] For example, the processor 520 may be configured to execute the aforementioned human-computer interaction method embodiment according to instructions in the computer program.

[0175] In some embodiments of the present disclosure, the processor 520 may include but is not limited to: a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0176] In some embodiments of the present disclosure, the memory 510 includes, but is not limited to, volatile memory and / or non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0177] In some embodiments of the present disclosure, the computer program may be divided into one or more modules, which are stored in the memory 510 and executed by the processor 520 to implement the human-computer interaction method provided by the present disclosure. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0178] As shown in Figure 1U, the electronic device 500 may further include a transceiver 530, which may be connected to the processor 520 or the memory 510. The processor 520 may control the transceiver 530 to communicate with other devices. Specifically, the transceiver 530 may send information or data to other devices or receive information or data sent by other devices. The transceiver 530 may include a transmitter and a receiver. The transceiver 530 may further include one or more antennas.

[0179] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0180] The present disclosure further provides a computer storage medium having a computer program stored thereon. When the computer program is executed by a computer, the computer is enabled to execute the human-computer interaction method of the above method embodiment.

[0181] An embodiment of the present disclosure further provides a computer program product including program instructions, which, when executed on an electronic device, enables the electronic device to execute the human-computer interaction method of the above method embodiment.

[0182] When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0183] The embodiments of the present disclosure can be applied to various application scenarios such as extended reality (XR), virtual reality (VR), augmented reality (AR), and mixed reality (MR).

[0184] First, some nouns or terms that appear in the description of the embodiments are explained as follows:

[0185] A virtual scene is a virtual scene displayed (or provided) when an application is running on a terminal or server. Optionally, the virtual scene is a simulation of the real world, a semi-simulation and semi-fictitious virtual environment, or a purely fictitious virtual environment. A virtual scene can be either a two-dimensional virtual scene or a three-dimensional virtual scene. The virtual environment can be the sky, land, ocean, etc. The land includes environmental elements such as deserts and cities. The virtual scene is a scene with complete game logic, including virtual objects such as user controls.

[0186] A virtual object refers to a dynamic object that can be controlled in a virtual scene. Optionally, the dynamic object can be a virtual person, virtual animal, anime character, etc. The virtual object is a character controlled by the player through an input device, or an artificial intelligence (AI) set up through training to compete in a virtual environment, or a non-player character (NPC) set up in a virtual scene battle. Optionally, the virtual object is a virtual person competing in the virtual scene. Optionally, the number of virtual objects in the virtual scene battle is preset or dynamically determined based on the number of clients joining the battle, which is not limited in the embodiments of the present disclosure. In one possible implementation, the user can control the virtual object to move in the virtual scene, for example, to control the virtual object to run, jump, crawl, etc., and can also control the virtual object to fight other virtual objects using skills, virtual props, etc. provided by the application. Optionally, a virtual object can also refer to a static object that can be interacted with in the virtual scene, such as a virtual object, virtual control, interface element, virtual props, etc.

[0187] Extended Reality (XR) is a concept that includes Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR). It refers to the technology that creates an environment where the virtual world is connected to the real world, and users can interact with the environment in real time.

[0188] Virtual Reality (VR) is a technology for creating and experiencing virtual worlds. It computationally generates a virtual environment, a multi-source information (VR mentioned in this article includes at least visual perception, but can also include auditory perception, tactile perception, motion perception, and even taste and olfactory perception). It achieves the fusion of interactive three-dimensional dynamic scenes and the simulation of physical behavior in the virtual environment, immersing users in the simulated three-dimensional environment. It has applications in various virtual environments such as mapping, gaming, video, education, healthcare, simulation, collaborative training, sales, assisted manufacturing, maintenance, and repair.

[0189] Augmented Reality (AR) is a technology that uses real-world (or 3D) camera pose parameters to calculate the camera's position in the real world (also known as the real world) in real time while the camera is capturing images. Based on these pose parameters, virtual elements are added to the captured images. These virtual elements include, but are not limited to, images, videos, and 3D models. The goal of AR technology is to overlay the virtual world on the real world for interactive viewing on a screen.

[0190] Mixed Reality (MR) is a simulated setting that integrates computer-generated sensory input (e.g., virtual objects) with sensory input from a physical setting, or its representation. In some MR settings, the computer-generated sensory input can adapt to changes in sensory input from the physical setting. Additionally, some electronic systems used to render MR settings can monitor the orientation and / or position relative to the physical setting to enable virtual objects to interact with real objects (i.e., physical elements from the physical setting, or their representations). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.

[0191] Augmented Virtuality (AV): An AV set is a computer-created or virtual set that incorporates at least one sensory input from a physical set. The one or more sensory inputs from the physical set may be a representation of at least one feature of the physical set. For example, a virtual object may exhibit the colors of a physical element captured by one or more imaging sensors. In another example, a virtual object may exhibit features consistent with actual weather conditions in the physical set, as identified via weather-related imaging sensors and / or online weather data. In another example, an augmented reality forest may have virtual trees and structures, but animals may have features accurately reproduced from images of the physical animals.

[0192] Virtual field of view refers to the area in the virtual environment that the user can perceive through the lens in the extended reality device. The field of view (FOV) of the virtual field of view is used to represent the perceived area.

[0193] Extended reality devices, terminals that achieve extended reality effects, can usually be provided in the form of glasses, helmet-mounted displays (Head Mount Display, HMD), and contact lenses to achieve visual perception and other forms of perception. Of course, the form of extended reality devices is not limited to this and can be further miniaturized or enlarged as needed.

[0194] The extended reality devices described in the embodiments of the present disclosure may include, but are not limited to, the following types:

[0195] Computer-based virtual reality (PCVR) devices use the PC to perform calculations and output data related to virtual reality functions. External computer-based extended reality devices use the data output by the PC to achieve virtual reality effects.

[0196] Mobile extended reality devices support the installation of a mobile terminal (such as a smartphone) in various ways (such as a head-mounted display with a dedicated card slot). Through a wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations for virtual reality functions and outputs data to the mobile extended reality device. For example, viewing virtual reality videos through an app on the mobile terminal.

[0197] The all-in-one extended reality device has a processor for performing related calculations for virtual functions, and thus has independent virtual reality input and output functions. It does not need to be connected to a PC or mobile terminal and has a high degree of freedom in use.

[0198] With the rise of the metaverse and the rapid development of XR (Extended Reality) technology, XR devices, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), have garnered widespread market attention and user interest. These devices leverage advanced hardware technologies to provide users with an immersive interactive experience. However, despite significant hardware advancements, XR devices still have numerous shortcomings in their built-in software systems, particularly operating systems and input methods.

[0199] Currently, most XR devices are based on the Android operating system. While Android is quite mature in the mobile device space, it struggles when directly applied to XR devices. This is primarily because the interaction methods of XR devices differ significantly from those of traditional mobile devices, and the Android operating system is not optimized to account for these differences. As a result, users often encounter issues such as inconvenient operation and unfriendly interfaces when using XR devices.

[0200] Input methods are particularly problematic. Traditional Android input method vendors have accumulated extensive experience in mobile electronic devices, but this experience isn't fully applicable to XR devices. XR device input primarily relies on gesture recognition and a virtual keyboard, which differs significantly from the physical keyboards and touch input of traditional mobile devices. Due to a lack of in-depth understanding of XR device input interactions, traditional input methods perform poorly on XR devices.

[0201] Furthermore, virtual keyboards on XR devices face another challenge: a lack of physical feedback. On traditional mobile devices, users experience physical feedback from the device when typing, which helps improve input accuracy and efficiency. However, virtual keyboards on XR devices lack this physical feedback, leading to frequent issues like accidental and missed touches.

[0202] Regarding the input methods on XR devices, current traditional input method manufacturers have a very limited understanding of the input interactions and have not fully explored new user input interactions in this scenario. In order to improve the input methods on XR devices, the embodiments of the present disclosure propose an interaction method. By designing a virtual keyboard with three-dimensional visual effects and a virtual physical feedback mechanism, pre-selected highlighting and press feedback for the target keys are implemented on the virtual keyboard, which can improve the interaction convenience and input efficiency of the virtual keyboard, as well as improve the intuitiveness and naturalness of the interaction.

[0203] It should be noted that the order of description of the following embodiments does not limit the priority order of the embodiments.

[0204] Each embodiment of the present disclosure provides an interaction method, which can be executed by a terminal or a server, or by both the terminal and the server. The embodiments of the present disclosure are described by taking the interaction method executed by a terminal device as an example.

[0205] Please refer to Figures 2A to 2P. Figure 2A is a schematic flow chart of the interaction method provided in an embodiment of the present disclosure. Figures 2B to 2P are schematic diagrams of application scenarios of the interaction method provided in an embodiment of the present disclosure. This method can be applied to a terminal device, which can include any of an extended reality device, a virtual reality device, an augmented reality device, and a mixed reality device. The method includes the following steps 110 to 130.

[0206] Step 110: Display the three-dimensional environment generated by the extended reality device.

[0207] First, using the technology of extended reality devices, a realistic three-dimensional environment is constructed. This environment provides users with an immersive experience, allowing them to feel as if they are in a real yet virtual world. The three-dimensional environment generated by the extended reality device includes not only real space environments but also virtual space environments. The three-dimensional environment can be a virtual reality environment or an extended reality environment.

[0208] First, extended reality devices can have powerful graphics processing capabilities to generate high-quality 3D environments in real time. This usually involves using advanced graphics rendering techniques such as ray tracing, shadow mapping, texture mapping, etc. to create realistic 3D scenes.

[0209] To provide an immersive experience, extended reality devices can also have depth perception and stereoscopic imaging capabilities. This can be achieved by using technologies such as infrared sensors, depth cameras or lidar, which can capture detailed information about the surrounding environment and convert it into a three-dimensional model.

[0210] When generating a 3D environment, the user's head and eye movements can also be taken into account. XR devices can have high-precision motion tracking capabilities, capturing inputs such as the user's head posture, eye movements, and gestures in real time. Based on this input, they can adjust the perspective and details of the 3D environment to provide a natural, immersive experience.

[0211] To enhance the user experience, the 3D environment can also be integrated with a user interface. This interface can be customized to the application's needs, providing various functions and controls such as menus, buttons, and input boxes. These interface elements can be realistically rendered in the 3D environment, allowing users to interact intuitively.

[0212] Step 120: Display a three-dimensional virtual keyboard in the three-dimensional environment, where a plurality of virtual keys are displayed on the virtual keyboard.

[0213] For example, a three-dimensional environment must first be modeled and rendered to match the real world. Then, a three-dimensional (3D) virtual keyboard model is created within this 3D environment. This model consists of multiple 3D virtual keys. Each virtual key can be customized based on actual needs, including its shape, size, position, and color. The key layout on the virtual keyboard can be customized to meet the needs of different application scenarios. Furthermore, the virtual keyboard can support multiple input methods, such as touch, gestures, and voice, allowing users to choose the interaction method that best suits their habits and preferences.

[0214] For example, the display effect of the virtual keyboard can be integrated with the three-dimensional environment to provide a more realistic feeling. For example, a semi-transparent design can be used to make the virtual keyboard and the background environment complement each other, and effects such as shadows and lighting can be used to enhance the visual effect.

[0215] In an embodiment of the present disclosure, a three-dimensional (3D) virtual keyboard model can be created using OpenGL for Embedded Systems (OpenGL ES) in an Android integrated development tool (such as Android Studio). OpenGL ES supports efficient 3D graphics rendering, and based on its capabilities, a realistic 3D virtual keyboard can be created. Each virtual key in the 3D virtual keyboard is an independent 3D model with its own texture, material, and mesh.

[0216] Among them, each virtual key in the 3D virtual keyboard is an independent 3D model, which means that each key has its own shape, size and appearance, and can be set with animation, lighting and other visual effects separately.

[0217] In 3D graphics, texture refers to an image or map applied to the surface of a model. It can be used to add texture, pattern, color or details to make the model look more realistic and rich.

[0218] Among them, the material defines the visual properties of the modeled object's surface, such as color, glossiness, reflection, etc. In 3D modeling, each modeled object or model can have one or more materials, which determine the appearance and texture of the modeled object when rendered.

[0219] A mesh is the basic structure of a 3D model, consisting of a series of vertices, edges, and faces. It defines the model's shape and appearance and determines how the model is rendered and displayed. Each independent 3D model has its own mesh structure.

[0220] For example, in the three-dimensional virtual keyboard 10 shown in FIG. 2B , the key layer where the virtual keys 11 are located may be slightly higher than the horizontal plane where the virtual keyboard 10 is located.

[0221] Step 130: Recognize the poking gesture of the current object.

[0222] For example, the current object is the user currently interacting with the virtual keyboard.

[0223] First, the system monitors the operation gestures of the current object in real time. In this case, a specific gesture is recognized as a poke gesture. A poke gesture is a form of gesture interaction. For example, the poke gesture can be a simple finger touch action, or a more complex combination of touch and press. For example, a poke gesture can refer to a touch or click operation performed by a user on a virtual keyboard using a finger or other object (such as a handle). By accurately recognizing this poke gesture, the system can understand the user's intention and operation.

[0224] The XR device's gesture recognition library can be used to identify the current object's manipulation gestures. This library provides an application programming interface (API) that can obtain information about the user's finger position and movement. For example, the Hand Tracking API can accurately track the user's finger movements, including speed, direction, and position.

[0225] Step 140 : determining an interaction event between the current object and the virtual keyboard according to the poke gesture, the poke gesture collision body corresponding to the poke gesture, and the hover collision body corresponding to the virtual key in the virtual keyboard.

[0226] After recognizing a poke gesture, the system further analyzes the relationship between the poke gesture and the virtual keyboard's position. For example, if the user makes a poke gesture above the virtual keyboard, the system will determine that this is an interaction event related to the virtual keyboard. This determination is made based on the location of the poke gesture and the display area of ​​the virtual keyboard.

[0227] The system needs to determine the poke gesture's corresponding collision volume. A collision volume is a geometric shape used to detect collisions and interactions within the virtual environment. Based on the location and direction of the poke gesture, the system generates a corresponding poke gesture collision volume within the virtual environment.

[0228] The system needs to determine the hover collider corresponding to the virtual key in the virtual keyboard. The hover collider can be a geometric shape used to detect the proximity of the user's fingertip to the virtual key when the fingertip used to perform the poke gesture is not pressed. The system can generate the corresponding hover collider in the virtual environment based on the position and size of the virtual key.

[0229] By comparing the position and state of the poke collider and the hover collider, you can determine the interaction event between the current object and the virtual keyboard. For example, if the poke collider overlaps or is close to the hover collider of a virtual key, the system can determine that the user intends to press the virtual key and trigger the corresponding interaction event.

[0230] By recognizing the poke gesture, determining the poke gesture collider and hover collider, and comparing their states, the interaction event between the current object and the virtual keyboard can be accurately determined. This interaction method can improve the convenience and input efficiency of the virtual keyboard, as well as make the interaction more intuitive and natural.

[0231] In some embodiments, determining the interaction event between the current object and the virtual keyboard based on the poke gesture, the poke gesture collision body corresponding to the poke gesture, and the hover collision body corresponding to the virtual key in the virtual keyboard includes: constructing a poke gesture collision body corresponding to a gesture interactor based on the poke gesture, the position of the poke gesture collision body corresponding to the fingertip position of the finger that generates the poke gesture; for any virtual key on the virtual keyboard, constructing a hover collision body corresponding to the virtual key, the position of the hover collision body corresponds to the position of the virtual key; obtaining the starting point and end point of a poke gesture trajectory based on the poke gesture; and determining the interaction event between the current object and the virtual keyboard based on the starting point and end point of the poke gesture trajectory, the poke gesture collision body and the hover collision body.

[0232] When a poke gesture is recognized, a Poke Interactor can be constructed based on the characteristics of the poke gesture. This Poke Interactor includes a Poke Collider. The Poke Collider represents the shape and size of the finger that generated the poke gesture, and its position corresponds to the position of the fingertip. This allows the system to more accurately simulate and track the actual movement trajectory of the finger.

[0233] For example, you can interact with the system through the gesture interactor (Poke Interactor) corresponding to the poke gesture to control the system to perform input and output.

[0234] For example, as shown in the interactive scene diagram of FIG2C , the gesture interactor 20 (Poke Interactor) needs to provide information of a poke gesture collision body 21 (Poke Collider) and a touch point 22 (Touch Point).

[0235] Among them, the gesture interactor 20 (Poke Interactor) acts as a direct interactor, allowing the user to complete the selection by directly touching the interactive object 1. For example, the interactive object 1 can be a user interface, a virtual keyboard, or a virtual button. The gesture interactor 20 (Poke Interactor) places a spherical poke gesture collider 21 (Poke Collider) on the fingertip and determines the interaction state based on the spatial relationship between the poke gesture collider 21 (Poke Collider) and the interactive object 1. Therefore, the interaction state cannot be determined by the input of the gesture interactor 20 itself. Some calculations need to be combined with the interaction operation with the interactive object 1 to perform gesture recognition.

[0236] The location where the poke gesture takes effect is the location of the poke gesture collider 21, which is determined by the aiming state (Aim Pose) output by the controller (Controller). The aiming state (Aim Pose) describes the user's aiming position and direction, which is determined based on the user's input (such as the position of the finger) and the device's posture information.

[0237] If the poke collider 21 is constructed based on a bare hand, the location where the poke gesture takes effect is the tip of the index finger.

[0238] If the poke collider 21 is constructed based on a handle, the location where the poke gesture takes effect is a point at the front end of the handle.

[0239] The effective radius of the poke gesture (Poke) is the radius of the poke gesture collision body 21 (Poke Collider).

[0240] The speed of the poke gesture (Poke) is determined by the speed of the poke gesture collider 21 (Poke Collider).

[0241] Scope of the Poke gesture: Applies only to general User Experience (UX) components.

[0242] To better detect and identify the user's interaction intent, a hover collider can be constructed for each virtual key on the virtual keyboard. The shape and size of these hover colliders can match the virtual key, and the position can correspond to the center or a specific position of each virtual key.

[0243] The motion trajectory of the poking gesture can be tracked, and the starting point and end point of the poking gesture trajectory can be determined. The tracking can be performed based on the motion of the poking gesture collision body, and can accurately capture the movement path of the finger in space.

[0244] The interaction event between the current object and the virtual keyboard can be determined based on the starting and ending points of the poke gesture trajectory, as well as the relationship between the poke gesture collider and the hover collider. For example, if the poke gesture trajectory intersects or approaches the hover collider of a virtual key, it is considered a valid interaction event and the corresponding action is performed.

[0245] In some embodiments, constructing a hover collision body corresponding to the virtual key includes: constructing a hover collision body corresponding to the virtual key in front of the virtual keyboard, the hover collision body including a start push layer, a selection exit layer, a selection entry layer and an end push layer constructed in sequence along the vertical direction of the virtual key perpendicular to the virtual keyboard; the method also includes: constructing a selection cancel layer behind the virtual keyboard.

[0246] For example, referring to Figures 2D to 2G , it is necessary to define the effective area for the poke gesture on the interactive object. The effective area can be constructed by referring to the structure of a three-dimensional virtual key. The effective area of ​​the poke gesture corresponding to each virtual key can include a hover collider 30, two presentation layers, and three state layers. The presentation layer is a visually related layer. The two presentation layers can include a start push layer A (Start Push Layer) and an end push layer D (End Push Layer). The state layer is used to determine the transition of the selection. The three state layers can include a select exit layer B (Select Exit Layer), a select enter layer C (Select Enter Layer), and a select cancel layer E (Select Cancel Layer). The start push layer A (Start Push Layer), the select exit layer B (Select Exit Layer), the select enter layer C (Select Enter Layer), and the end push layer D (End Push Layer) can be located within the hover collider 30 (Hover collider). The select cancel layer E (Select Cancel Layer) can be located behind the virtual keyboard.

[0247] Hover collider 30: When the fingertip (poke gesture collider 21) enters the hover collider 30 from the front or side, the hover state can be triggered, and the hover state is left when the fingertip leaves the hover collider 30.

[0248] Start Push Layer A: If the interactive object is similar to a three-dimensional virtual button 11, the Start Push Layer A is the position of the virtual button 11 when it is released.

[0249] End Push Layer D: If the interactive object is similar to a three-dimensional virtual button 11, the End Push Layer D is the position where the virtual button 11 is pressed to the bottom.

[0250] For example, for a two-dimensional virtual keyboard, the end push layer (End Push Layer) may be the location of the virtual keyboard.

[0251] FIG2F shows a schematic diagram of a three-dimensional virtual key. The pressing direction of virtual key 11 is the Z axis, and the key surface is located in the XY plane. A cube is the hover collider 30 corresponding to virtual key 11. The plane where virtual key 11 is located when lifted is the Start Push Layer A or Start Push Plane, and the plane where it is located when pressed is the End Push Layer D or End Push Plane.

[0252] Among them, during the process of poking or grabbing, it is difficult for the user to control the position of the fingertip to remain absolutely still at one point. The position of the poke gesture collider 21 (poke collider) may have slight jitter, especially during the process of poking, grabbing or dragging. In the absence of a physical keyboard to provide physical feedback, it is difficult for the finger to maintain the same distance from the virtual keyboard during the movement. If only one position is defined to trigger the selection event, there will be a problem in which the poke gesture collider 21 (poke collider) repeatedly triggers the interactive event of entering the selection state (Select Entered) and the interactive event of exiting the selection state (Select Exited) near the position of the operated virtual key. In order to solve this problem, it is necessary to define two thresholds for entering the selection state and exiting the selection state. Usually, the threshold for exiting the selection state is looser than the threshold for entering the selection state, that is, it is more difficult to leave the selection state after entering the selection state, which can achieve the effect of debouncing.

[0253] Select Enter Layer C: Located between End Push Layer D and Start Push Layer A, it defines the depth at which the poke collider 21 enters the Select state during a user's press action.

[0254] Select Exit Layer B: Located between End Push Layer D and Start Push Layer A, this layer defines the depth where the poke collider 21 leaves the Select state during a user's lift action. This layer is typically placed before Select Enter Layer C.

[0255] Select Cancel Layer E: Located behind the virtual keyboard, it is the location where the Select state is canceled. Because virtual keys are virtual and cannot limit the user's actual range of motion, when the user pokes an interactive object (such as a virtual key), the physical hand can pass through the interactive object, and in most cases it is difficult to control the physical hand to stay just before the End Push Layer. Therefore, it is necessary to define a Select Cancel Layer. When the user's finger only unconsciously passes through the End Push Layer, the selection state can be maintained. Only when there is a large offset value on the Z axis will it be determined that the user is consciously controlling the physical hand to leave the effective range of the interactive object. At this time, the selection event triggered by the cancellation history is canceled.

[0256] As shown in the front view of Figure 2G, Select Cancel Margin 40 is similar to Select Cancel Layer E, but differs in that it defines the boundary of the object's deselected state in the XY plane. This defines the amount of XY movement allowed for the user's fingertip (poke gesture collision body 21) after entering the selected state, i.e., the Select Exit Threshold K5. For example, for a scroll view, a larger margin in the scrolling direction is required.

[0257] In some embodiments, determining the interaction event between the current object and the virtual keyboard based on the starting point and end point of the poke gesture trajectory, and the poke gesture collision body and the hovering collision body includes: when a first poke gesture collision body corresponding to the starting point of the poke gesture trajectory is outside the hovering collision body, and a second poke gesture collision body corresponding to the end point of the poke gesture trajectory enters the hovering collision body along the movement direction of the poke gesture trajectory, and the second poke gesture collision body is between the starting push layer and the ending push layer, determining the interaction event as entering the hovering state; when the first poke gesture collision body is in the hovering state, and the second poke gesture collision body leaves the hovering collision body along the movement direction of the poke gesture trajectory, determining the interaction event as Exit the hovering state; when the first poke gesture collider is in the hovering state and the second poke gesture collider enters the area between the selection entry layer and the end push layer along the movement direction of the poke gesture trajectory, the interaction event is determined to be entering the selection state; when the first poke gesture collider is in the selection state and the second poke gesture collider leaves the area between the selection exit layer and the end push layer along the movement direction of the poke gesture trajectory, the interaction event is determined to be exiting the selection state; when the first poke gesture collider is in the selection state and the second poke gesture collider passes through the selection cancellation layer in the vertical direction along the movement direction of the poke gesture trajectory and is located outside the selection cancellation layer, the interaction event is determined to be canceling the selection state.

[0258] In some embodiments, when the first poke gesture collider is in a hovering state and the second poke gesture collider enters the area between the selection entry layer and the end push layer along the movement direction of the poke gesture trajectory, determining that the interaction event is entering the selection state includes: when the poke gesture trajectory passes through the selection entry layer from the front of the hovering collider, when the first poke gesture collider is located before the selection entry layer and the second poke gesture collider is located after the selection entry layer, determining that the interaction event is entering the selection state; when the poke gesture trajectory enters the selection entry layer from the side of the hovering collider, when the second poke gesture collider enters the hovering collider between the selection entry layer and the end push layer, determining that the interaction event is entering the selection state.

[0259] Among them, regarding the triggering conditions of the interaction event, first, set the judgment variable a and the judgment variable b.

[0260] Judgment variable a: represents the starting point of the poke trajectory, that is, the poke point of the previous frame; during the judgment process, the judgment variable a can be converted into the first poke gesture collision body corresponding to the starting point of the poke gesture trajectory.

[0261] Judgment variable b: represents the end point of the poke gesture trajectory, that is, the poke point of the current frame; during the judgment process, the judgment variable b can be converted into a second poke gesture collision body corresponding to the end point of the poke gesture trajectory.

[0262] The first and second poke colliders can correspond to the fingertip of the same finger performing the poke gesture. The difference between the two is that they are located at different positions in three-dimensional space. The first poke collider is activated at the fingertip position corresponding to the starting point of the poke gesture trajectory, while the second poke collider is activated at the fingertip position corresponding to the end point of the poke gesture trajectory.

[0263] Interaction event: Entering the hover state (Hover Entered).

[0264] Trigger condition: b enters the hover collider and is located before the end push layer.

[0265] Interaction event: Exit the hover state (Hover Exited).

[0266] Trigger condition: a was previously in the hover state, and b leaves the hover collider.

[0267] Interaction event: Enter the selected state (Select Entered).

[0268] Trigger condition: When the start and end points of the poke trajectory are inconsistent, that is, when the fingertip moves quickly, check the relationship between a, b and the Select Enter Layer to calculate whether it is a valid selection.

[0269] Among them, when the poke gesture trajectory passes through the Select Enter Layer plane, it needs to meet the following conditions at the same time: a Before the Select Enter Layer; b After the Select Enter Layer; Among them, when the poke gesture enters from the side of the hover collider: it is only allowed to enter the hover collider between the Select Enter Layer and the End Push Layer, where the Select Enter condition is a subset of the Hover Enter condition.

[0270] Interaction event: Exit the selected state (Select Exited).

[0271] Trigger condition: a was previously selected, and b is no longer in the valid selection area: the Z-axis coordinate is raised to before the Select Exit Layer, or the XY-axis coordinates leave the Select Exit Margin.

[0272] Interaction event: Cancel selection state (Select Canceled).

[0273] Trigger condition: a was previously selected, and b's position exceeds the Select Cancel Layer on the Z axis, or exceeds the Select Cancel Margin on the XY axis.

[0274] As shown in Figures 2D and 2E , a Select Enter Threshold K1, a Select Exit Threshold K2, a Start Push Layer Threshold K3, and a Cancel Threshold K4 can be set. These thresholds can correspond to the degree of selectivity. For example, the Select Enter Threshold K1 can correspond to a selectivity of 0.9, the Select Exit Threshold K2 can correspond to a selectivity of 0.1, the Start Push Layer Threshold K3 can correspond to a selectivity of 0, and the Cancel Threshold K4 can correspond to a selectivity greater than or equal to 1.

[0275] As shown in Figures 2H and 2I, during the button pressing process, the left coordinate axis represents the degree of selection (Selectedness) of the button component. Taking the Select Enter Threshold K1 (Select Enter Threshold) as 0.9 as an example, when the pressing degree exceeds the Select Enter Threshold K1, the interaction event of entering the selected state (Select Entered) is triggered; after entering the selected state, when the button is lifted to the Select Exit Threshold K2 (Select Exit Threshold), the interaction event of exiting the selected state (Select Exited) is triggered.

[0276] Please refer to Figures 2D and 2I for the calculation of the selectedness. When the poke collider 21 is at the start push layer A, the selectedness is 0. When it is pressed to the end push layer D, the selectedness is 1. In between, the corresponding selectedness is calculated by normalizing the Z-axis position of the poke collider 21 and the relative relationship between the two planes (start push layer A and end push layer D).

[0277] For indirect interactors, such as the Gaze Pinch Interactor, the interactor provides a select progress independent of the interacted object, indicating the degree of selection by the interactor, such as the pinch amount. On the interacted object side, the select progress is converted to selectedness.

[0278] When an interactive object receives selection operations from multiple interactors, the final selectedness is determined by the maximum value.

[0279] Step 150: Output interaction feedback information in response to the interaction event, where the interaction feedback information at least includes a visual indication.

[0280] When an interaction event is detected, the system generates corresponding interaction feedback information based on the type and nature of the interaction event. This feedback information can be presented in various forms, including at least visual indications. Visual indications are an intuitive and easy-to-understand way to indicate the effects of user operations through visual elements such as graphics, animations, and colors. For example, when a user selects a virtual button, the button may emit a different color or animate to help the user better understand their operation. For example, when a user presses a virtual button, the button may display an animation effect of being pressed.

[0281] In addition to visual instructions, the system can also output other forms of interactive feedback information as needed, such as auditory instructions (sound effects), vibration feedback, etc. These feedback information can enhance the user's perception and improve the intuitiveness and naturalness of the interaction.

[0282] In some embodiments, the output of interaction feedback information in response to the interaction event includes: when the interaction event is entering a hovering state, identifying a target button, and displaying a visual indication of a light effect on the target button, the target button being a virtual button with the poke gesture collision body hovering above it; when the interaction event is exiting the hovering state, canceling the visual indication of the light effect of the target button; when the interaction event is entering a selection state, displaying a visual indication that the target button sinks; and when the interaction event is exiting a selection state or canceling a selection state, displaying a visual indication that the target button returns to its initial position.

[0283] For example, when a user's gesture hovers over a virtual key on the virtual keyboard, the system identifies the target key. To guide the user's gaze and actions, the system displays a visual indicator with a light effect on the target key. This light effect can be a simple color change, a highlight, or a dynamic halo effect, making the target key stand out more among other virtual keys.

[0284] For example, when a user's finger appears above the input method's virtual keyboard, gazing at specific content will cause an enhanced lighting effect to appear within the keyboard container, helping the user quickly locate the currently hovered key. This lighting effect guides the user's gaze, making it easier for the user to focus on the keyboard element they are operating. This makes it easier for the user to find the character or command they want to enter, thereby improving input accuracy and efficiency.

[0285] For example, as shown in FIG2J , when the user's finger hovers over the virtual keyboard 10, the target key 111 at the corresponding position will gradually become highlighted to guide the user's sight and operation. This highlighting can be achieved by modifying the material or texture of the 3D model, or by changing the lighting or shadow effects to enhance the visual effect.

[0286] For example, when the user's gesture leaves the virtual keyboard or exits the hover state, the system removes the visual indication of the light effect on the target key. This is done to clearly indicate that the user's operation intention has ended and let the user know what to do next.

[0287] For example, when the interactive event is to enter the selection state, that is, when the user explicitly selects a virtual key as the target, the system will confirm the operation by displaying a sinking visual indication. The sinking effect can simulate the effect of pressing a physical key, enhancing the user's perception and immersion. For example, when the user's finger continues to move downward and collides with the virtual keyboard, the virtual key at the corresponding position will trigger a touch event. The touch event can be a collision event or a touch event. Once the touch event is detected, the system will change the position or shape of the virtual key by modifying the transformation (Transform) property of the 3D model. For example, by lowering the Z-axis coordinate value of the virtual key, the effect of the key sinking can be simulated to visually simulate physical feedback. This physical feedback can enable the user to more clearly feel the impact of the user's operation on the virtual keyboard, thereby enhancing the realism of the interaction.

[0288] For example, as shown in FIG2K , when the user's finger is in full contact with the target key 111 and a downward pressing action occurs, the system will further detect the action, and the target key 111 of the virtual keyboard 11 will generate a visual indication of sinking.

[0289] For example, if the user cancels the action or moves to another location after selecting a virtual key, the system will reflect this action by visually indicating that the target key has returned to its original position. This helps keep the virtual keyboard clean and consistent and helps users better understand and track their actions.

[0290] Furthermore, specific functions can be implemented based on the user's finger movements. For example, as the user's finger moves across the virtual keyboard, the user's desired character can be predicted based on the finger's trajectory, and corresponding candidate characters will automatically pop up for the user to select. This predictive technology can greatly improve input speed and accuracy, providing users with a more intelligent and efficient input experience.

[0291] For example, in addition to generating sinking feedback, the sound and touch of a virtual key being fully pressed can also be simulated. This provides users with a complete and continuous input experience, making them feel like they are really operating a physical keyboard.

[0292] Throughout the input process, the system needs to continuously detect user gestures and operations, and adjust and optimize based on user input and feedback. For example, when the user stops inputting for a period of time, the system can automatically restore to its initial state or enter sleep mode to save resources.

[0293] In addition, in order to provide more natural and intelligent interaction, the system can also analyze and learn based on user habits and historical input data, thereby gradually optimizing the layout, key size, spacing and other parameters of the virtual keyboard to better adapt to the needs and habits of different users.

[0294] In some embodiments, displaying a visual indication of a bright effect on the target key includes: displaying a visual indication of a bright effect on the target key by changing a parameter of a shader corresponding to the target key.

[0295] When a Poke Interactor identifies an interactive object (such as a target button) as a valid target, the interaction state enters hover mode, or selected pending mode. Hover mode signals the user's interaction intent but typically does not change the state of the interactive object. This state transition can cause some visual feedback, such as a change in cursor or button color, which is defined as Poke hover feedback.

[0296] When the system detects a user's finger approaching a virtual key, it identifies the virtual key as the target key and then creates a lighting effect for the target key by changing the parameters of the shader. This requires writing the shader using the shader language (GLSL) of the Open Graphics Library for Embedded Systems (OpenGL ES).

[0297] In some embodiments, the visual indication of displaying a bright effect on the target key includes: when the distance between the poke gesture collision body and the virtual keyboard is a first preset distance, displaying a halo effect on the virtual keyboard; when the poke gesture collision body gradually approaches the virtual keyboard and the distance between the poke gesture collision body and the target key is a third preset distance, displaying a visual indication component with a highlight effect in front of the target key based on a fourth preset distance.

[0298] In some embodiments, displaying a halo effect on the virtual keyboard further includes: when the poke gesture collision body gradually approaches the virtual keyboard, and the distance between the poke gesture collision body and the target key is between the second preset distance and the third preset distance, displaying the halo effect gradually becoming shorter and brighter as the distance approaches.

[0299] In some embodiments, the visual indication component that displays a highlight effect in front of the target key based on the fourth preset distance includes: when the poke gesture collision body gradually approaches the virtual keyboard, a visual indication component is displayed in front of the target key with a highlight degree gradually changing from small to large based on the fourth preset distance.

[0300] As shown in FIG2L , when the distance between the poke gesture collision body 21 corresponding to the finger and the virtual keyboard 10 is a first preset distance (e.g., 20 cm), a halo effect 50 appears on the virtual keyboard 10. When the distance between the poke gesture collision body 21 corresponding to the finger and the virtual keyboard 10 is between a second preset distance (e.g., 10 cm) and a third preset distance (e.g., 3 cm), the halo effect 50 gradually becomes shorter and brighter as the distance gets closer.

[0301] As shown in Figures 2M and 2N, when the poke gesture collision body 21 corresponding to the finger continues to approach the virtual keyboard 10, and the distance between the poke gesture collision body 21 and the target key (clickable element) on the virtual keyboard 10 is a third preset distance (for example, 3 cm), a visual indication component 60 with a highlight effect appears in front of the target key at a fourth preset distance (for example, 0.8 cm). For example, the visual indication component 60 can be a translucent white sheet.

[0302] As shown in FIG. 2O , the user can complete a click event by pressing the visual indication component 60 onto the virtual keyboard 10 .

[0303] For example, the visual indication component 60 (white piece) can be formed by superimposing two layers of color: one layer is a color layer in which the entire component shape is filled with 8% white; and the other layer is a color layer with a softlight halo.

[0304] For example, the shape of the visual indication component 60 can be drawn as a rounded rectangle with a fixed corner radius of 16dp according to the outer border of the target key, which is read by the development framework and passed to the runtime environment so that the shape of the white hot area coincides with the outline of the target key.

[0305] The attraction area shown in FIG. 2L , FIG. 2N , and FIG. 2O may represent the depth (dp) of the finger pressing, for example, the depth (dp) may be 8.

[0306] In some embodiments, displaying the visual indication of the target button sinking includes: following the moving speed and moving distance of the poke gesture, displaying the visual indication of the visual indication component sinking until the visual indication component sinks to the end push layer and stops sinking.

[0307] When a user's finger interacts with a virtual key on the virtual keyboard, the system tracks the speed and distance of the poke gesture in real time. This tracking is based on the motion trajectory of the poke gesture's collision volume, accurately capturing the finger's path in space. To simulate the effect of pressing a physical key, the system displays a visual indicator component on the target key. This visual indicator dynamically adjusts based on the speed and distance of the poke gesture, creating a visual effect of sinking. This sinking effect gradually deepens as the poke gesture progresses until it reaches the desired depth.

[0308] When the visual indicator component sinks to the end push layer, the sinking action will stop, which can provide users with a more realistic and natural button pressing experience.

[0309] To enhance user perception and immersion, the visual indicator component's sinking effect simulates the real-world situation of pressing a physical button. Specifically, the visual indicator component sinks along the tip of the finger performing the poking gesture, so that the user feels like they are actually pressing a real button.

[0310] To maintain consistency and naturalness, the visual indicator's descending speed matches the speed of the poke gesture. This means that if the user quickly presses the virtual button, the visual indicator will descend at the same speed. Conversely, if the user's poke gesture is slow, the visual indicator will descend more slowly.

[0311] In addition to the sinking effect, the system also simulates the rebound effect of a physical button. This rebound effect is dynamically adjusted based on the location of the poke gesture and the user's action intention. For example, if the user cancels the operation or moves to another location after pressing the virtual button, the visual indicator component will produce a rebound effect to simulate the rebound action when the physical button is released. This rebound effect is also adjusted accordingly based on the location and speed of the poke gesture to provide a more realistic and natural interactive experience.

[0312] In some embodiments, the interactive feedback information further includes an auditory indication; when displaying the visual indication that the target button has sunk, the method further includes: outputting the auditory indication, wherein the auditory indication is used to represent that the target button has been pressed.

[0313] To provide a more immersive experience when displaying the visual indication of a target key sinking, Android's built-in SoundPool class can be used to play a short sound effect as an auditory indicator when a key sinks. This sound effect simulates the sound of a virtual key being pressed, providing auditory feedback to the user. The SoundPool class is used in Android to play short sound effects. It is more efficient than the Media Player and is suitable for playing simple sound effects.

[0314] In some embodiments, the method further includes: presenting a user interaction interface in the three-dimensional environment, wherein an input box is displayed on the user interaction interface; when it is detected that the interaction event is an exit selection state, obtaining character information corresponding to the target button, and displaying the input content corresponding to the character information in the input box.

[0315] Wherein, a user interaction interface is presented in a three-dimensional environment, and an input box is displayed on the user interaction interface.

[0316] For example, the user interface is displayed in a three-dimensional environment. This typically involves using specific virtual reality technologies, such as head-mounted displays (HMDs) and controllers, to create and display the three-dimensional environment. Within this three-dimensional environment, users can view computer-generated virtual images and receive visual feedback through devices such as the HMD. When displaying the user interface, it is necessary to ensure that it is presented clearly and accurately to the user within the three-dimensional environment. The user interface's position, content, layout, and color can be pre-configured to provide a user-friendly interactive experience. It is also necessary to ensure that the user interface interacts properly with other virtual objects (such as virtual objects and avatars) within the three-dimensional environment, allowing users to interact with the virtual environment through various operations. For example, the user interface can be a user interface corresponding to a client. This user interface can be customized based on the specific needs of the client to provide the functions and information required by the user. For example, multiple clients or applications can be presented in the three-dimensional environment. In response to a launch operation for a client, the user interface corresponding to the launched client can be presented within the three-dimensional environment, with an input box displayed on the user interface.

[0317] For example, an input box can be set up on the user interface. The input box is an important interface for users to interact with the system. Users can enter text, numbers, instructions, and other information through the input box. The input box must conform to the overall style and visual effects of the 3D environment.

[0318] To ensure that users can easily find and use the input box, you can use some guidance and markings on the user interface. For example, you can use obvious labels or icons to indicate the location of the input box, or add dynamic effects around the input box to attract users' attention.

[0319] For example, a virtual keyboard can be displayed in a three-dimensional environment in response to a first operation on an input box. For example, the first operation includes but is not limited to a somatosensory control operation, a gesture control operation, an eye movement operation, a touch operation, a voice control instruction, or an operation on an external control device. For example, a user can select an input box in the user interaction interface by triggering a preset button on an extended reality controller (such as a handle of a VR device), and select the input box by, for example, a virtual cursor to trigger an editing instruction for the input box, thereby waking up the virtual keyboard to display the virtual keyboard in the three-dimensional environment. The virtual keyboard is displayed with multiple virtual keys. For example, gesture control operation is also a common method. The user can make specific gestures in the three-dimensional environment, such as pinching, sliding, or tapping, to trigger the first operation on the input box. The system recognizes these gestures and then displays the virtual keyboard in the three-dimensional environment. For example, eye movement operation is also a novel human-computer interaction method. By using eye tracking technology, the system can detect the user's eye movement. When the user's gaze is focused on the input box, it can be regarded as the first operation on the input box, and then the virtual keyboard is displayed in the three-dimensional environment. For example, the extended reality device also supports touch functions. The user can directly click or touch the input box on the user interaction interface presented in the three-dimensional environment to trigger the first operation for the input box, and then display the virtual keyboard in the three-dimensional environment. For example, voice control instructions are also a convenient operation method. The user can select the input box or issue an editing instruction by voice, which can be regarded as the first operation for the input box, and then display the virtual keyboard in the three-dimensional environment. For example, for users who use external control devices, such as game controllers or remote controls, they can perform corresponding operations by connecting to the extended reality device, triggering the first operation for the input box, and then displaying the virtual keyboard in the three-dimensional environment.

[0320] When a user's interaction with the virtual keyboard is detected as "exiting selection," the system immediately retrieves the character information corresponding to the target key selected by the user. This character information may be letters, numbers, symbols, etc., depending on the virtual key selected by the user. After obtaining this character information, the system converts it into the corresponding input content and displays it in real time in the input box. This process allows users to intuitively see their input content and verify or modify it.

[0321] For example, when the user's finger leaves the target key, the system detects this action and begins processing the user's input. First, the system needs to determine the character the user pressed when they left the target key. This can be determined by identifying the target key where the user's finger last rested. For example, if the user presses the "A" key and then leaves, the system will recognize that the user entered the character "A". Once the character entered by the user is determined, the system needs to pass it to the input box. In the Android system, the input method manager (InputMethodManager) is used to manage the focus interaction of the input box. By calling methods of the Android InputMethodManager class, the system can pass the character entered by the user to the input box that currently has focus. In this way, the user's input can be correctly received and displayed in the corresponding input box.

[0322] The disclosed embodiments provide a virtual keyboard with a 3D visual effect by transforming traditional 2D input methods into 3D. By displaying the virtual keyboard in a 3D environment, the user's interaction with the virtual keyboard is enhanced. The multiple virtual keys on the virtual keyboard allow users to quickly and accurately input information, improving input efficiency. When users interact with the virtual keyboard, immediate feedback is provided through visual indicators, allowing users to intuitively understand the effects of their actions, making the interaction more intuitive and natural. By providing interactive feedback, the disclosed embodiments complement the pressure feedback provided when contacting the virtual keyboard in XR space, enhancing the user's typing experience. Specifically, when a user's finger contacts the virtual keyboard, the relative spatial position of the finger, the virtual keyboard, and the keyboard keys is identified. When a finger is over a key on the keyboard in XR space, the corresponding virtual key below it displays a selected highlight. As the finger approaches the virtual key surface, the highlighting becomes more pronounced, helping the user locate the target key on the virtual keyboard. When the finger contacts the target virtual key, the state of the virtual key changes, completing the press action. At the same time, a pressing sound effect is generated to provide feedback to the user. At the same time, the input box affects the input behavior and the user completes the input action.

[0323] To facilitate better implementation of the interactive method of the embodiment of the present disclosure, please refer to the processing flow diagram shown in Figure 2P.

[0324] The entire processing flow may include an initialization phase, a rendering phase, a gesture detection phase, and an input feedback phase.

[0325] The initialization phase may include steps S1 to S7.

[0326] S1. Initialization: Initialization is the first step of the product, that is, the new product startup stage.

[0327] S2. Activity initialization (Activity onCreate): In the lifecycle of an Android application, when a new activity is started, onCreate() is the first method to be executed. It is used to perform some basic initialization settings, such as creating the user interface. After executing step S2, you can execute step S3 or step D6.

[0328] S3. Calling the Open Graphics Library for Embedded Systems (OpenGL ES): OpenGL ES is a subset of the 3D graphics application programming interface OpenGL, designed for embedded devices such as mobile phones.

[0329] S4. Loading 3D models and textures: A 3D virtual keyboard is created. Each virtual key in the 3D virtual keyboard is an independent 3D model and has its own texture, material, and mesh.

[0330] S5. Set camera parameters and initial shader: When developing graphics-related applications, "setting camera parameters" and "initializing shader" are two important steps.

[0331] 1) Setting Camera Parameters: Camera parameters generally refer to the important factors that define the view matrix, such as the viewpoint position, observation point, and viewing angle. These parameters control the perspective from which the user sees the 3D world. For example, you can change the camera's position to simulate "movement" or change the observation point to "see" in different directions.

[0332] 2) Initializing Shaders: Shaders control specific stages in the rendering pipeline. For example, vertex shaders process vertex data, and pixel shaders process pixel colors. Initializing shaders involves creating and configuring these shaders.

[0333] S6. Start the input method manager (inputMethodManager): InputMethodManager is an Android system service responsible for managing keyboard input.

[0334] S7. Activity on Resume: This method is typically used in Android to indicate that an activity is now visible and that normal user interaction has resumed. Generally, after onResume() returns, the app will directly receive user input, such as keyboard taps.

[0335] The rendering stage may include steps S8 to S10.

[0336] S8. Rendering a 3D virtual keyboard: Rendering a 3D virtual keyboard using a graphics processing unit (GPU) and a shader program.

[0337] S9. Clear the buffer: gIClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT) is a method used in the OpenGL graphics library to clear the buffer. GL_COLOR_BUFFER_BIT and GL_DEPTH_BUFFER_BIT represent the color buffer and depth buffer, respectively, in preparation for the next rendering step.

[0338] S10. Render each virtual key using the primitive drawing function (glDrawElements): glDrawElements is a critical function in OpenGL for rendering graphic elements (such as points, lines, and triangles). This function renders all the virtual keys of a virtual keyboard. After defining the geometry and position of each virtual key, it is drawn by calling glDrawElements. In this step, OpenGL uses the previously set vertex and shader data.

[0339] The gesture detection stage may include steps S11 to S16.

[0340] S11. Calling the Hand Tracking API: The Hand Tracking API is a feature provided by the operating system (OS) to applications for identifying and tracking hand gestures and movements. In VR, applications use this API to monitor the user's fingers or entire hand to control the virtual keyboard.

[0341] S12. Gesture detection: The operation gesture of the current object can be recognized based on the gesture recognition library of the XR device.

[0342] S13. Determine whether the finger position is close to the key; if so, execute step S14; if not, return to execute step S12.

[0343] S14. Adjusting shader parameters: This is a common operation in 3D graphics programming. It can dynamically change the display of graphics, such as changing the color of an object. When the user's finger is over a certain button, the shader is used to change the color of the button to give the user feedback that the button has been selected.

[0344] S15. Displays a visual indication that the key is highlighted.

[0345] S16. Determine whether the finger presses the key; if so, execute step S17; if not, return to execute step D15.

[0346] The input feedback stage may include steps S17 to S20.

[0347] S17. Control the Z-axis of a view to change state and display a visual indication of a button being depressed: The z-axis refers to the height of the view on the screen. This visual indication gives the impression of a button being depressed.

[0348] S18. Outputting sound effect feedback based on the sound effect pool (SoundPool): The sound effect pool (SoundPool) is a class in Android that is used to efficiently play short and large numbers of audio clips. In the embodiment of the present disclosure, it can be used to play sound effect feedback of key presses.

[0349] S19. Get input behavior: InputMethodManager updateFullscreenMode()doStartinput() is a class in the Android system that manages the input of the system's virtual keyboard. Applications can use it to get user input behavior.

[0350] S20. Complete the input.

[0351] All of the above technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0352] The disclosed embodiments display a three-dimensional environment generated by an extended reality device, and then display a three-dimensional virtual keyboard within the three-dimensional environment. The virtual keyboard displays multiple virtual keys. A poke gesture of the current object is recognized. Based on the poke gesture, the poke gesture collision body corresponding to the poke gesture, and the hover collision body corresponding to the virtual key in the virtual keyboard, an interaction event between the current object and the virtual keyboard is determined. In response to the interaction event, interaction feedback information is output. The interaction feedback information includes at least a visual indication. The disclosed embodiments provide a virtual keyboard with a three-dimensional visual effect. By displaying the three-dimensional virtual keyboard within the three-dimensional environment, the keyboard layout and keys are more intuitive, enhancing the convenience of user interaction with the virtual keyboard. The multiple virtual keys on the virtual keyboard enable users to quickly and accurately input information, improving input efficiency. When a user interacts with the virtual keyboard, the recognition of the user's poke gesture can understand the user's intent, further optimizing interaction with the virtual keyboard. By combining the poke gesture collision body corresponding to the poke gesture and the hover collision body corresponding to the virtual key in the virtual keyboard, the interaction event between the current object and the virtual keyboard can be accurately determined. Instant feedback of interaction information through visual indications allows users to intuitively understand the effects of their operations, improving the intuitiveness and naturalness of interaction.

[0353] In order to facilitate better implementation of the interaction method of the embodiment of the present disclosure, the embodiment of the present disclosure also provides an interaction device. Please refer to Figure 2Q, which is a structural diagram of the interaction device provided by the embodiment of the present disclosure. Among them, the interaction device 200 may include: a first display unit 210, for displaying a three-dimensional environment generated by an extended reality device; a second display unit 220, for displaying a three-dimensional virtual keyboard in the three-dimensional environment, and a plurality of virtual keys are displayed on the virtual keyboard; an identification unit 230, for identifying the poke gesture of the current object; a determination unit 240, for determining the interaction event between the current object and the virtual keyboard based on the poke gesture, the poke gesture collision body corresponding to the poke gesture, and the hover collision body corresponding to the virtual key in the virtual keyboard; an interaction unit 250, for outputting interaction feedback information in response to the interaction event between the current object and the virtual keyboard, and the interaction feedback information at least includes a visual indication.

[0354] In some embodiments, the determination unit 240 is used to: construct a poke gesture collision body corresponding to the gesture interactor based on the poke gesture, and the position of the poke gesture collision body corresponds to the fingertip position of the finger that generates the poke gesture; for any virtual key on the virtual keyboard, construct a hover collision body corresponding to the virtual key, and the position of the hover collision body corresponds to the position of the virtual key; obtain the starting point and end point of the poke gesture trajectory based on the poke gesture; determine the interaction event between the current object and the virtual keyboard according to the starting point and end point of the poke gesture trajectory, and the poke gesture collision body and the hover collision body.

[0355] In some embodiments, when constructing the hover collision body corresponding to the virtual key, the determination unit 240 is used to: construct the hover collision body corresponding to the virtual key in front of the virtual keyboard, and the hover collision body includes a starting push layer, a selection exit layer, a selection entry layer and an end push layer constructed in sequence along the vertical direction of the virtual key perpendicular to the virtual keyboard; the determination unit 240 is also used to construct a selection cancel layer behind the virtual keyboard.

[0356] In some embodiments, when determining the interaction event between the current object and the virtual keyboard based on the starting point and end point of the poke gesture trajectory, and the poke gesture collision body and the hovering collision body, the determination unit 240 can be used to: when the first poke gesture collision body corresponding to the starting point of the poke gesture trajectory is outside the hovering collision body, and the second poke gesture collision body corresponding to the end point of the poke gesture trajectory along the movement direction of the poke gesture trajectory enters the hovering collision body, and the second poke gesture collision body is located between the starting push layer and the ending push layer, determine that the interaction event is entering the hovering state; when the first poke gesture collision body is in the hovering state, and the second poke gesture collision body leaves the hovering collision body along the movement direction of the poke gesture trajectory, determine that The interaction event is to exit the hovering state; when the first poke gesture collision body is in the hovering state and the second poke gesture collision body enters the area between the selection entry layer and the end push layer along the movement direction of the poke gesture trajectory, the interaction event is determined to be to enter the selection state; when the first poke gesture collision body is in the selection state and the second poke gesture collision body leaves the area between the selection exit layer and the end push layer along the movement direction of the poke gesture trajectory, the interaction event is determined to be to exit the selection state; when the first poke gesture collision body is in the selection state and the second poke gesture collision body passes through the selection cancellation layer in the vertical direction along the movement direction of the poke gesture trajectory and is located outside the selection cancellation layer, the interaction event is determined to be to cancel the selection state.

[0357] In some embodiments, when determining that the interaction event is entering the selection state, the determination unit 240 can be used to: when the poke gesture trajectory passes through the selection entry layer from the front of the hovering collision body, when the first poke gesture collision body is located before the selection entry layer and the second poke gesture collision body is located after the selection entry layer, determine that the interaction event is entering the selection state; when the poke gesture trajectory enters the selection entry layer from the side of the hovering collision body, when the second poke gesture collision body enters the hovering collision body between the selection entry layer and the end push layer, determine that the interaction event is entering the selection state.

[0358] In some embodiments, when the interaction unit 250 outputs interaction feedback information in response to the interaction event, it can be used to: when the interaction event is entering a hovering state, identify the target button and display a visual indication of a bright effect on the target button, where the target button is a virtual button with the poke gesture collision body hovering above it; when the interaction event is exiting the hovering state, cancel the visual indication of the bright effect of the target button; when the interaction event is entering a selection state, display a visual indication that the target button sinks; and when the interaction event is exiting a selection state or canceling a selection state, display a visual indication that the target button returns to its initial position.

[0359] In some embodiments, when the interaction unit 250 displays a visual indication of a bright effect on the target key, it can be used to: display a halo effect on the virtual keyboard when the distance between the poke gesture collision body and the virtual keyboard is a first preset distance; when the poke gesture collision body gradually approaches the virtual keyboard and the distance between the poke gesture collision body and the target key is a third preset distance, display a visual indication component with a highlight effect in front of the target key based on a fourth preset distance.

[0360] In some embodiments, when the interaction unit 250 displays a visual indication component with a highlight effect in front of the target key based on a fourth preset distance, it can be used to: when the poke gesture collision body gradually approaches the virtual keyboard, display a visual indication component with a highlight degree gradually changing from small to large in front of the target key based on the fourth preset distance.

[0361] In some embodiments, when displaying the visual indication of the target button sinking, the interaction unit 250 can be used to: follow the moving speed and moving distance of the poke gesture, display the visual indication of the sinking of the visual indication component until the visual indication component sinks to the end push layer and stops sinking.

[0362] In some embodiments, when the interaction unit 250 displays the visual indication of the light effect on the target key, it can be used to: display the visual indication of the light effect on the target key by changing the parameters of the shader corresponding to the target key.

[0363] In some embodiments, the interactive feedback information also includes an auditory indication; when displaying the visual indication that the target button has sunk, the interactive unit 250 can also be used to: output the auditory indication, where the auditory indication is used to indicate that the target button has been pressed.

[0364] In some embodiments, the interaction unit 250 can also be used to: present a user interaction interface in the three-dimensional environment, and an input box is displayed on the user interaction interface; when it is detected that the interaction event is an exit selection state, the character information corresponding to the target button is obtained, and the input content corresponding to the character information is displayed in the input box.

[0365] Each unit in the interactive device 200 may be implemented in whole or in part by software, hardware, or a combination thereof. Each unit may be embedded in or independent of a processor in a terminal device in the form of hardware, or may be stored in a memory in the terminal device in the form of software, so that the processor can call and execute the corresponding operations of each unit.

[0366] The interactive device 200 may be integrated into a terminal or server that has a storage device and a processor and has computing capabilities, or the interactive device 200 may be the terminal or server.

[0367] The disclosed embodiments provide a virtual keyboard with a three-dimensional visual effect. By displaying the three-dimensional virtual keyboard in the three-dimensional environment, the keyboard layout and keys are made more intuitive, enhancing the convenience of user interaction with the virtual keyboard. The multiple virtual keys on the virtual keyboard allow users to input information quickly and accurately, improving input efficiency. When a user interacts with the virtual keyboard, the user's poke gesture is recognized to understand the user's intention and further optimize the interaction with the virtual keyboard. The poke gesture collision body corresponding to the poke gesture and the hover collision body corresponding to the virtual key in the virtual keyboard can be combined to accurately determine the interaction event between the current object and the virtual keyboard. The interaction information is immediately fed back in the form of visual indications, allowing the user to intuitively understand the effect of their operation, thereby improving the intuitiveness and naturalness of the interaction.

[0368] In some embodiments, the present disclosure further provides a terminal device including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0369] As shown in FIG2R , FIG2R is a schematic diagram of the structure of a terminal device provided in an embodiment of the present disclosure. The terminal device 300 can generally be provided in the form of glasses, a head-mounted display (HMD), or contact lenses to achieve visual perception and other forms of perception. Of course, the form of the terminal device is not limited to this and can be further miniaturized or enlarged as needed. The terminal device 300 may include, but is not limited to, the following components.

[0370] Detection module 301: Uses various sensors to detect the user's operation commands and acts on the virtual environment, such as continuously updating the image displayed on the display screen following the user's line of sight, realizing the user's interaction with the virtual and scene, for example, continuously updating the real content based on the detected direction of the user's head rotation.

[0371] Feedback module 302: Receives data from sensors and provides real-time feedback to the user. The feedback module 302 may be configured to display a graphical user interface, such as a virtual environment, and may include a display screen.

[0372] Sensor 303: On the one hand, it receives operation commands from the user and applies them to the virtual environment; on the other hand, it provides the results generated after the operation to the user in the form of various feedback.

[0373] Control module 304: controls sensors and various input / output devices, including obtaining user data (such as movements and voice) and outputting perception data, such as images, vibrations, temperature, and sounds, etc., to affect the user, the virtual environment, and the real world.

[0374] Modeling module 305: constructs a three-dimensional model of the virtual environment, and may also include various feedback mechanisms such as sound and touch in the three-dimensional model.

[0375] In an embodiment of the present disclosure, a virtual scene in a three-dimensional environment can be constructed by the modeling module 305; the three-dimensional environment generated by the extended reality device can be displayed by the feedback module 302, and a three-dimensional virtual keyboard can be displayed in the three-dimensional environment, with multiple virtual keys displayed on the virtual keyboard; the poke gesture of the current object can be identified by the detection module 301 and / or the sensor 303; the interaction event between the current object and the virtual keyboard can be determined by the control module 304 based on the poke gesture, the poke gesture collision body corresponding to the poke gesture, and the hover collision body corresponding to the virtual key in the virtual keyboard; and the feedback module 302 can output interaction feedback information in response to the interaction event, and the interaction feedback information can include at least a visual indication.

[0376] In some embodiments, as shown in FIG2S , which is another schematic diagram of the structure of a terminal device provided in an embodiment of the present disclosure, the terminal device 300 further includes a processor 310 having one or more processing cores, a memory 320 having one or more computer-readable storage media, and a computer program stored in the memory 320 and executable on the processor. The processor 310 is electrically connected to the memory 320. Those skilled in the art will appreciate that the terminal device structure shown in the figure does not limit the terminal device, and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0377] The processor 310 is the control center of the terminal device 300. It uses various interfaces and lines to connect various parts of the entire terminal device 300. By running or loading software programs and / or modules stored in the memory 320, and calling data stored in the memory 320, it executes various functions of the terminal device 300 and processes data, thereby monitoring the terminal device 300 as a whole.

[0378] In an embodiment of the present disclosure, the processor 310 in the terminal device 300 will load instructions corresponding to the processes of one or more applications into the memory 320 in accordance with the following steps, and the processor 310 will run the applications stored in the memory 320 to implement various functions: displaying a three-dimensional environment generated by an extended reality device; displaying a three-dimensional virtual keyboard in the three-dimensional environment, with multiple virtual keys displayed on the virtual keyboard; identifying a poke gesture of a current object; determining an interaction event between the current object and the virtual keyboard based on the poke gesture, the poke gesture collision body corresponding to the poke gesture, and the hover collision body corresponding to the virtual key in the virtual keyboard; and outputting interaction feedback information in response to the interaction event, wherein the interaction feedback information includes at least a visual indication.

[0379] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0380] In some embodiments, the processor 310 may include a detection module 301 , a control module 304 , and a modeling module 305 .

[0381] In some embodiments, as shown in FIG2S , terminal device 300 further includes: a radio frequency circuit 306, an audio circuit 307, and a power supply 308. The processor 310 is electrically connected to the memory 320, the feedback module 302, the sensor 303, the radio frequency circuit 306, the audio circuit 307, and the power supply 308, respectively. Those skilled in the art will appreciate that the terminal device structures shown in FIG2R or FIG2S do not limit the terminal device, and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0382] The radio frequency circuit 306 may be used to transmit and receive radio frequency signals, so as to establish wireless communication with a network device or other terminal devices through wireless communication, and to transmit and receive signals with the network device or other terminal devices.

[0383] The audio circuit 307 can be used to provide an audio interface between the user and the terminal device through a speaker and microphone. The audio circuit 307 can convert received audio data into electrical signals and transmit them to the speaker, which then converts them into sound signals for output. The microphone, on the other hand, converts collected sound signals into electrical signals, which are then received by the audio circuit 307 and converted into audio data. The audio data is then processed by the output processor 310 and transmitted via the RF circuit 306 to, for example, another terminal device, or the audio data is output to a memory for further processing. The audio circuit 307 may also include an earphone jack to provide communication between an external headset and the terminal device.

[0384] The power supply 308 is used to supply power to various components of the terminal device 300 .

[0385] Although not shown in FIG. 2R or FIG. 2S , the terminal device 300 may further include a camera, a wireless fidelity module, a Bluetooth module, an input module, etc., which will not be described in detail here.

[0386] In some embodiments, the present disclosure further provides a computer-readable storage medium for storing a computer program. The computer-readable storage medium can be applied to a terminal device or a server, and the computer program causes the terminal device or server to execute the corresponding processes in the interaction method in the embodiments of the present disclosure. For the sake of brevity, these processes are not further described here.

[0387] In some embodiments, the present disclosure further provides a computer program product, comprising a computer program stored in a computer-readable storage medium. A processor of a terminal device reads the computer program from the computer-readable storage medium and executes the computer program, causing the terminal device to perform the corresponding processes of the interaction method in the embodiments of the present disclosure. For the sake of brevity, these processes are not further described here.

[0388] The present disclosure further provides a computer program, which includes a computer program stored in a computer-readable storage medium. A processor of a terminal device reads the computer program from the computer-readable storage medium and executes the computer program, causing the terminal device to perform the corresponding process of the interaction method in the embodiment of the present disclosure. For the sake of brevity, the details are not repeated here.

[0389] It should be understood that the processor of the embodiments of the present disclosure may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above-mentioned method embodiment can be completed by hardware integrated logic circuits in the processor or by software instructions. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above-mentioned method.

[0390] It is understood that the memory in the embodiments of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0391] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0392] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0393] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0394] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0395] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of this embodiment.

[0396] In addition, each functional unit in the embodiment of the present disclosure may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a terminal device (which may be a personal computer, a server) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk.

[0397] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A human-computer interaction method, comprising: Displaying at least one interactive object in a virtual space, each of the interactive objects comprising at least one virtual control, each of the virtual controls comprising at least two response layers; Determining a real-time operation position of the touch operation according to a touch operation on any of the virtual controls; When it is determined that the real-time operation position of the touch operation is located at any response layer of the virtual control, the virtual control is controlled to execute an interactive event associated with the response layer.

2. The method according to claim 1, wherein the at least two response layers include a hover collision layer and a touch end layer, and when determining that the real-time operation position of the touch operation is located in any response layer of the virtual control, controlling the virtual control to execute an interaction event associated with the response layer comprises: When the real-time operation position of the touch operation is located in the hover collision layer, visually highlighting the virtual control; When the real-time operation position of the touch operation is located at the touch end layer, the interaction event associated with the virtual control is executed on the interaction object. 3 . The method according to claim 2 , wherein visually highlighting the virtual control comprises at least one of: size enlargement and color adjustment.

4. The method according to claim 2, wherein the at least two response layers further include a touch start layer, the touch start layer being located between the hover collision layer and the touch end layer; Accordingly, when determining that the real-time operation position of the touch operation is located at any response layer of the virtual control, controlling the virtual control to execute an interactive event associated with the response layer includes: When the real-time operation position of the touch operation is located in the touch start layer, a touch operation layer supporting user touch is displayed. The method according to claim 4 , wherein a display position of the touch operation layer changes following a real-time operation position of the touch operation.

6. The method according to claim 4, wherein the at least two response layers further include a touch cancellation layer, and the touch cancellation layer is located after the touch end layer; Accordingly, when determining that the real-time operation position of the touch operation is located at any response layer of the virtual control, controlling the virtual control to execute an interactive event associated with the response layer includes: When the real-time operation position of the touch operation is located in the touch cancellation layer, the selection operation on the virtual control is canceled.

7. The method according to claim 6, wherein the at least two response layers further include a selection input layer and a selection exit layer, the selection input layer and the selection exit layer are located between the touch start layer and the touch end layer, and the selection input layer is located after the selection exit layer; Accordingly, when determining that the real-time operation position of the touch operation is located at any response layer of the virtual control, controlling the virtual control to execute an interactive event associated with the response layer includes: When the real-time operation position of the touch operation is located in the selected input layer, updating the display mode of the virtual control; When the real-time operation position of the touch operation is located at the selection and exit layer, the interaction event associated with the virtual control is executed on the interaction object.

8. The method according to any one of claims 2 to 7, further comprising: When it is determined that the real-time operation position of the touch operation is located in any response layer of the virtual control, outputting vibration feedback and / or sound effect feedback corresponding to the response layer; Wherein, each response layer of the virtual control corresponds to different vibration feedback and / or sound effect feedback.

9. The method according to claim 1, wherein the input direction of the touch operation on any of the virtual controls includes a forward direction and a side direction, and the method further comprises: When the input direction of the touch operation on any of the virtual controls is a lateral direction, the virtual control is controlled to execute an interaction event corresponding to the position information according to the position information of the touch operation in the lateral direction.

10. A human-computer interaction device, comprising: A display module, configured to display at least one interactive object in a virtual space, each of the interactive objects comprising at least one virtual control, and each of the virtual controls comprising at least two response layers; a determination module, configured to determine a real-time operation position of the touch operation according to the touch operation on any of the virtual controls; The control module is configured to control the virtual control to execute an interactive event associated with the response layer when determining that the real-time operation position of the touch operation is located at any response layer of the virtual control.

11. An interactive method comprising: Displaying a three-dimensional environment generated by an extended reality device; Displaying a three-dimensional virtual keyboard in the three-dimensional environment, wherein a plurality of virtual keys are displayed on the virtual keyboard; Recognize the poke gesture of the current object; determining, according to the poke gesture, a poke gesture collision body corresponding to the poke gesture, and a hover collision body corresponding to the virtual key in the virtual keyboard, an interaction event between the current object and the virtual keyboard; In response to the interaction event, interaction feedback information is output, where the interaction feedback information at least includes a visual indication.

12. The interaction method according to claim 11, wherein determining the interaction event between the current object and the virtual keyboard based on the poke gesture, the poke gesture collision body corresponding to the poke gesture, and the hover collision body corresponding to the virtual key in the virtual keyboard comprises: Constructing a poke gesture collision body corresponding to a gesture interactor based on the poke gesture, wherein a position of the poke gesture collision body corresponds to a fingertip position of a finger generating the poke gesture; For any virtual key on the virtual keyboard, construct a hover collision body corresponding to the virtual key, where the position of the hover collision body corresponds to the position of the virtual key; Obtaining a start point and an end point of a poking gesture trajectory based on the poking gesture; An interaction event between the current object and the virtual keyboard is determined according to a starting point and an end point of the poke gesture trajectory, and the poke gesture collision body and the hovering collision body.

13. The interaction method according to claim 12, wherein constructing a hover collision volume corresponding to the virtual button comprises: Constructing a hover collision body corresponding to the virtual key in front of the virtual keyboard, the hover collision body including a start push layer, a selection exit layer, a selection entry layer, and an end push layer sequentially constructed along a direction perpendicular to the virtual key and the virtual keyboard; The method further comprises: A selection cancellation layer is constructed behind the virtual keyboard.

14. The interaction method according to claim 13, wherein determining the interaction event between the current object and the virtual keyboard based on the starting point and the end point of the poke gesture trajectory, and the poke gesture collision body and the hovering collision body comprises: When a first poke gesture collision body corresponding to the starting point of the poke gesture trajectory is outside the hovering collision body, and a second poke gesture collision body corresponding to the end point of the poke gesture trajectory along the movement direction of the poke gesture trajectory enters the hovering collision body, and the second poke gesture collision body is between the starting push layer and the ending push layer, the interaction event is determined to be entering the hovering state; When the first poke gesture collision body is in a hovering state and the second poke gesture collision body leaves the hovering collision body along the movement direction of the poke gesture trajectory, determining that the interaction event is exiting the hovering state; When the first poke gesture collision body is in a hovering state and the second poke gesture collision body enters the area between the selection entry layer and the end push layer along the movement direction of the poke gesture trajectory, the interaction event is determined to be entering the selection state; When the first poke gesture collision body is in the selection state and the second poke gesture collision body leaves the area between the selection exit layer and the end push layer along the movement direction of the poke gesture trajectory, the interaction event is determined to be the exit selection state; When the first poke gesture collision body is in the selected state and the second poke gesture collision body passes through the selection cancellation layer in the vertical direction along the movement direction of the poke gesture trajectory and is located outside the selection cancellation layer, the interaction event is determined to be in the deselected state.

15. The interaction method according to claim 14, wherein when the first poke gesture collision body is in a hovering state and the second poke gesture collision body enters the area between the selection entry layer and the end push layer along the movement direction of the poke gesture trajectory, determining that the interaction event is entering the selection state comprises: In a case where the poke gesture trajectory passes through the selection entry layer from the front of the hovering collision body, when the first poke gesture collision body is located before the selection entry layer and the second poke gesture collision body is located after the selection entry layer, determining that the interaction event is an entry selection state; In the case where the poke gesture trajectory enters the selection entry layer from a side of the hovering collision body, when the second poke gesture collision body enters the hovering collision body from between the selection entry layer and the end push layer, the interaction event is determined to be entering the selection state.

16. The interaction method according to any one of claims 13 to 15, wherein the outputting interaction feedback information in response to the interaction event comprises: When the interaction event is entering a hover state, identifying a target button and displaying a visual indication of a light effect on the target button, the target button being a virtual button with the poke gesture collision body hovering above it; When the interaction event is exiting the hover state, canceling the visual indication of the light effect of the target button; When the interaction event is entering a selection state, displaying a visual indication that the target button is sinking; as well as When the interaction event is exiting the selection state or canceling the selection state, a visual indication is displayed that the target button is restored to the initial position.

17. The interaction method according to claim 16, wherein the visual indication of displaying a light effect on the target key comprises: When the distance between the poke gesture collision body and the virtual keyboard is a first preset distance, displaying a halo effect on the virtual keyboard; When the poke gesture collision body gradually approaches the virtual keyboard and the distance between the poke gesture collision body and the target key is a third preset distance, a visual indication component with a highlight effect is displayed in front of the target key based on a fourth preset distance.

18. The interaction method according to claim 17, wherein the visual indication component that displays a highlight effect in front of the target key based on the fourth preset distance comprises: When the poke gesture collision body gradually approaches the virtual keyboard, a visual indication component with a highlight level gradually changing from small to large is displayed in front of the target key based on a fourth preset distance.

19. The interaction method according to claim 18, wherein displaying a visual indication that the target button is sunk comprises: Following the moving speed and moving distance of the poking gesture, a visual indication of the visual indication component sinking is displayed until the visual indication component sinks to the end pushing layer and stops sinking.

20. The interaction method according to claim 16, wherein the visual indication of displaying a light effect on the target button comprises: By changing the parameters of the shader corresponding to the target button, a visual indication of the light effect is displayed on the target button.

21. The interactive method according to claim 16, wherein the interactive feedback information further comprises an auditory indication; When displaying the visual indication that the target button is sinking, the method further includes: The auditory indication is output, where the auditory indication is used to indicate that the target key is pressed.

22. The interactive method according to claim 16, wherein the method further comprises: Presenting a user interaction interface in the three-dimensional environment, wherein the user interaction interface displays an input box; When it is detected that the interaction event is an exit selection state, character information corresponding to the target button is obtained, and input content corresponding to the character information is displayed in the input box.

23. An interactive device comprising: A first display unit, configured to display a three-dimensional environment generated by an extended reality device; a second display unit, configured to display a three-dimensional virtual keyboard in the three-dimensional environment, wherein the virtual keyboard displays a plurality of virtual keys; A recognition unit, used to recognize the poking gesture of the current object; a determining unit, configured to determine an interaction event between the current object and the virtual keyboard according to the poke gesture, a poke gesture collision body corresponding to the poke gesture, and a hover collision body corresponding to the virtual key in the virtual keyboard; The interaction unit is configured to output interaction feedback information in response to the interaction event, where the interaction feedback information at least includes a visual indication.

24. A computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by a processor to execute the human-computer interaction method according to any one of claims 1 to 9 or the interaction method according to any one of claims 11 to 22.

25. A terminal device, comprising a processor and a memory, wherein a computer program is stored in the memory, and the processor is configured to execute the human-computer interaction method according to any one of claims 1 to 9 or the interaction method according to any one of claims 11 to 22 by calling the computer program stored in the memory.

26. A computer program product, the computer program product being tangibly stored in a computer storage medium and comprising computer executable instructions, which, when executed by a device, cause the device to perform the human-computer interaction method according to any one of claims 1 to 9 or the interaction method according to any one of claims 11 to 22.

Images