Interaction method and device, storage medium and equipment

Abstract

The invention discloses an interaction method and device, a storage medium and equipment. The method comprises the steps of displaying a three-dimensional environment generated by augmented reality equipment; displaying a three-dimensional virtual keyboard in the three-dimensional environment, wherein a plurality of virtual keys are displayed on the virtual keyboard; identifying a stamping gesture of the current object; determining an interaction event between the current object and the virtual keyboard according to a stamp gesture, a stamp gesture collision body corresponding to the stamp gesture and a hovering collision body corresponding to a virtual key in the virtual keyboard; in response to the interaction event, interaction feedback information is output, and the interaction feedback information at least comprises the visual indication. According to the virtual keyboard with the three-dimensional visual effect, interaction feedback information can be output, interaction convenience and input efficiency of the virtual keyboard are improved, and interaction intuition and naturalness are improved.

Description

Technical Field

[0001] The present application relates to the field of extended reality (XR) technology, and in particular to an interaction method, apparatus, storage medium, and device. Background Art

[0002] With the rise of the metaverse and the widespread adoption of XR devices, users are placing higher demands on the interactive experience and input efficiency of XR devices. However, traditional XR device input methods have many shortcomings, such as their limited understanding of XR device input interactions and the lack of physical feedback from virtual keyboards. These issues result in a poor user experience and low input efficiency when using XR devices. Summary of the Invention

[0003] The embodiments of the present application provide an interaction method, apparatus, storage medium, and device that can output interaction feedback information, improve the interaction convenience and input efficiency of a virtual keyboard, and improve the intuitiveness and naturalness of the interaction.

[0004] In one aspect, an embodiment of the present application provides an interaction method, the method comprising:

[0005] Displaying a three-dimensional environment generated by an extended reality device;

[0006] Recognize the poke gesture of the current object;

[0007] determining, according to the poke gesture, a poke gesture collision body corresponding to the poke gesture, and a hover collision body corresponding to the virtual key in the virtual keyboard, an interaction event between the current object and the virtual keyboard;

[0008] Displaying a three-dimensional virtual keyboard in the three-dimensional environment, wherein a plurality of virtual keys are displayed on the virtual keyboard;

[0009] In response to the interaction event, interaction feedback information is output, where the interaction feedback information at least includes a visual indication.

[0010] On the other hand, an embodiment of the present application provides an interactive device, comprising:

[0011] A first display unit, configured to display a three-dimensional environment generated by an extended reality device;

[0012] a second display unit, configured to display a virtual keyboard in the three-dimensional environment, wherein the virtual keyboard displays a plurality of virtual keys;

[0013] A recognition unit, used to recognize the poking gesture of the current object;

[0014] a determining unit, configured to determine an interaction event between the current object and the virtual keyboard according to the poke gesture, a poke gesture collision body corresponding to the poke gesture, and a hover collision body corresponding to the virtual key in the virtual keyboard;

[0015] The interaction unit is used for the interaction event to output interaction feedback information, and the interaction feedback information at least includes a visual indication.

[0016] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded by a processor to execute the interaction method described in any of the above embodiments.

[0017] On the other hand, an embodiment of the present application provides a terminal device, which includes a processor and a memory, wherein a computer program is stored in the memory, and the processor is used to execute the interaction method described in any of the above embodiments by calling the computer program stored in the memory.

[0018] The embodiment of the present application displays a three-dimensional environment generated by an extended reality device; displays a three-dimensional virtual keyboard in the three-dimensional environment, wherein a plurality of virtual keys are displayed on the virtual keyboard; recognizes a poke gesture of a current object; determines an interaction event between the current object and the virtual keyboard based on the poke gesture, the poke gesture collision body corresponding to the poke gesture, and the hover collision body corresponding to the virtual key in the virtual keyboard; and outputs interaction feedback information in response to the interaction event, wherein the interaction feedback information includes at least a visual indication. The embodiment of the present application provides a virtual keyboard with a three-dimensional visual effect. By displaying the three-dimensional virtual keyboard in the three-dimensional environment, the layout and keys of the keyboard are made more intuitive, thereby enhancing the convenience of user interaction with the virtual keyboard. The multiple virtual keys on the virtual keyboard enable users to input information quickly and accurately, thereby improving input efficiency. When a user interacts with the virtual keyboard, by recognizing the user's poke gesture, the user's intention can be understood, further optimizing the interaction with the virtual keyboard. By combining the poke gesture collision body corresponding to the poke gesture and the hover collision body corresponding to the virtual key in the virtual keyboard, the interaction event between the current object and the virtual keyboard can be accurately determined. The interaction information is immediately fed back in the form of a visual indication, allowing the user to intuitively understand the effect of their operation, thereby improving the intuitiveness and naturalness of the interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A flowchart of the interaction method provided in an embodiment of the present application.

[0021] Figure 2 A schematic diagram of a first application scenario of the interaction method provided in an embodiment of the present application.

[0022] Figure 3 A schematic diagram of a second application scenario of the interaction method provided in an embodiment of the present application.

[0023] Figure 4 A schematic diagram of a third application scenario of the interaction method provided in an embodiment of the present application.

[0024] Figure 5 A schematic diagram of a fourth application scenario of the interaction method provided in an embodiment of the present application.

[0025] Figure 6 A schematic diagram of the fifth application scenario of the interaction method provided in an embodiment of the present application.

[0026] Figure 7 A schematic diagram of the sixth application scenario of the interaction method provided in an embodiment of the present application.

[0027] Figure 8 Schematic diagram of the seventh application scenario of the interaction method provided in an embodiment of the present application.

[0028] Figure 9 Schematic diagram of the eighth application scenario of the interaction method provided in an embodiment of the present application.

[0029] Figure 10 A ninth application scenario diagram of the interaction method provided in an embodiment of the present application.

[0030] Figure 11 This is a schematic diagram of the tenth application scenario of the interaction method provided in an embodiment of the present application.

[0031] Figure 12 Schematic diagram of the eleventh application scenario of the interaction method provided in an embodiment of the present application.

[0032] Figure 13 Schematic diagram of the twelfth application scenario of the interaction method provided in an embodiment of the present application.

[0033] Figure 14 Schematic diagram of the thirteenth application scenario of the interaction method provided in an embodiment of the present application.

[0034] Figure 15 Schematic diagram of the fourteenth application scenario of the interaction method provided in an embodiment of the present application.

[0035] Figure 16Schematic diagram of the fifteenth application scenario of the interaction method provided in an embodiment of the present application.

[0036] Figure 17 A schematic diagram of the structure of the interactive device provided in an embodiment of the present application.

[0037] Figure 18 A first structural diagram of a terminal device provided in an embodiment of the present application.

[0038] Figure 19 A second structural diagram of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0040] The embodiments of the present application can be applied to various application scenarios such as extended reality (XR), virtual reality (VR), augmented reality (AR), and mixed reality (MR).

[0041] First, some nouns or terms that appear in the description of the embodiments are explained as follows:

[0042] A virtual scene is a virtual scene displayed (or provided) when an application is running on a terminal or server. Optionally, the virtual scene is a simulation of the real world, a semi-simulation and semi-fictional virtual environment, or a purely fictional virtual environment. A virtual scene can be any of a two-dimensional virtual scene and a three-dimensional virtual scene. The virtual environment can be the sky, land, ocean, etc., where the land includes environmental elements such as deserts and cities. A virtual scene is a scene with complete game logic, including virtual objects such as user controls.

[0043] A virtual object refers to a dynamic object that can be controlled in a virtual scene. Optionally, the dynamic object can be a virtual person, virtual animal, cartoon character, etc. The virtual object is a character controlled by the player through an input device, or an artificial intelligence (AI) set up in a virtual environment through training, or a non-player character (NPC) set up in a virtual scene battle. Optionally, the virtual object is a virtual person competing in the virtual scene. Optionally, the number of virtual objects in the virtual scene battle is preset or dynamically determined based on the number of clients joining the battle, which is not limited in the embodiments of the present application. In one possible implementation, the user can control the virtual object to move in the virtual scene, for example, to control the virtual object to run, jump, crawl, etc., and can also control the virtual object to fight other virtual objects using skills, virtual props, etc. provided by the application. Optionally, a virtual object can also refer to a static object that can be interacted with in the virtual scene, such as a virtual object, virtual control, interface element, virtual props, etc.

[0044] Extended Reality (XR) is a concept that includes Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR). It refers to the technology that creates an environment where the virtual world is connected to the real world, and users can interact with the environment in real time.

[0045] Virtual Reality (VR) is a technology for creating and experiencing a virtual world. It generates a virtual environment through computational means. It is a multi-source information (the virtual reality mentioned in this article includes at least visual perception, and can also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.). It realizes the simulation of the fusion of virtual environment, interactive three-dimensional dynamic vision and entity behavior, immersing users in a simulated three-dimensional environment, and realizing applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

[0046] Augmented Reality (AR) is a technology that uses real-world (or 3D) camera pose parameters to calculate the camera's position in the real world (also known as the real world) in real time while the camera is capturing images. Based on these pose parameters, virtual elements are added to the captured images. These virtual elements include, but are not limited to, images, videos, and 3D models. The goal of AR technology is to overlay the virtual world on the real world for interactive viewing on a screen.

[0047] Mixed Reality (MR) is a simulated setting that integrates computer-generated sensory input (e.g., virtual objects) with sensory input from a physical setting, or its representation. In some MR settings, the computer-generated sensory input can adapt to changes in sensory input from the physical setting. Additionally, some electronic systems used to render MR settings can monitor the orientation and / or position relative to the physical setting to enable virtual objects to interact with real objects (i.e., physical elements from the physical setting, or their representations). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.

[0048] Augmented Virtuality (AV): An AV set is a computer-created or virtual set that incorporates at least one sensory input from a physical set. The one or more sensory inputs from the physical set may be a representation of at least one feature of the physical set. For example, a virtual object may exhibit the colors of a physical element captured by one or more imaging sensors. In another example, a virtual object may exhibit features consistent with actual weather conditions in the physical set, as identified via weather-related imaging sensors and / or online weather data. In another example, an augmented reality forest may have virtual trees and structures, but animals may have features accurately reproduced from images of the physical animals.

[0049] Virtual field of view refers to the area in the virtual environment that the user can perceive through the lens in the extended reality device. The field of view (FOV) of the virtual field of view is used to represent the perceived area.

[0050] Extended reality devices, terminals that achieve extended reality effects, can usually be provided in the form of glasses, helmet-mounted displays (Head Mount Display, HMD), and contact lenses to achieve visual perception and other forms of perception. Of course, the form of extended reality devices is not limited to this and can be further miniaturized or enlarged as needed.

[0051] The extended reality devices described in the embodiments of this application may include but are not limited to the following types:

[0052] Computer-based virtual reality (PCVR) devices use the PC to perform calculations and output data related to virtual reality functions. External computer-based extended reality devices use the data output by the PC to achieve virtual reality effects.

[0053] Mobile extended reality devices support the configuration of mobile terminals (such as smartphones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations for virtual reality functions and outputs data to the mobile extended reality device, such as watching virtual reality videos through the mobile terminal's APP.

[0054] The all-in-one extended reality device has a processor for performing related calculations for virtual functions, and thus has independent virtual reality input and output functions. It does not need to be connected to a PC or mobile terminal and has a high degree of freedom in use.

[0055] With the rise of the metaverse and the rapid development of XR (Extended Reality) technology, XR devices, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), have garnered widespread market attention and user interest. These devices leverage advanced hardware technologies to provide users with an immersive interactive experience. However, despite significant hardware advancements, XR devices still have numerous shortcomings in their built-in software systems, particularly operating systems and input methods.

[0056] Currently, most XR devices are based on the Android operating system. While Android is quite mature in the mobile device space, it struggles when directly applied to XR devices. This is primarily because the interaction methods of XR devices differ significantly from those of traditional mobile devices, and the Android operating system is not optimized to account for these differences. As a result, users often encounter issues such as inconvenient operation and unfriendly interfaces when using XR devices.

[0057] Input methods are particularly problematic. Traditional Android input method vendors have accumulated extensive experience in mobile electronic devices, but this experience isn't fully applicable to XR devices. XR device input primarily relies on gesture recognition and a virtual keyboard, which differs significantly from the physical keyboards and touch input of traditional mobile devices. Due to a lack of in-depth understanding of XR device input interactions, traditional input methods perform poorly on XR devices.

[0058] Furthermore, virtual keyboards on XR devices face another challenge: a lack of physical feedback. On traditional mobile devices, users experience physical feedback from the device when typing, which helps improve input accuracy and efficiency. However, virtual keyboards on XR devices lack this physical feedback, leading to frequent issues like accidental and missed touches.

[0059] Regarding the input methods on XR devices, current traditional input method manufacturers have a very limited understanding of the input interactions and have not fully explored new user input interactions in this scenario. In order to improve the input methods on XR devices, the embodiments of the present application propose an interaction method. By designing a virtual keyboard with three-dimensional visual effects and a virtual physical feedback mechanism, pre-selected highlighting and press feedback for the target keys are implemented on the virtual keyboard, which can improve the interaction convenience and input efficiency of the virtual keyboard, as well as improve the intuitiveness and naturalness of the interaction.

[0060] It should be noted that the order of description of the following embodiments does not limit the priority order of the embodiments.

[0061] Each embodiment of the present application provides an interaction method, which can be executed by a terminal or a server, or jointly by a terminal and a server; the embodiments of the present application are described by taking the interaction method executed by a terminal device as an example.

[0062] See also Figures 1 to 16 , Figure 1 A flow chart of the interaction method provided in the embodiment of the present application is provided. Figures 2 to 16 These are all schematic diagrams of application scenarios of the interaction method provided in the embodiments of the present application. The method can be applied to a terminal device, which can include any one of an extended reality device, a virtual reality device, an augmented reality device, and a mixed reality device. The method includes the following steps 110 to 130:

[0063] Step 110: Display the three-dimensional environment generated by the extended reality device.

[0064] First, using the technology of extended reality devices, a realistic three-dimensional environment is constructed. This environment provides users with an immersive experience, allowing them to feel as if they are in a real yet virtual world. The three-dimensional environment generated by the extended reality device includes not only real space environments but also virtual space environments. The three-dimensional environment can be a virtual reality environment or an extended reality environment.

[0065] First, extended reality devices can have powerful graphics processing capabilities to generate high-quality 3D environments in real time. This usually involves using advanced graphics rendering techniques such as ray tracing, shadow mapping, texture mapping, etc. to create realistic 3D scenes.

[0066] To provide an immersive experience, extended reality devices can also have depth perception and stereoscopic imaging capabilities. This can be achieved by using technologies such as infrared sensors, depth cameras or lidar, which can capture detailed information about the surrounding environment and convert it into a three-dimensional model.

[0067] When generating a 3D environment, the user's head and eye movements can also be taken into account. XR devices can have high-precision motion tracking capabilities, capturing inputs such as the user's head posture, eye movements, and gestures in real time. Based on this input, they can adjust the perspective and details of the 3D environment to provide a natural, immersive experience.

[0068] To enhance the user experience, the 3D environment can also be integrated with a user interface. This interface can be customized to the application's needs, providing various functions and controls such as menus, buttons, and input boxes. These interface elements can be realistically rendered in the 3D environment, allowing users to interact intuitively.

[0069] Step 120: Display a three-dimensional virtual keyboard in the three-dimensional environment, where a plurality of virtual keys are displayed on the virtual keyboard.

[0070] For example, a three-dimensional environment must first be modeled and rendered to match the real world. Then, a three-dimensional (3D) virtual keyboard model is created within this 3D environment. This model consists of multiple 3D virtual keys. Each virtual key can be customized based on actual needs, including its shape, size, position, and color. The key layout on the virtual keyboard can be customized to meet the needs of different application scenarios. Furthermore, the virtual keyboard can support multiple input methods, such as touch, gestures, and voice, allowing users to choose the interaction method that best suits their habits and preferences.

[0071] For example, the display effect of the virtual keyboard can be integrated with the three-dimensional environment to provide a more realistic feeling. For example, a semi-transparent design can be used to make the virtual keyboard and the background environment complement each other, and effects such as shadows and lighting can be used to enhance the visual effect.

[0072] In an embodiment of the present application, a three-dimensional (3D) virtual keyboard model can be created using OpenGL for Embedded Systems (OpenGL ES) in an Android integrated development tool (such as Android Studio). OpenGL ES supports efficient 3D graphics rendering, and based on its capabilities, a realistic 3D virtual keyboard can be created. Each virtual key in the 3D virtual keyboard is an independent 3D model with its own texture, material, and mesh.

[0073] Among them, each virtual key in the 3D virtual keyboard is an independent 3D model, which means that each key has its own shape, size and appearance, and can be set with animation, lighting and other visual effects separately.

[0074] In 3D graphics, texture refers to an image or map applied to the surface of a model. It can be used to add texture, pattern, color or details to make the model look more realistic and rich.

[0075] Among them, the material defines the visual properties of the modeled object's surface, such as color, glossiness, reflection, etc. In 3D modeling, each modeled object or model can have one or more materials, which determine the appearance and texture of the modeled object when rendered.

[0076] A mesh is the basic structure of a 3D model, consisting of a series of vertices, edges, and faces. It defines the model's shape and appearance and determines how the model is rendered and displayed. Each independent 3D model has its own mesh structure.

[0077] For example, Figure 2 In the three-dimensional virtual keyboard 10 shown, the key layer where the virtual keys 11 are located may be slightly higher than the horizontal plane where the virtual keyboard 10 is located.

[0078] Step 130: Recognize the poking gesture of the current object.

[0079] For example, the current object is the user currently interacting with the virtual keyboard.

[0080] First, the system monitors the operation gestures of the current object in real time. In this case, a specific gesture is recognized as a poke gesture. A poke gesture is a form of gesture interaction. For example, the poke gesture can be a simple finger touch action, or a more complex combination of touch and press. For example, a poke gesture can refer to a touch or click operation performed by a user on a virtual keyboard using a finger or other object (such as a handle). By accurately recognizing this poke gesture, the system can understand the user's intention and operation.

[0081] The XR device's gesture recognition library can be used to identify the current object's manipulation gestures. This library provides an application programming interface (API) that can obtain information about the user's finger position and movement. For example, the Hand Tracking API can accurately track the user's finger movements, including speed, direction, and position.

[0082] Step 140 : determining an interaction event between the current object and the virtual keyboard according to the poke gesture, the poke gesture collision body corresponding to the poke gesture, and the hover collision body corresponding to the virtual key in the virtual keyboard.

[0083] After recognizing a poke gesture, the system further analyzes the relationship between the poke gesture and the virtual keyboard's position. For example, if the user makes a poke gesture above the virtual keyboard, the system will determine that this is an interaction event related to the virtual keyboard. This determination is made based on the location of the poke gesture and the display area of the virtual keyboard.

[0084] The system needs to determine the poke gesture's corresponding collision volume. A collision volume is a geometric shape used to detect collisions and interactions within the virtual environment. Based on the location and direction of the poke gesture, the system generates a corresponding poke gesture collision volume within the virtual environment.

[0085] The system needs to determine the hover collider corresponding to the virtual key in the virtual keyboard. The hover collider can be a geometric shape used to detect the proximity of the user's fingertip to the virtual key when the fingertip used to perform the poke gesture is not pressed. The system can generate the corresponding hover collider in the virtual environment based on the position and size of the virtual key.

[0086] By comparing the position and state of the poke collider and the hover collider, you can determine the interaction event between the current object and the virtual keyboard. For example, if the poke collider overlaps or is close to the hover collider of a virtual key, the system can determine that the user intends to press the virtual key and trigger the corresponding interaction event.

[0087] By recognizing the poke gesture, determining the poke gesture collider and hover collider, and comparing their states, the interaction event between the current object and the virtual keyboard can be accurately determined. This interaction method can improve the convenience and input efficiency of the virtual keyboard, as well as make the interaction more intuitive and natural.

[0088] In some embodiments, determining an interaction event between the current object and the virtual keyboard based on the poke gesture, a poke gesture collision body corresponding to the poke gesture, and a hover collision body corresponding to the virtual key in the virtual keyboard includes:

[0089] Constructing a poke gesture collision body corresponding to a gesture interactor based on the poke gesture, wherein a position of the poke gesture collision body corresponds to a fingertip position of a finger generating the poke gesture;

[0090] For any virtual key on the virtual keyboard, construct a hover collision body corresponding to the virtual key, where the position of the hover collision body corresponds to the position of the virtual key;

[0091] Obtaining a start point and an end point of a poking gesture trajectory based on the poking gesture;

[0092] An interaction event between the current object and the virtual keyboard is determined according to a starting point and an end point of the poke gesture trajectory, and the poke gesture collision body and the hovering collision body.

[0093] When a poke gesture is recognized, a PokeInteractor can be constructed based on the characteristics of the poke gesture. This PokeInteractor includes a Poke Collider. The PokeCollider represents the shape and size of the finger that generated the poke gesture, and its position corresponds to the position of the fingertip. This allows the system to more accurately simulate and track the actual movement trajectory of the finger.

[0094] For example, you can interact with the system through the gesture interactor (Poke Interactor) corresponding to the poke gesture to control the system to perform input and output.

[0095] For example, Figure 3 As shown in the interactive scene diagram, the gesture interactor 20 (Poke Interactor) needs to provide information about the poke gesture collision body 21 (Poke Collider) and the touch point 22 (Touch Point).

[0096] Among them, the gesture interactor 20 (Poke Interactor) acts as a direct interactor, allowing the user to complete the selection by directly touching the interactive object 1. For example, the interactive object 1 can be a user interface, a virtual keyboard, or a virtual button. The gesture interactor 20 (Poke Interactor) places a spherical poke gesture collider 21 (Poke Collider) on the fingertip and determines the interaction state based on the spatial relationship between the poke gesture collider 21 (Poke Collider) and the interactive object 1. Therefore, the interaction state cannot be determined by the input of the gesture interactor 20 itself. Some calculations need to be combined with the interaction operation with the interactive object 1 to perform gesture recognition.

[0097] The location where the poke gesture takes effect is the location of the poke gesture collider 21, which is determined by the aiming state (Aim Pose) output by the controller (Controller). The aiming state (Aim Pose) describes the user's aiming position and direction, which is determined based on the user's input (such as the position of the finger) and the device's posture information.

[0098] If the poke collider 21 is constructed based on a bare hand, the location where the poke gesture takes effect is the tip of the index finger.

[0099] If the poke collider 21 is constructed based on a handle, the location where the poke gesture takes effect is a point at the front end of the handle.

[0100] The effective radius of the poke gesture (Poke) is the radius of the poke gesture collision body 21 (Poke Collider).

[0101] The speed of the poke gesture (Poke) is determined by the speed of the poke gesture collider 21 (Poke Collider).

[0102] Scope of the Poke gesture: Applies only to general User Experience (UX) components.

[0103] To better detect and identify the user's interaction intent, a hover collider can be constructed for each virtual key on the virtual keyboard. The shape and size of these hover colliders can match the virtual key, and the position can correspond to the center or a specific position of each virtual key.

[0104] The motion trajectory of the poking gesture can be tracked, and the starting point and end point of the poking gesture trajectory can be determined. The tracking can be performed based on the motion of the poking gesture collision body, and can accurately capture the movement path of the finger in space.

[0105] The interaction event between the current object and the virtual keyboard can be determined based on the starting and ending points of the poke gesture trajectory, as well as the relationship between the poke gesture collider and the hover collider. For example, if the poke gesture trajectory intersects or approaches the hover collider of a virtual key, it is considered a valid interaction event and the corresponding action is performed.

[0106] In some embodiments, constructing a hover collision volume corresponding to the virtual button includes:

[0107] Constructing a hover collision body corresponding to the virtual key in front of the virtual keyboard, the hover collision body including a start push layer, a selection exit layer, a selection entry layer, and an end push layer sequentially constructed along a direction perpendicular to the virtual key and the virtual keyboard;

[0108] The method further includes: constructing a selection cancellation layer behind the virtual keyboard.

[0109] For example, see Figures 4 to 7, it is necessary to define the effective area where the poke gesture (Poke) takes effect on the interactive object. The effective area can be constructed by referring to the structure of three-dimensional virtual keys. The effective area of the poke gesture corresponding to each virtual key can include a hover collider 30 (Hover collider), two presentation layers and three state layers. Among them, the presentation layer is a layer related to vision. The two presentation layers can include the start push layer A (Start Push Layer) and the end push layer D (EndPush Layer). The state layer is used to determine the selection transition. The three state layers can include the select exit layer B (Select Exit Layer), the select enter layer C (Select Enter Layer) and the select cancel layer E (Select Cancel Layer). Among them, the start push layer A (Start Push Layer), the select exit layer B (Select Exit Layer), the select enter layer C (Select Enter Layer) and the end push layer D (End Push Layer) can be located within the hover collider 30 (Hover collider). The select cancel layer E (Select Cancel Layer) can be located behind the virtual keyboard.

[0110] Hover collider 30: When the fingertip (poke gesture collider 21) enters the hover collider 30 from the front or side, the hover state can be triggered, and the hover state is left when the fingertip leaves the hover collider 30.

[0111] Start Push Layer A: If the interactive object is similar to a three-dimensional virtual button 11, the Start Push Layer A is the position of the virtual button 11 when it is released.

[0112] End Push Layer D: If the interactive object is similar to a three-dimensional virtual button 11, the End Push Layer D is the position where the virtual button 11 is pressed to the bottom.

[0113] For example, for a two-dimensional virtual keyboard, the end push layer (End Push Layer) may be the location of the virtual keyboard.

[0114] like Figure 6The diagram shows a three-dimensional virtual key. Virtual key 11 is pressed in the Z-axis direction, and the key surface is located in the XY plane. A cube is the hover collider 30 corresponding to virtual key 11. The plane where virtual key 11 is lifted is called the Start Push Layer A or Start Push Plane, and the plane where it is pressed is called the End Push Layer D or End Push Plane.

[0115] Among them, during the process of poking or grabbing, it is difficult for the user to control the position of the fingertip to remain absolutely still at one point. The position of the poke gesture collider 21 (poke collider) may have slight jitter, especially during the process of poking, grabbing or dragging. In the absence of a physical keyboard to provide physical feedback, it is difficult for the finger to maintain the same distance from the virtual keyboard during movement. If only one position is defined to trigger the selection event, there will be a problem in which the poke gesture collider 21 (poke collider) repeatedly triggers the interactive event of entering the selection state (Select Entered) and the interactive event of exiting the selection state (Select Exited) near the position of the operated virtual key. In order to solve this problem, it is necessary to define two thresholds for entering the selection state and exiting the selection state. Usually, the threshold for exiting the selection state is looser than the threshold for entering the selection state, that is, it is more difficult to leave the selection state after entering the selection state, which can achieve the effect of debouncing.

[0116] Select Enter Layer C: Located between End Push Layer D and Start Push Layer A, it defines the depth at which the poke collider 21 enters the Select state during a user's press action.

[0117] Select Exit Layer B: Located between End Push Layer D and Start Push Layer A, this layer defines the depth where the poke collider 21 leaves the Select state during a user's lift action. This layer is typically placed before Select Enter Layer C.

[0118] Select Cancel Layer E: Located behind the virtual keyboard, it is the location where the Select state is canceled. Because virtual keys are virtual and cannot limit the user's actual range of motion, when the user pokes an interactive object (such as a virtual key), the physical hand can pass through the interactive object, and in most cases it is difficult to control the physical hand to stay just before the End Push Layer. Therefore, it is necessary to define a Select Cancel Layer. When the user's finger only unconsciously passes through the End Push Layer, the selection state can be maintained. Only when there is a large offset value on the Z axis will it be determined that the user is consciously controlling the physical hand to leave the effective range of the interactive object. At this time, the selection event triggered by the cancellation history is canceled.

[0119] like Figure 7 In the front view shown, the Select Cancel Margin 40 is similar to the Select Cancel Layer E. However, the Select Cancel Margin 40 defines the boundary of the object's deselected state in the XY plane. This defines the amount of XY movement allowed for the user's fingertip (the poke gesture collision body 21) after entering the selected state, i.e., the Select Exit Threshod K5. For example, for a scroll view, a larger margin in the scrolling direction is required.

[0120] In some embodiments, determining the interaction event between the current object and the virtual keyboard according to the starting point and the end point of the poke gesture trajectory, and the poke gesture collision body and the hover collision body includes:

[0121] When a first poke gesture collision body corresponding to the starting point of the poke gesture trajectory is outside the hovering collision body, and a second poke gesture collision body corresponding to the end point of the poke gesture trajectory along the movement direction of the poke gesture trajectory enters the hovering collision body, and the second poke gesture collision body is between the starting push layer and the ending push layer, the interaction event is determined to be entering the hovering state;

[0122] When the first poke gesture collision body is in a hovering state and the second poke gesture collision body leaves the hovering collision body along the movement direction of the poke gesture trajectory, determining that the interaction event is exiting the hovering state;

[0123] When the first poke gesture collision body is in a hovering state and the second poke gesture collision body enters the area between the selection entry layer and the end push layer along the movement direction of the poke gesture trajectory, the interaction event is determined to be entering the selection state;

[0124] When the first poke gesture collision body is in the selection state and the second poke gesture collision body leaves the area between the selection exit layer and the end push layer along the movement direction of the poke gesture trajectory, the interaction event is determined to be the exit selection state;

[0125] When the first poke gesture collision body is in the selected state and the second poke gesture collision body passes through the selection cancellation layer in the vertical direction along the movement direction of the poke gesture trajectory and is located outside the selection cancellation layer, the interaction event is determined to be in the deselected state.

[0126] In some embodiments, when the first poke gesture collision body is in a hovering state and the second poke gesture collision body enters an area between the selection entry layer and the end push layer along the movement direction of the poke gesture trajectory, determining that the interaction event is entering the selection state includes:

[0127] In a case where the poke gesture trajectory passes through the selection entry layer from the front of the hovering collision body, when the first poke gesture collision body is located before the selection entry layer and the second poke gesture collision body is located after the selection entry layer, determining that the interaction event is an entry selection state;

[0128] In the case where the poke gesture trajectory enters the selection entry layer from a side of the hovering collision body, when the second poke gesture collision body enters the hovering collision body from between the selection entry layer and the end push layer, the interaction event is determined to be entering the selection state.

[0129] Regarding the triggering conditions of the interaction event, first, set the judgment variables a and b:

[0130] Judgment variable a: represents the starting point of the poke trajectory, that is, the poke point of the previous frame; during the judgment process, the judgment variable a can be converted into the first poke gesture collision body corresponding to the starting point of the poke gesture trajectory.

[0131] Judgment variable b: represents the end point of the poke gesture trajectory, that is, the poke point of the current frame; during the judgment process, the judgment variable b can be converted into a second poke gesture collision body corresponding to the end point of the poke gesture trajectory.

[0132] The first and second poke colliders can correspond to the fingertip of the same finger performing the poke gesture. The difference between the two is that they are located at different positions in three-dimensional space. The first poke collider is activated at the fingertip position corresponding to the starting point of the poke gesture trajectory, while the second poke collider is activated at the fingertip position corresponding to the end point of the poke gesture trajectory.

[0133] Interaction event: enter the hover state (Hover Entered);

[0134] Trigger condition: b enters the hover collider and is located before the end push layer.

[0135] Interaction event: exit the hover state (Hover Exited);

[0136] Trigger condition: a was previously in the hover state, and b leaves the hover collider.

[0137] Interaction event: Enter the selected state (Select Entered);

[0138] Trigger condition: When the start and end points of the poke trajectory are inconsistent, that is, when the fingertip moves quickly, check the relationship between a, b and the Select Enter Layer to calculate whether it is a valid selection;

[0139] When the poke trajectory passes through the Select Enter Layer plane, it must simultaneously meet the following conditions: a) before the Select Enter Layer; b) after the Select Enter Layer.

[0140] Among them, when the poke gesture enters from the side of the hover collider: it is only allowed to enter the hover collider between the Select Enter Layer and the End Push Layer, where the Select Enter condition is a subset of the Hover Enter condition.

[0141] Interaction event: Exit selection state (Select Exited);

[0142] Trigger condition: a was previously selected, and b is no longer in the valid selection area: the Z-axis coordinate is raised to before the Select Exit Layer, or the XY-axis coordinates leave the Select Exit Margin.

[0143] Interaction event: cancel selection state (Select Canceled);

[0144] Trigger condition: a was previously selected, and b's position exceeds the SelectCancel Layer on the Z axis, or exceeds the SelectCancel Margin on the XY axis.

[0145] like Figure 4 and Figure 5 As shown, a Select Enter Threshold K1, a Select Exit Threshold K2, a Start Push Layer Threshold K3, and a Cancel Threshold K4 can be set. These thresholds can correspond to the degree of selectivity. For example, the Select Enter Threshold K1 can correspond to a selectivity of 0.9, the Select Exit Threshold K2 can correspond to a selectivity of 0.1, the Start Push Layer Threshold K3 can correspond to a selectivity of 0, and the Cancel Threshold K4 can correspond to a selectivity greater than or equal to 1.

[0146] like Figure 8 and Figure 9 As shown in the figure, during the button pressing process, the left coordinate axis represents the degree of selection (Selectedness) of the button component. Taking the Select Enter Threshold K1 (Select Enter Threshold) as 0.9 as an example, when the pressing degree exceeds the Select Enter Threshold K1, the interaction event of entering the selected state (Select Entered) is triggered; after entering the selected state, when the button is lifted to the Select Exit Threshold K2 (Select Exit Threshod), the interaction event of exiting the selected state (Select Exited) is triggered.

[0147] See also Figure 4 and Figure 9Regarding the calculation of selectedness, when the Poke Collider 21 is at the start push layer A (Start Push Layer), the selectedness is 0, and when it is pressed to the end push layer D (End Push Layer), the selectedness is 1. In between, the Z-axis position of the Poke Collider 21 (Poke Collider) and the relative relationship between the two planes (Start Push Layer A and End Push Layer D) are used to perform normalized calculation to obtain the corresponding selectedness.

[0148] For indirect interactors, such as the Gaze PinchInteractor, the interactor provides a select progress that is independent of the interacting object, indicating the degree of selection by the interactor, such as the pinch amount. On the interacting object side, the select progress is converted to selectedness.

[0149] When an interactive object receives selection operations from multiple interactors, the final selectedness is determined by the maximum value.

[0150] Step 150: Output interaction feedback information in response to the interaction event, where the interaction feedback information at least includes a visual indication.

[0151] When an interaction event is detected, the system generates corresponding interaction feedback information based on the type and nature of the interaction event. This feedback information can be presented in various forms, including at least visual indications. Visual indications are an intuitive and easy-to-understand way to indicate the effects of user operations through visual elements such as graphics, animations, and colors. For example, when a user selects a virtual button, the button may emit a different color or animate to help the user better understand their operation. For example, when a user presses a virtual button, the button may display an animation effect of being pressed.

[0152] In addition to visual instructions, the system can also output other forms of interactive feedback information as needed, such as auditory instructions (sound effects), vibration feedback, etc. These feedback information can enhance the user's perception and improve the intuitiveness and naturalness of the interaction.

[0153] In some embodiments, the outputting interaction feedback information in response to the interaction event includes:

[0154] When the interaction event is entering a hover state, identifying a target button and displaying a visual indication of a light effect on the target button, the target button being a virtual button with the poke gesture collision body hovering above it;

[0155] When the interaction event is exiting the hover state, canceling the visual indication of the light effect of the target button;

[0156] When the interaction event is entering a selection state, displaying a visual indication that the target button is sinking; and

[0157] When the interaction event is exiting the selection state or canceling the selection state, a visual indication is displayed that the target button is restored to the initial position.

[0158] For example, when a user's gesture hovers over a virtual key on the virtual keyboard, the system identifies the target key. To guide the user's gaze and actions, the system displays a visual indicator with a light effect on the target key. This light effect can be a simple color change, a highlight, or a dynamic halo effect, making the target key stand out more among other virtual keys.

[0159] For example, when a user's finger appears above the input method's virtual keyboard, gazing at specific content will cause an enhanced lighting effect to appear within the keyboard container, helping the user quickly locate the currently hovered key. This lighting effect guides the user's gaze, making it easier for the user to focus on the keyboard element they are operating. This makes it easier for the user to find the character or command they want to enter, thereby improving input accuracy and efficiency.

[0160] For example, Figure 10 As shown, when the user's finger hovers over the virtual keyboard 10, the target key 111 at the corresponding position will gradually become highlighted to guide the user's sight and operation. This highlighting can be achieved by modifying the material or texture of the 3D model, or by changing the lighting or shadow effects to enhance the visual effect.

[0161] For example, when the user's gesture leaves the virtual keyboard or exits the hover state, the system removes the visual indication of the light effect on the target key. This is done to clearly indicate that the user's operation intention has ended and let the user know what to do next.

[0162] For example, when the interactive event is to enter the selection state, that is, when the user explicitly selects a virtual key as the target, the system will confirm the operation by displaying a sinking visual indication. The sinking effect can simulate the effect of pressing a physical key, enhancing the user's perception and immersion. For example, when the user's finger continues to move downward and collides with the virtual keyboard, the virtual key at the corresponding position will trigger a touch event. The touch event can be a collision event or a touch event. Once the touch event is detected, the system will change the position or shape of the virtual key by modifying the transformation (Transform) property of the 3D model. For example, by lowering the Z-axis coordinate value of the virtual key, the effect of the key sinking can be simulated to visually simulate physical feedback. This physical feedback can enable the user to more clearly feel the impact of the user's operation on the virtual keyboard, thereby enhancing the realism of the interaction.

[0163] For example, Figure 11 As shown, when the user's finger is in full contact with the target key 111 and a downward pressing action occurs, the system will further detect the action, and the target key 111 of the virtual keyboard 11 will generate a visual indication of sinking.

[0164] For example, if the user cancels the action or moves to another location after selecting a virtual key, the system will reflect this action by visually indicating that the target key has returned to its original position. This helps keep the virtual keyboard clean and consistent and helps users better understand and track their actions.

[0165] Furthermore, specific functions can be implemented based on the user's finger movements. For example, as the user's finger moves across the virtual keyboard, the user's desired character can be predicted based on the finger's trajectory, and corresponding candidate characters will automatically pop up for the user to select. This predictive technology can greatly improve input speed and accuracy, providing users with a more intelligent and efficient input experience.

[0166] For example, in addition to generating sinking feedback, the sound and touch of a virtual key being fully pressed can also be simulated. This provides users with a complete and continuous input experience, making them feel like they are really operating a physical keyboard.

[0167] Throughout the input process, the system needs to continuously detect user gestures and operations, and adjust and optimize based on user input and feedback. For example, when the user stops inputting for a period of time, the system can automatically restore to its initial state or enter sleep mode to save resources.

[0168] In addition, in order to provide more natural and intelligent interaction, the system can also analyze and learn based on user habits and historical input data, thereby gradually optimizing the layout, key size, spacing and other parameters of the virtual keyboard to better adapt to the needs and habits of different users.

[0169] In some embodiments, displaying a visual indication of a bright effect on the target key includes: displaying a visual indication of a bright effect on the target key by changing a parameter of a shader corresponding to the target key.

[0170] When a Poke Interactor identifies an interactive object (such as a target button) as a valid target, the interaction state enters hover mode, or selected pending mode. Hover mode signals the user's interaction intent but typically does not change the state of the interactive object. This state transition can cause some visual feedback, such as a change in cursor or button color, which is defined as Poke hover feedback.

[0171] When the system detects a user's finger approaching a virtual key, it identifies the virtual key as the target key and then creates a lighting effect for the target key by changing the parameters of the shader. This requires writing the shader using the shader language (GLSL) of the Open Graphics Library for Embedded Systems (OpenGL ES).

[0172] In some embodiments, the visual indication of displaying a light effect on the target key includes:

[0173] When the distance between the poke gesture collision body and the virtual keyboard is a first preset distance, displaying a halo effect on the virtual keyboard;

[0174] When the poke gesture collision body gradually approaches the virtual keyboard and the distance between the poke gesture collision body and the target key is a third preset distance, a visual indication component with a highlight effect is displayed in front of the target key based on a fourth preset distance.

[0175] In some embodiments, displaying a halo effect on the virtual keyboard further includes:

[0176] When the poke gesture collision body gradually approaches the virtual keyboard, and the distance between the poke gesture collision body and the target key is between the second preset distance and the third preset distance, the halo effect is displayed to become shorter and brighter as the distance approaches.

[0177] In some embodiments, the visual indication component that displays a highlight effect in front of the target key based on the fourth preset distance includes:

[0178] When the poke gesture collision body gradually approaches the virtual keyboard, a visual indication component with a highlight level gradually changing from small to large is displayed in front of the target key based on a fourth preset distance.

[0179] like Figure 12 As shown, when the distance between the poke gesture collision body 21 corresponding to the finger and the virtual keyboard 10 is a first preset distance (e.g., 20 cm), a halo effect 50 appears on the virtual keyboard 10. When the distance between the poke gesture collision body 21 corresponding to the finger and the virtual keyboard 10 is between a second preset distance (e.g., 10 cm) and a third preset distance (e.g., 3 cm), the halo effect 50 gradually becomes shorter and brighter as the distance gets closer.

[0180] like Figure 13 and Figure 14 As shown, when the poke gesture collision body 21 corresponding to the finger continues to approach the virtual keyboard 10, and the distance between the poke gesture collision body 21 and the target key (clickable element) on the virtual keyboard 10 is a third preset distance (for example, 3 cm), a visual indication component 60 with a highlight effect appears in front of the target key at a fourth preset distance (for example, 0.8 cm). For example, the visual indication component 60 can be a translucent white sheet.

[0181] like Figure 15 As shown, the user can complete a click event by pressing the visual indication component 60 onto the virtual keyboard 10 .

[0182] For example, the visual indication component 60 (white piece) can be formed by superimposing two layers of color: one layer is a color layer in which the entire component shape is filled with 8% white; and the other layer is a color layer with a softlight halo.

[0183] For example, the shape of the visual indication component 60 can be drawn as a rounded rectangle with a fixed corner radius of 16dp according to the outer border of the target key, which is read by the development framework and passed to the runtime environment so that the shape of the white hot area coincides with the outline of the target key.

[0184] like Figure 12 、 Figure 14 and Figure 15 The attraction area shown may represent the depth (dp) of the finger pressed, for example, the depth (dp) may be 8.

[0185] In some embodiments, displaying a visual indication that the target button has sunk includes:

[0186] Following the moving speed and moving distance of the poking gesture, a visual indication of the visual indication component sinking is displayed until the visual indication component sinks to the end pushing layer and stops sinking.

[0187] When a user's finger interacts with a virtual key on the virtual keyboard, the system tracks the speed and distance of the poke gesture in real time. This tracking is based on the motion trajectory of the poke gesture's collision volume, accurately capturing the finger's path in space. To simulate the effect of pressing a physical key, the system displays a visual indicator component on the target key. This visual indicator dynamically adjusts based on the speed and distance of the poke gesture, creating a visual effect of sinking. This sinking effect gradually deepens as the poke gesture progresses until it reaches the desired depth.

[0188] When the visual indicator component sinks to the end push layer, the sinking action will stop, which can provide users with a more realistic and natural button pressing experience.

[0189] To enhance user perception and immersion, the visual indicator component's sinking effect simulates the real-world situation of pressing a physical button. Specifically, the visual indicator component sinks along the tip of the finger performing the poking gesture, so that the user feels like they are actually pressing a real button.

[0190] To maintain consistency and naturalness, the visual indicator's descending speed matches the speed of the poke gesture. This means that if the user quickly presses the virtual button, the visual indicator will descend at the same speed. Conversely, if the user's poke gesture is slow, the visual indicator will descend more slowly.

[0191] In addition to the sinking effect, the system also simulates the rebound effect of a physical button. This rebound effect is dynamically adjusted based on the location of the poke gesture and the user's action intention. For example, if the user cancels the operation or moves to another location after pressing the virtual button, the visual indicator component will produce a rebound effect to simulate the rebound action when the physical button is released. This rebound effect is also adjusted accordingly based on the location and speed of the poke gesture to provide a more realistic and natural interactive experience.

[0192] In some embodiments, the interactive feedback information further includes an auditory indication;

[0193] When displaying the visual indication that the target button is sinking, the method further includes:

[0194] The auditory indication is output, where the auditory indication is used to indicate that the target key is pressed.

[0195] To provide a more immersive experience when displaying the visual indication of a target key sinking, Android's built-in SoundPool class can be used to play a short sound effect as an auditory indicator when a key sinks. This sound effect simulates the sound of a virtual key being pressed, providing auditory feedback to the user. The SoundPool class is used in Android to play short sound effects. It is more efficient than the Media Player and is suitable for playing simple sound effects.

[0196] In some embodiments, the method further comprises:

[0197] Presenting a user interaction interface in the three-dimensional environment, wherein the user interaction interface displays an input box;

[0198] When it is detected that the interaction event is an exit selection state, character information corresponding to the target button is obtained, and input content corresponding to the character information is displayed in the input box.

[0199] Wherein, a user interaction interface is presented in a three-dimensional environment, and an input box is displayed on the user interaction interface.

[0200] For example, the user interface is displayed in a three-dimensional environment. This typically involves using specific virtual reality technologies, such as head-mounted displays (HMDs) and controllers, to create and display the three-dimensional environment. Within this three-dimensional environment, users can view computer-generated virtual images and receive visual feedback through devices such as the HMD. When displaying the user interface, it is necessary to ensure that it is presented clearly and accurately to the user within the three-dimensional environment. The user interface's position, content, layout, and color can be pre-configured to provide a user-friendly interactive experience. It is also necessary to ensure that the user interface interacts properly with other virtual objects (such as virtual objects and avatars) within the three-dimensional environment, allowing users to interact with the virtual environment through various operations. For example, the user interface can be a user interface corresponding to a client. This user interface can be customized based on the specific needs of the client to provide the functions and information required by the user. For example, multiple clients or applications can be presented in the three-dimensional environment. In response to a launch operation for a client, the user interface corresponding to the launched client can be presented within the three-dimensional environment, with an input box displayed on the user interface.

[0201] For example, an input box can be set up on the user interface. The input box is an important interface for users to interact with the system. Users can enter text, numbers, instructions, and other information through the input box. The input box must conform to the overall style and visual effects of the 3D environment.

[0202] To ensure that users can easily find and use the input box, you can use some guidance and markings on the user interface. For example, you can use obvious labels or icons to indicate the location of the input box, or add dynamic effects around the input box to attract users' attention.

[0203] For example, a virtual keyboard can be displayed in a three-dimensional environment in response to a first operation on an input box. For example, the first operation includes but is not limited to a somatosensory control operation, a gesture control operation, an eye movement operation, a touch operation, a voice control instruction, or an operation on an external control device. For example, a user can select an input box in the user interaction interface by triggering a preset button on an extended reality controller (such as a handle of a VR device), and select the input box by, for example, a virtual cursor to trigger an editing instruction for the input box, thereby waking up the virtual keyboard to display the virtual keyboard in the three-dimensional environment. The virtual keyboard is displayed with multiple virtual keys. For example, gesture control operation is also a common method. The user can make specific gestures in the three-dimensional environment, such as pinching, sliding, or tapping, to trigger the first operation on the input box. The system recognizes these gestures and then displays the virtual keyboard in the three-dimensional environment. For example, eye movement operation is also a novel human-computer interaction method. By using eye tracking technology, the system can detect the user's eye movement. When the user's gaze is focused on the input box, it can be regarded as the first operation on the input box, and then the virtual keyboard is displayed in the three-dimensional environment. For example, the extended reality device also supports touch functions. The user can directly click or touch the input box on the user interaction interface presented in the three-dimensional environment to trigger the first operation for the input box, and then display the virtual keyboard in the three-dimensional environment. For example, voice control instructions are also a convenient operation method. The user can select the input box or issue an editing instruction by voice, which can be regarded as the first operation for the input box, and then display the virtual keyboard in the three-dimensional environment. For example, for users who use external control devices, such as game controllers or remote controls, they can perform corresponding operations by connecting to the extended reality device, triggering the first operation for the input box, and then displaying the virtual keyboard in the three-dimensional environment.

[0204] When a user's interaction with the virtual keyboard is detected as "exiting selection," the system immediately retrieves the character information corresponding to the target key selected by the user. This character information may be letters, numbers, symbols, etc., depending on the virtual key selected by the user. After obtaining this character information, the system converts it into the corresponding input content and displays it in real time in the input box. This process allows users to intuitively see their input content and verify or modify it.

[0205] For example, when the user's finger leaves the target key, the system detects this action and begins processing the user's input. First, the system needs to determine the character the user pressed when they left the target key. This can be determined by identifying the target key where the user's finger last rested. For example, if the user presses the "A" key and then leaves, the system will recognize that the user entered the character "A". Once the character entered by the user is determined, the system needs to pass it to the input box. In the Android system, the input method manager (InputMethodManager) is used to manage the focus interaction of the input box. By calling methods of the Android InputMethodManager class, the system can pass the character entered by the user to the input box that currently has focus. In this way, the user's input can be correctly received and displayed in the corresponding input box.

[0206] The embodiments of the present application provide a virtual keyboard with a three-dimensional visual effect by converting the traditional 2D input method into a 3D one. By displaying the virtual keyboard in a three-dimensional environment, the user's interaction with the virtual keyboard is enhanced. The multiple virtual keys on the virtual keyboard allow the user to input information quickly and accurately, improving input efficiency. When the user interacts with the virtual keyboard, the interaction information is immediately fed back through visual indications, allowing the user to intuitively understand the effect of their operation, thereby improving the intuitiveness and naturalness of the interaction. By providing interactive feedback information, the embodiments of the present application compensate for the press feedback when contacting the virtual keyboard in XR space, and enhance the user's typing experience when inputting. Specifically, when the user's finger contacts the virtual keyboard, the relative spatial position of the current finger and the virtual keyboard and keyboard keys is identified. When the finger is above a key on the keyboard in XR space, the corresponding virtual key below will display a selected highlight effect. As the finger approaches the surface of the virtual key, the highlight effect will become more obvious, in the hope of helping the user's finger locate the target key on the virtual keyboard. When a finger touches the virtual key that is the target key, the state of the virtual key changes, completing the press action, and a press sound effect is generated to provide feedback to the user. At the same time, the input box affects the input behavior, and the user completes the input action.

[0207] To better implement the interactive method of the present application, please refer to Figure 16 The processing flow diagram shown.

[0208] The entire processing flow may include an initialization phase, a rendering phase, a gesture detection phase, and an input feedback phase.

[0209] The initialization phase may include steps S1 to S7:

[0210] S1. Initialization: Initialization is the first step of the product, that is, the new product startup stage.

[0211] S2. Activity initialization (Activity onCreate): In the lifecycle of an Android application, when a new activity is started, onCreate() is the first method to be executed. It is used to perform some basic initialization settings, such as creating the user interface. After executing step S2, you can execute step S3 or step D6.

[0212] S3. Calling the Open Graphics Library for Embedded Systems (OpenGL ES): OpenGL ES is a subset of the 3D graphics application programming interface OpenGL, designed for embedded devices such as mobile phones.

[0213] S4. Loading 3D models and textures: A 3D virtual keyboard is created. Each virtual key in the 3D virtual keyboard is an independent 3D model and has its own texture, material, and mesh.

[0214] S5. Set camera parameters and initial shader: When developing graphics-related applications, "setting camera parameters" and "initializing shader" are two important steps.

[0215] 1) Setting Camera Parameters: Camera parameters generally refer to the important factors that define the view matrix, such as the viewpoint position, observation point, and viewing angle. These parameters control the perspective from which the user sees the 3D world. For example, you can change the camera's position to simulate "movement" or change the observation point to "see" in different directions.

[0216] 2) Initializing Shaders: Shaders control specific stages in the rendering pipeline. For example, vertex shaders process vertex data, and pixel shaders process pixel colors. Initializing shaders involves creating and configuring these shaders.

[0217] S6. Start the input method manager (inputMethodManager): InputMethodManager is an Android system service responsible for managing keyboard input.

[0218] S7. Activity on Resume: This method is typically used in Android to indicate that an activity is now visible and that normal user interaction has resumed. Generally, after onResume() returns, the app will directly receive user input, such as keyboard taps.

[0219] The rendering stage may include steps S8 to S10:

[0220] S8. Rendering a 3D virtual keyboard: Rendering a 3D virtual keyboard using a graphics processing unit (GPU) and a shader program.

[0221] S9. Clear the cache:

[0222] gIClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT) is a method used to clear the buffer in the development graphics library (OpenGL). GL_COLOR_BUFFER_BIT and GL_DEPTH_BUFFER_BIT represent the color buffer and depth buffer respectively, preparing for the next rendering step.

[0223] S10. Render each virtual button based on the primitive drawing function (glDrawElements):

[0224] glDrawElements is a crucial function in OpenGL for rendering graphic elements such as points, lines, and triangles. Here, we're rendering all the virtual keys of a virtual keyboard. After defining the geometry and position of each virtual key, we can draw it by calling glDrawElements. In this step, OpenGL uses the previously set vertex and shader data.

[0225] The gesture detection stage may include steps S11 to S16:

[0226] S11. Calling the Hand Tracking API: The Hand Tracking API is a feature provided by the operating system (OS) to applications for identifying and tracking hand gestures and movements. In VR, applications use this API to monitor the user's fingers or entire hand to control the virtual keyboard.

[0227] S12. Gesture detection: The operation gesture of the current object can be recognized based on the gesture recognition library of the XR device.

[0228] S13. Determine whether the finger position is close to the key; if so, execute step S14; if not, return to execute step S12.

[0229] S14. Adjusting shader parameters: This is a common operation in 3D graphics programming. It can dynamically change the display of graphics, such as changing the color of an object. When the user's finger is over a certain button, the shader is used to change the color of the button to give the user feedback that the button has been selected.

[0230] S15. Displays a visual indication that the key is highlighted.

[0231] S16. Determine whether the finger presses the key; if so, execute step S17; if not, return to execute step D15.

[0232] The input feedback stage may include steps S17 to S20:

[0233] S17. Control the Z-axis of a view to change state and display a visual indication of a button being depressed: The z-axis refers to the height of the view on the screen. This visual indication gives the impression of a button being depressed.

[0234] S18. Output sound effect feedback based on the sound effect pool (SoundPool): The sound effect pool (SoundPool) is a class in Android, which is used to efficiently play short and large numbers of audio clips. In the embodiment of the present application, it can be used to play the sound effect feedback of key pressing.

[0235] S19. Get input behavior: InputMethodManager updateFullscreenMode()doStartinput() is a class in the Android system that manages the input of the system's virtual keyboard. Applications can use it to get user input behavior.

[0236] S20. Complete the input.

[0237] All of the above technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0238] The embodiment of the present application displays a three-dimensional environment generated by an extended reality device; displays a three-dimensional virtual keyboard in the three-dimensional environment, wherein a plurality of virtual keys are displayed on the virtual keyboard; recognizes a poke gesture of a current object; determines an interaction event between the current object and the virtual keyboard based on the poke gesture, the poke gesture collision body corresponding to the poke gesture, and the hover collision body corresponding to the virtual key in the virtual keyboard; and outputs interaction feedback information in response to the interaction event, wherein the interaction feedback information includes at least a visual indication. The embodiment of the present application provides a virtual keyboard with a three-dimensional visual effect. By displaying the three-dimensional virtual keyboard in the three-dimensional environment, the layout and keys of the keyboard are made more intuitive, thereby enhancing the convenience of user interaction with the virtual keyboard. The multiple virtual keys on the virtual keyboard enable users to input information quickly and accurately, thereby improving input efficiency. When a user interacts with the virtual keyboard, by recognizing the user's poke gesture, the user's intention can be understood, further optimizing the interaction with the virtual keyboard. By combining the poke gesture collision body corresponding to the poke gesture and the hover collision body corresponding to the virtual key in the virtual keyboard, the interaction event between the current object and the virtual keyboard can be accurately determined. The interaction information is immediately fed back in the form of a visual indication, allowing the user to intuitively understand the effect of their operation, thereby improving the intuitiveness and naturalness of the interaction.

[0239] In order to better implement the interactive method of the embodiment of the present application, the embodiment of the present application also provides an interactive device. Figure 17 , Figure 17 This is a schematic diagram of the structure of the interactive device provided in an embodiment of the present application. The interactive device 200 may include:

[0240] A first display unit 210 is used to display a three-dimensional environment generated by an extended reality device;

[0241] A second display unit 220 is configured to display a three-dimensional virtual keyboard in the three-dimensional environment, wherein the virtual keyboard has a plurality of virtual keys;

[0242] Recognition unit 230, for recognizing the poking gesture of the current object;

[0243] a determining unit 240 configured to determine an interaction event between the current object and the virtual keyboard based on the poke gesture, a poke gesture collision body corresponding to the poke gesture, and a hover collision body corresponding to the virtual key in the virtual keyboard;

[0244] The interaction unit 250 is configured to output interaction feedback information in response to an interaction event between the current object and the virtual keyboard, where the interaction feedback information at least includes a visual indication.

[0245] In some embodiments, the determination unit 240 is used to: construct a poke gesture collision body corresponding to the gesture interactor based on the poke gesture, and the position of the poke gesture collision body corresponds to the fingertip position of the finger that generates the poke gesture; for any virtual key on the virtual keyboard, construct a hover collision body corresponding to the virtual key, and the position of the hover collision body corresponds to the position of the virtual key; obtain the starting point and end point of the poke gesture trajectory based on the poke gesture; determine the interaction event between the current object and the virtual keyboard according to the starting point and end point of the poke gesture trajectory, and the poke gesture collision body and the hover collision body.

[0246] In some embodiments, when constructing the hover collision volume corresponding to the virtual key, the determining unit 240 is configured to:

[0247] Constructing a hover collision body corresponding to the virtual key in front of the virtual keyboard, the hover collision body including a start push layer, a selection exit layer, a selection entry layer, and an end push layer sequentially constructed along a direction perpendicular to the virtual key and the virtual keyboard;

[0248] The determining unit 240 is further configured to construct a selection cancellation layer behind the virtual keyboard.

[0249] In some embodiments, when determining the interaction event between the current object and the virtual keyboard based on the start and end points of the poke gesture trajectory and the poke gesture collision body and the hover collision body, the determining unit 240 may be configured to:

[0250] When a first poke gesture collision body corresponding to the starting point of the poke gesture trajectory is outside the hovering collision body, and a second poke gesture collision body corresponding to the end point of the poke gesture trajectory along the movement direction of the poke gesture trajectory enters the hovering collision body, and the second poke gesture collision body is between the starting push layer and the ending push layer, the interaction event is determined to be entering the hovering state;

[0251] When the first poke gesture collision body is in a hovering state and the second poke gesture collision body leaves the hovering collision body along the movement direction of the poke gesture trajectory, determining that the interaction event is exiting the hovering state;

[0252] When the first poke gesture collision body is in a hovering state and the second poke gesture collision body enters the area between the selection entry layer and the end push layer along the movement direction of the poke gesture trajectory, the interaction event is determined to be entering the selection state;

[0253] When the first poke gesture collision body is in the selection state and the second poke gesture collision body leaves the area between the selection exit layer and the end push layer along the movement direction of the poke gesture trajectory, the interaction event is determined to be the exit selection state;

[0254] When the first poke gesture collision body is in the selected state and the second poke gesture collision body passes through the selection cancellation layer in the vertical direction along the movement direction of the poke gesture trajectory and is located outside the selection cancellation layer, the interaction event is determined to be in the deselected state.

[0255] In some embodiments, when determining that the interaction event is entering the selection state, the determination unit 240 can be used to: when the poke gesture trajectory passes through the selection entry layer from the front of the hovering collision body, when the first poke gesture collision body is located before the selection entry layer and the second poke gesture collision body is located after the selection entry layer, determine that the interaction event is entering the selection state; when the poke gesture trajectory enters the selection entry layer from the side of the hovering collision body, when the second poke gesture collision body enters the hovering collision body between the selection entry layer and the end push layer, determine that the interaction event is entering the selection state.

[0256] In some embodiments, when the interaction unit 250 outputs the interaction feedback information in response to the interaction event, it can be used to:

[0257] When the interaction event is entering a hover state, identifying a target button and displaying a visual indication of a light effect on the target button, the target button being a virtual button with the poke gesture collision body hovering above it;

[0258] When the interaction event is exiting the hover state, canceling the visual indication of the light effect of the target button;

[0259] When the interaction event is entering a selection state, displaying a visual indication that the target button is sinking; and

[0260] When the interaction event is exiting the selection state or canceling the selection state, a visual indication is displayed that the target button is restored to the initial position.

[0261] In some embodiments, when the interaction unit 250 displays a visual indication of a bright effect on the target key, it can be used to: display a halo effect on the virtual keyboard when the distance between the poke gesture collision body and the virtual keyboard is a first preset distance; when the poke gesture collision body gradually approaches the virtual keyboard and the distance between the poke gesture collision body and the target key is a third preset distance, display a visual indication component with a highlight effect in front of the target key based on a fourth preset distance.

[0262] In some embodiments, when the interaction unit 250 displays a visual indication component with a highlight effect in front of the target key based on a fourth preset distance, it can be used to: when the poke gesture collision body gradually approaches the virtual keyboard, display a visual indication component with a highlight degree gradually changing from small to large in front of the target key based on the fourth preset distance.

[0263] In some embodiments, when displaying the visual indication of the target button sinking, the interaction unit 250 can be used to: follow the moving speed and moving distance of the poke gesture, display the visual indication of the sinking of the visual indication component until the visual indication component sinks to the end push layer and stops sinking.

[0264] In some embodiments, when the interaction unit 250 displays the visual indication of the light effect on the target key, it can be used to: display the visual indication of the light effect on the target key by changing the parameters of the shader corresponding to the target key.

[0265] In some embodiments, the interactive feedback information further includes an auditory indication;

[0266] When displaying the visual indication that the target button is sinking, the interaction unit 250 may further be used to:

[0267] The auditory indication is output, where the auditory indication is used to indicate that the target key is pressed.

[0268] In some embodiments, the interaction unit 250 can also be used to: present a user interaction interface in the three-dimensional environment, and an input box is displayed on the user interaction interface; when it is detected that the interaction event is an exit selection state, the character information corresponding to the target button is obtained, and the input content corresponding to the character information is displayed in the input box.

[0269] Each unit in the interactive device 200 may be implemented in whole or in part by software, hardware, or a combination thereof. Each unit may be embedded in or independent of a processor in a terminal device in the form of hardware, or may be stored in a memory in the terminal device in the form of software, so that the processor can call and execute the corresponding operations of each unit.

[0270] The interactive device 200 may be integrated into a terminal or server that has a storage device and a processor and has computing capabilities, or the interactive device 200 may be the terminal or server.

[0271] In some embodiments, the present application also provides a terminal device including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0272] like Figure 18 As shown, Figure 18 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. The terminal device 300 can generally be provided in the form of glasses, a head-mounted display (HMD), or contact lenses to achieve visual perception and other forms of perception. Of course, the form of the terminal device is not limited to this and can be further miniaturized or enlarged as needed. The terminal device 300 may include but is not limited to the following components:

[0273] Detection module 301: Uses various sensors to detect the user's operation commands and acts on the virtual environment, such as continuously updating the image displayed on the display screen following the user's line of sight, realizing the user's interaction with the virtual and scene, for example, continuously updating the real content based on the detected direction of the user's head rotation.

[0274] Feedback module 302: Receives data from sensors and provides real-time feedback to the user. The feedback module 302 may be configured to display a graphical user interface, such as a virtual environment, and may include a display screen.

[0275] Sensor 303: On the one hand, it receives operation commands from the user and applies them to the virtual environment; on the other hand, it provides the results generated after the operation to the user in the form of various feedback.

[0276] Control module 304: controls sensors and various input / output devices, including obtaining user data (such as movements and voice) and outputting perception data, such as images, vibrations, temperature, and sounds, etc., to affect the user, the virtual environment, and the real world.

[0277] Modeling module 305: constructs a three-dimensional model of the virtual environment, and may also include various feedback mechanisms such as sound and touch in the three-dimensional model.

[0278] In an embodiment of the present application, a virtual scene in a three-dimensional environment can be constructed by the modeling module 305; the three-dimensional environment generated by the extended reality device can be displayed by the feedback module 302, and a three-dimensional virtual keyboard can be displayed in the three-dimensional environment, where multiple virtual keys are displayed on the virtual keyboard; the poke gesture of the current object can be identified by the detection module 301 and / or the sensor 303; the interaction event between the current object and the virtual keyboard can be determined by the control module 304 based on the poke gesture, the poke gesture collision body corresponding to the poke gesture, and the hover collision body corresponding to the virtual key in the virtual keyboard; and the feedback module 302 can output interaction feedback information in response to the interaction event, where the interaction feedback information includes at least a visual indication.

[0279] In some embodiments, as Figure 19 As shown, Figure 19 This is another schematic diagram of the structure of a terminal device provided in an embodiment of the present application. The terminal device 300 also includes a processor 310 having one or more processing cores, a memory 320 having one or more computer-readable storage media, and a computer program stored in the memory 320 and executable on the processor. The processor 310 is electrically connected to the memory 320. Those skilled in the art will appreciate that the terminal device structure shown in the figure does not limit the terminal device, and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0280] The processor 310 is the control center of the terminal device 300. It uses various interfaces and lines to connect various parts of the entire terminal device 300. By running or loading software programs and / or modules stored in the memory 320, and calling data stored in the memory 320, it executes various functions of the terminal device 300 and processes data, thereby monitoring the terminal device 300 as a whole.

[0281] In the embodiment of the present application, the processor 310 in the terminal device 300 loads instructions corresponding to one or more application processes into the memory 320 according to the following steps, and the processor 310 runs the application stored in the memory 320 to implement various functions:

[0282] Displaying a three-dimensional environment generated by an extended reality device; displaying a three-dimensional virtual keyboard in the three-dimensional environment, wherein a plurality of virtual keys are displayed on the virtual keyboard; recognizing a poke gesture of a current object; determining an interaction event between the current object and the virtual keyboard based on the poke gesture, a poke gesture collision body corresponding to the poke gesture, and a hover collision body corresponding to the virtual key in the virtual keyboard; and outputting interaction feedback information in response to the interaction event, wherein the interaction feedback information includes at least a visual indication.

[0283] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0284] In some embodiments, the processor 310 may include a detection module 301 , a control module 304 , and a modeling module 305 .

[0285] In some embodiments, as Figure 19 As shown, the terminal device 300 further includes: a radio frequency circuit 306, an audio circuit 307, and a power supply 308. Among them, the processor 310 is electrically connected to the memory 320, the feedback module 302, the sensor 303, the radio frequency circuit 306, the audio circuit 307, and the power supply 308 respectively. Those skilled in the art will understand that Figure 18 or Figure 19 The terminal device structure shown in the figure does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0286] The radio frequency circuit 306 may be used to transmit and receive radio frequency signals, so as to establish wireless communication with a network device or other terminal devices through wireless communication, and to transmit and receive signals with the network device or other terminal devices.

[0287] The audio circuit 307 can be used to provide an audio interface between the user and the terminal device through a speaker and microphone. The audio circuit 307 can convert received audio data into electrical signals and transmit them to the speaker, which then converts them into sound signals for output. The microphone, on the other hand, converts collected sound signals into electrical signals, which are then received by the audio circuit 307 and converted into audio data. The audio data is then processed by the output processor 310 and transmitted via the RF circuit 306 to, for example, another terminal device, or the audio data is output to a memory for further processing. The audio circuit 307 may also include an earphone jack to provide communication between an external headset and the terminal device.

[0288] The power supply 308 is used to supply power to various components of the terminal device 300 .

[0289] although Figure 18 or Figure 19 Not shown in the figure, the terminal device 300 may further include a camera, a wireless fidelity module, a Bluetooth module, an input module, etc., which will not be described in detail here.

[0290] In some embodiments, the present application further provides a computer-readable storage medium for storing a computer program. The computer-readable storage medium can be applied to a terminal device or a server, and the computer program causes the terminal device or server to execute the corresponding process of the interaction method in the embodiments of the present application. For the sake of brevity, it is not further described here.

[0291] In some embodiments, the present application further provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a terminal device reads the computer program from the computer-readable storage medium and executes the computer program, causing the terminal device to perform the corresponding process of the interaction method in the embodiments of the present application. For the sake of brevity, the details are not repeated here.

[0292] The present application also provides a computer program, which includes a computer program stored in a computer-readable storage medium. A processor of a terminal device reads the computer program from the computer-readable storage medium and executes the computer program, causing the terminal device to perform the corresponding process of the interaction method in the embodiment of the present application. For the sake of brevity, the details are not repeated here.

[0293] It should be understood that the processor of the embodiments of the present application may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by hardware integrated logic circuits in the processor or software instructions. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly implemented as a hardware decoding processor, or can be implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0294] It is understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0295] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0296] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0297] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0298] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0299] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of this embodiment.

[0300] In addition, each functional unit in the embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a terminal device (which can be a personal computer, a server) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk.

[0301] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An interactive method, characterized in that: The method comprises: Displaying a three-dimensional environment generated by an extended reality device; Displaying a three-dimensional virtual keyboard in the three-dimensional environment, wherein a plurality of virtual keys are displayed on the virtual keyboard; Recognize the poke gesture of the current object; determining, according to the poke gesture, a poke gesture collision body corresponding to the poke gesture, and a hover collision body corresponding to the virtual key in the virtual keyboard, an interaction event between the current object and the virtual keyboard; In response to the interaction event, interaction feedback information is output, where the interaction feedback information at least includes a visual indication.

2. The interactive method according to claim 1, wherein: The determining, based on the poke gesture, the poke gesture collision body corresponding to the poke gesture, and the hover collision body corresponding to the virtual key in the virtual keyboard, an interaction event between the current object and the virtual keyboard includes: Constructing a poke gesture collision body corresponding to a gesture interactor based on the poke gesture, wherein a position of the poke gesture collision body corresponds to a fingertip position of a finger generating the poke gesture; For any virtual key on the virtual keyboard, construct a hover collision body corresponding to the virtual key, where the position of the hover collision body corresponds to the position of the virtual key; Obtaining a start point and an end point of a poking gesture trajectory based on the poking gesture; An interaction event between the current object and the virtual keyboard is determined according to a starting point and an end point of the poke gesture trajectory, and the poke gesture collision body and the hovering collision body.

3. The interactive method according to claim 2, wherein: The constructing of the hover collision body corresponding to the virtual button includes: Constructing a hover collision body corresponding to the virtual key in front of the virtual keyboard, the hover collision body including a start push layer, a selection exit layer, a selection entry layer, and an end push layer sequentially constructed along a direction perpendicular to the virtual key and the virtual keyboard; The method further comprises: A selection cancellation layer is constructed behind the virtual keyboard.

4. The interactive method according to claim 3, wherein: The determining, based on the starting point and the end point of the poke gesture trajectory, and the poke gesture collision body and the hovering collision body, of an interaction event between the current object and the virtual keyboard includes: When a first poke gesture collision body corresponding to the starting point of the poke gesture trajectory is outside the hovering collision body, and a second poke gesture collision body corresponding to the end point of the poke gesture trajectory along the movement direction of the poke gesture trajectory enters the hovering collision body, and the second poke gesture collision body is between the starting push layer and the ending push layer, the interaction event is determined to be entering the hovering state; When the first poke gesture collision body is in a hovering state and the second poke gesture collision body leaves the hovering collision body along the movement direction of the poke gesture trajectory, determining that the interaction event is exiting the hovering state; When the first poke gesture collision body is in a hovering state and the second poke gesture collision body enters the area between the selection entry layer and the end push layer along the movement direction of the poke gesture trajectory, the interaction event is determined to be entering the selection state; When the first poke gesture collision body is in the selection state and the second poke gesture collision body leaves the area between the selection exit layer and the end push layer along the movement direction of the poke gesture trajectory, the interaction event is determined to be the exit selection state; When the first poke gesture collision body is in the selected state and the second poke gesture collision body passes through the selection cancellation layer in the vertical direction along the movement direction of the poke gesture trajectory and is located outside the selection cancellation layer, the interaction event is determined to be in the deselected state.

5. The interactive method according to claim 4, wherein: When the first poke gesture collision body is in a hovering state and the second poke gesture collision body enters an area between the selection entry layer and the end push layer along the movement direction of the poke gesture trajectory, determining that the interaction event is entering the selection state includes: In a case where the poke gesture trajectory passes through the selection entry layer from the front of the hovering collision body, when the first poke gesture collision body is located before the selection entry layer and the second poke gesture collision body is located after the selection entry layer, determining that the interaction event is an entry selection state; In the case where the poke gesture trajectory enters the selection entry layer from a side of the hovering collision body, when the second poke gesture collision body enters the hovering collision body from between the selection entry layer and the end push layer, the interaction event is determined to be entering the selection state.

6. The interactive method according to any one of claims 3 to 5, characterized in that: The outputting interaction feedback information in response to the interaction event includes: When the interaction event is entering a hover state, identifying a target button and displaying a visual indication of a light effect on the target button, the target button being a virtual button with the poke gesture collision body hovering above it; When the interaction event is exiting the hover state, canceling the visual indication of the light effect of the target button; When the interaction event is entering a selection state, displaying a visual indication that the target button is sinking; and When the interaction event is exiting the selection state or canceling the selection state, a visual indication is displayed that the target button is restored to the initial position.

7. The interactive method according to claim 6, wherein: The visual indication of displaying a light effect on the target key includes: When the distance between the poke gesture collision body and the virtual keyboard is a first preset distance, displaying a halo effect on the virtual keyboard; When the poke gesture collision body gradually approaches the virtual keyboard and the distance between the poke gesture collision body and the target key is a third preset distance, a visual indication component with a highlight effect is displayed in front of the target key based on a fourth preset distance.

8. The interactive method according to claim 7, wherein: The visual indication component that displays a highlight effect in front of the target key based on the fourth preset distance includes: When the poke gesture collision body gradually approaches the virtual keyboard, a visual indication component with a highlight level gradually changing from small to large is displayed in front of the target key based on a fourth preset distance.

9. The interactive method according to claim 8, wherein: The visual indication of displaying the target key sinking includes: Following the moving speed and moving distance of the poking gesture, a visual indication of the visual indication component sinking is displayed until the visual indication component sinks to the end pushing layer and stops sinking.

10. The interactive method according to claim 6, wherein: The visual indication of displaying a light effect on the target key includes: By changing the parameters of the shader corresponding to the target button, a visual indication of the light effect is displayed on the target button.

11. The interactive method according to claim 6, wherein: The interactive feedback information also includes auditory indications; When displaying the visual indication that the target button is sinking, the method further includes: The auditory indication is output, where the auditory indication is used to indicate that the target key is pressed.

12. The interactive method according to claim 6, wherein: The method further comprises: Presenting a user interaction interface in the three-dimensional environment, wherein the user interaction interface displays an input box; When it is detected that the interaction event is an exit selection state, character information corresponding to the target button is obtained, and input content corresponding to the character information is displayed in the input box.

13. An interactive device, characterized in that: The device comprises: A first display unit, configured to display a three-dimensional environment generated by an extended reality device; a second display unit, configured to display a three-dimensional virtual keyboard in the three-dimensional environment, wherein the virtual keyboard displays a plurality of virtual keys; A recognition unit, used to recognize the poking gesture of the current object; a determining unit, configured to determine an interaction event between the current object and the virtual keyboard according to the poke gesture, a poke gesture collision body corresponding to the poke gesture, and a hover collision body corresponding to the virtual key in the virtual keyboard; The interaction unit is configured to output interaction feedback information in response to the interaction event, where the interaction feedback information at least includes a visual indication.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the interaction method according to any one of claims 1 to 12.

15. A terminal device, characterized in that: The terminal device includes a processor and a memory, wherein a computer program is stored in the memory, and the processor is configured to execute the interaction method according to any one of claims 1 to 12 by calling the computer program stored in the memory.

Images