Interaction method and device, storage medium and equipment

Abstract

The invention discloses an interaction method and device, a storage medium and equipment, and the method comprises the steps: presenting a user interaction interface in a three-dimensional environment, and displaying an input box on the user interaction interface; in response to a first operation for the input box, displaying a virtual keyboard in the three-dimensional environment; displaying a virtual screen in a region adjacent to the display position of the virtual keyboard, wherein an extraction frame is displayed on the virtual screen; and in response to a second operation for the virtual keyboard, synchronously displaying the corresponding input content in the extraction box and the input box. According to the method and the device, the interaction convenience and the input efficiency on the augmented reality equipment can be improved, so that smoother and more natural input experience is provided for a user.

Description

Technical Field

[0001] This application relates to the technical field of Extended Reality (XR), and particularly to an interaction method, device, storage medium, and equipment. Background Art

[0002] Currently, the application scenarios of XR technology are becoming increasingly widespread, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), etc.

[0003] In recent years, with the popularization of mobile devices and the rapid development of technology, two-dimensional (2D) input methods have become standard on mobile devices such as mobile phones, tablets, and laptop computers. As the main means of user interaction with these devices, their role has become increasingly crucial. In the applications of emerging extended reality devices, the performance of the input method still remains at the mobile phone era level, resulting in inconvenient interaction and low efficiency. Summary of the Invention

[0004] Embodiments of this application provide an interaction method, device, storage medium, and equipment, which can improve the interaction convenience and input efficiency on extended reality devices, thereby providing a more fluent and natural input experience for users.

[0005] On the one hand, embodiments of this application provide an interaction method, and the method includes:

[0006] Display a three-dimensional environment generated by an extended reality device;

[0007] Present a user interaction interface within the three-dimensional environment, and an input box is displayed on the user interaction interface;

[0008] In response to a first operation on the input box, display a virtual keyboard in the three-dimensional environment;

[0009] Display a virtual screen in an adjacent area of the display position of the virtual keyboard, and an extraction box is displayed on the virtual screen;

[0010] In response to a second operation on the virtual keyboard, synchronously display corresponding input content in the extraction box and the input box.

[0011] On the other hand, embodiments of this application provide an interaction device, and the device includes:

[0012] A first display unit, configured to display a three-dimensional environment generated by an extended reality device;

[0013] A second display unit, configured to present a user interaction interface within the three-dimensional environment, where an input box is displayed on the user interaction interface;

[0014] A third display unit, configured to display a virtual keyboard in the three-dimensional environment in response to a first operation on the input box;

[0015] A fourth display unit, configured to display a virtual screen in an adjacent area of the display position of the virtual keyboard, where an extraction box is displayed on the virtual screen;

[0016] An interaction unit, configured to synchronously display corresponding input content in the extraction box and the input box in response to a second operation on the virtual keyboard.

[0017] On the other hand, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and the computer program is adapted to be loaded by a processor to execute the interaction method described in any one of the foregoing embodiments.

[0018] On the other hand, an embodiment of the present application provides a terminal device, where the terminal device includes a processor and a memory, and a computer program is stored in the memory. The processor is configured to execute the interaction method described in any one of the foregoing embodiments by calling the computer program stored in the memory.

[0019] In the embodiment of the present application, a three-dimensional environment generated by a display extended reality device is displayed; a user interaction interface is presented within the three-dimensional environment, and an input box is displayed on the user interaction interface; in response to a first operation on the input box, a virtual keyboard is displayed in the three-dimensional environment; a virtual screen is displayed in an adjacent area of the display position of the virtual keyboard, and an extraction box is displayed on the virtual screen; in response to a second operation on the virtual keyboard, corresponding input content is synchronously displayed in the extraction box and the input box. In the embodiment of the present application, a virtual screen with a prompt box is displayed in an adjacent area of the display position of the virtual keyboard, and corresponding input content is synchronously displayed in the extraction box and the input box in response to a related operation on the virtual keyboard, which is equivalent to mapping the input box to an extraction box in the same plane area as the virtual keyboard, so as to convert the input behavior of the user in the three-dimensional environment into an input behavior in the plane space, thereby optimizing the relative positions of the input box and the virtual keyboard, so that when the user lowers the head to operate the virtual keyboard, the specific input content can more conveniently appear within the user's line of sight, so as to achieve the effect of not needing to frequently raise the head to confirm during input, which can improve the interaction convenience and input efficiency on the extended reality device, thereby providing a more fluent and natural input experience for the user. Description of the Drawings

[0020] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those skilled in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0021] Figure 1 It is a schematic flowchart of the interaction method provided by the embodiment of the present application.

[0022] Figure 2 It is a schematic diagram of the first application scenario of the interaction method provided by the embodiment of the present application.

[0023] Figure 3 It is a schematic diagram of the second application scenario of the interaction method provided by the embodiment of the present application.

[0024] Figure 4 It is a schematic diagram of the third application scenario of the interaction method provided by the embodiment of the present application.

[0025] Figure 5 It is a schematic diagram of the fourth application scenario of the interaction method provided by the embodiment of the present application.

[0026] Figure 6 It is a schematic diagram of the fifth application scenario of the interaction method provided by the embodiment of the present application.

[0027] Figure 7 It is a schematic structural diagram of the interaction device provided by the embodiment of the present application.

[0028] Figure 8 It is the first schematic structural diagram of the terminal device provided by the embodiment of the present application.

[0029] Figure 9 It is the second schematic structural diagram of the terminal device provided by the embodiment of the present application. Specific embodiments

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of protection of the present application.

[0031] The embodiments of the present application can be applied to various application scenarios such as Extended Reality (XR), Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).

[0032] First, some nouns or terms that appear in the process of describing the embodiments are explained as follows:

[0033] A virtual scene is a virtual scene displayed (or provided) when an application runs on a terminal or a server. Optionally, the virtual scene is a simulation environment of the real world, or a semi-simulated and semi-fictional virtual environment, or a purely fictional virtual environment. The virtual scene is either a two-dimensional virtual scene or a three-dimensional virtual scene. The virtual environment can be the sky, land, ocean, etc., where the land includes environmental elements such as deserts and cities. Among them, the virtual scene is a scene for the complete game logic of virtual objects such as user control.

[0034] A virtual object refers to a dynamic object that can be controlled in a virtual scene. Optionally, the dynamic object can be a virtual character, a virtual animal, an anime character, etc. The virtual object is a character controlled by a player through an input device, or an artificial intelligence (AI) set in a virtual environment battle through training, or a non-player character (NPC) set in a virtual scene battle. Optionally, the virtual object is a virtual character competing in a virtual scene. Optionally, the number of virtual objects in the virtual scene battle is preset, or dynamically determined according to the number of clients joining the battle. This application embodiment does not limit this. In a possible implementation manner, a user can control the virtual object to move in the virtual scene. For example, control the virtual object to run, jump, crawl, etc., and can also control the virtual object to use skills, virtual props, etc. provided by the application to fight with other virtual objects. Optionally, the virtual object can also refer to a static object that can be interacted with in a virtual scene, such as a virtual object, a virtual control, an interface element, a virtual prop, etc.

[0035] Extended Reality (XR) is a concept that includes Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), representing a technology that connects the virtual world and the real world, enabling users to interact with this environment in real time.

[0036] Virtual Reality (VR) is a technology for creating and experiencing virtual worlds. It computationally generates a virtual environment, which is a multi-source information (the virtual reality mentioned in this article includes at least visual perception, and may also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.). It realizes the integration of a virtual environment, an interactive three-dimensional dynamic visual scene, and the simulation of entity behavior, enabling users to immerse themselves in the simulated three-dimensional environment and realizing applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisting manufacturing, maintenance, and repair.

[0037] Augmented Reality (AR) is a technology that, during the process of a camera capturing an image, calculates the camera pose parameters of the camera in the real world (or three-dimensional world, real world) in real time and adds virtual elements to the image captured by the camera. Virtual elements include, but are not limited to: images, videos, and 3D models. The goal of AR technology is to interact by overlaying the virtual world on the real world on the screen.

[0038] Mixed Reality (MR) is a simulated scene that integrates computer-generated sensory inputs (such as virtual objects) with sensory inputs or their representations from a physical setting. In some MR settings, the computer-generated sensory inputs can adapt to changes in the sensory inputs from the physical setting. Additionally, some electronic systems for presenting MR settings can monitor the orientation and / or position relative to the physical setting so that virtual objects can interact with real objects (i.e., physical elements or their representations from the physical setting). For example, the system can monitor motion so that a virtual plant appears stationary relative to a physical building.

[0039] Augmented Virtuality (AV): An AV setting refers to a simulated scene in which a computer-generated setting or virtual setting incorporates at least one sensory input from a physical setting. One or more sensory inputs from the physical setting can be representations of at least one feature of the physical setting. For example, a virtual object can present the color of a physical element captured by one or more imaging sensors. Another example is that a virtual object can present features consistent with the actual weather conditions in the physical setting, such as those identified via weather-related imaging sensors and / or online weather data. In another example, an augmented reality forest can have virtual trees and structures, but animals can have features accurately reproduced from images of physical animals.

[0040] Virtual field of view refers to the area in the virtual environment that a user can perceive through a lens in an extended reality device, and the field of view angle (Field Of View, FOV) of the virtual field of view is used to represent the perceived area.

[0041] An extended reality device, a terminal for realizing extended reality effects, can usually be provided in the form of glasses, a head-mounted display (HMD), contact lenses, etc., for realizing visual perception and other forms of perception. Of course, the form implemented by the extended reality device is not limited to this, and it can be further miniaturized or enlarged according to needs.

[0042] The extended reality devices described in the embodiments of the present application may include but are not limited to the following types:

[0043] A PC-based virtual reality (PCVR) device uses the PC to perform relevant calculations and data output for virtual reality functions. The externally connected PC-based extended reality device uses the data output by the PC to achieve virtual reality effects.

[0044] A mobile extended reality device supports setting a mobile terminal (such as a smartphone) in various ways (such as a head-mounted display with a dedicated card slot). Through a wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations for virtual reality functions and outputs data to the mobile extended reality device. For example, watch virtual reality videos through the APP of the mobile terminal.

[0045] An all-in-one extended reality device has a processor for performing relevant calculations for virtual functions, and thus has independent virtual reality input and output functions, without the need to be connected to a PC or a mobile terminal, and has a high degree of freedom of use.

[0046] In recent years, with the popularization of mobile devices and the rapid development of technology, two-dimensional (2D) input methods have become standard on mobile devices such as mobile phones, tablets, and laptops. As the main means for users to interact with these devices, their role has become increasingly crucial. In the applications of emerging extended reality devices, the performance of input methods still remains at the mobile phone era, resulting in inconvenient interaction and low efficiency.

[0047] The differences in input behaviors between traditional mobile devices and extended reality devices are mainly reflected in the following two major aspects:

[0048] 1. The relative position difference between the keyboard layer and the input box: On traditional mobile devices such as mobile phones, the keyboard and the input box are on the same plane. When the user performs an input operation, they only need to lower their head to tap the keyboard, and then they can directly see the content change in the input box. However, in the three-dimensional environment presented by extended reality devices, the keyboard and the input box are usually not on the same plane. The user needs to lower their head to operate the keyboard and then raise their head to view the content in the input box. This frequent head movement not only increases the user's operation burden, but also greatly reduces the input efficiency and affects the user's comfort at the same time.

[0049] 2. Difficulty of operating on the content in the input box: For mobile phone users, basic operations such as adding, deleting, or modifying the text, pinyin, etc. in the input box are easy. However, in the three-dimensional environment presented by the extended reality device, due to the increased distance between the input box and the user, these seemingly simple operations become extremely difficult. Users need to adopt additional interaction methods to select the input box and make modifications, resulting in great difficulty for users when processing the content in the input box. This undoubtedly increases the complexity of the operation and further affects the user's input experience.

[0050] In summary, the commonly used 2D input method on traditional mobile devices has obvious problems of inconvenient interaction and low efficiency on extended reality devices.

[0051] Therefore, the embodiments of this application propose an interaction method, which can improve the interaction convenience and input efficiency on extended reality devices, thereby providing a more fluent and natural input experience for users.

[0052] The following will be described in detail respectively. It should be noted that the description order of the following embodiments does not limit the priority order of the embodiments.

[0053] Each embodiment of this application provides an interaction method, which can be executed by a terminal or a server, or jointly executed by a terminal and a server; the embodiments of this application take the interaction method being executed by a terminal device as an example for illustration.

[0054] Please refer to Figures 1 to 6 , Figure 1 which is a schematic flowchart of the interaction method provided by the embodiments of this application, Figures 2 to 6 and

[0055] Step 110, display the three-dimensional environment generated by the extended reality device.

[0056] First, through the technology of the extended reality device, a realistic three-dimensional environment is constructed. This environment provides an immersive experience for users, making them feel as if they are in a real and virtual world. Among them, the three-dimensional environment generated by the extended reality device not only includes the real space environment, but can also be a virtual space environment. The three-dimensional environment can be a virtual reality environment or an extended reality environment.

[0057] First, the extended reality device can possess powerful graphics processing capabilities to generate high-quality three-dimensional environments in real time. This typically involves using advanced graphics rendering techniques such as ray tracing, shadow mapping, texture mapping, etc., to create realistic three-dimensional scenes.

[0058] To provide an immersive experience, the extended reality device can also possess depth perception and stereoscopic imaging capabilities. This can be achieved by using technologies such as infrared sensors, depth cameras, or lidar, which can capture detailed information about the surrounding environment and convert it into a three-dimensional model.

[0059] When generating the three-dimensional environment, the movements of the user's head and eyes can also be considered. The extended reality device can possess high-precision motion tracking capabilities, capable of capturing in real time the user's head pose, eye movements, and gestures, etc., and adjusting the perspective and details of the three-dimensional environment according to these inputs to provide a natural immersive experience.

[0060] To enhance the user's interaction experience, the three-dimensional environment can also be integrated with the user interface. The user interface can be customized according to the requirements of the application, providing various functions and controls such as menus, buttons, input boxes, etc. These interface elements can be presented in a realistic manner in the three-dimensional environment, enabling the user to interact intuitively.

[0061] Step 120, present a user interface within the three-dimensional environment, and an input box is displayed on the user interface.

[0062] Among them, the display of the user interface is carried out in the three-dimensional environment. This typically involves using specific virtual reality technologies such as head-mounted displays and controllers, etc., to create and display the three-dimensional environment. In this three-dimensional environment, the user can see the virtual images generated by the computer and obtain visual feedback through devices such as head-mounted displays. When displaying the user interface, it is necessary to ensure that the user interface can be clearly and accurately presented to the user in the three-dimensional environment. The interface position, interface content, layout, color, etc. of the user interface can be preset to provide a user-friendly interaction experience. At the same time, it is also necessary to ensure that the interaction between the user interface and other virtual objects (such as virtual objects, virtual characters, etc.) in the three-dimensional environment can work properly, so that the user can interact with the virtual environment through various operations.

[0063] For example, the user interaction interface can be the user interaction interface corresponding to the client. This user interaction interface can be customized according to the specific requirements of the client to provide the functions and information required by the user. For example, multiple clients or applications can be presented in a three-dimensional environment. In response to a startup operation for a certain client, the user interaction interface corresponding to the started client can be presented within the three-dimensional environment, and an input box is also displayed on the user interaction interface.

[0064] Among them, on the user interaction interface, an input box can be set. The input box is an important interface for the user to interact with the system. The user can input information such as text, numbers, and instructions through the input box. The input box needs to conform to the overall style and visual effect of the three-dimensional environment.

[0065] To ensure that the user can easily find and use the input box, some guidance and markings can be adopted on the user interaction interface. For example, obvious labels or icons can be used to identify the position of the input box, or dynamic effects can be added around the input box to attract the user's attention.

[0066] As Figure 2 shown, the user interaction interface 10 is presented within the three-dimensional environment 1, and an input box 11 is displayed on the user interaction interface 10. For example, the user interaction interface 10 can be the user interaction interface corresponding to a video playback client.

[0067] Step 130, in response to a first operation on the input box, display a virtual keyboard in the three-dimensional environment.

[0068] For example, the first operation includes but is not limited to somatosensory control operations, gesture control operations, eye movement operations, touch operations, voice control instructions, or operations on external control devices. For example, the user can select the input box in the user interaction interface by triggering a preset button on an extended reality controller (such as the handle of a VR device), and select the input box through a virtual cursor, etc., to trigger an edit instruction for the input box, thereby waking up the virtual keyboard to display the virtual keyboard in the three-dimensional environment. Among them, multiple virtual keys are displayed on the virtual keyboard.

[0069] For example, gesture control operation is also a common method. The user can make specific gestures in the three-dimensional environment, such as pinching, swiping, or tapping, etc., to trigger the first operation on the input box. The system recognizes these gestures and then displays the virtual keyboard in the three-dimensional environment.

[0070] For example, eye movement operation is also a novel human-computer interaction method. By using eye tracking technology, the system can detect the user's eye movement. When the user's line of sight focuses on the input box, it can be regarded as the first operation on the input box, and then the virtual keyboard is displayed in the three-dimensional environment.

[0071] For example, the extended reality device also supports a touch function. The user can directly click or touch an input box on the user interaction interface presented in the three-dimensional environment to trigger a first operation on the input box, and then display a virtual keyboard in the three-dimensional environment.

[0072] For example, a voice control instruction is also a convenient operation method. The user can use voice to select an input box or issue an editing instruction, which can be regarded as a first operation on the input box, and then display a virtual keyboard in the three-dimensional environment.

[0073] For example, for users using an external control device, such as a gamepad or a remote control, etc., they can perform corresponding operations through the connection with the extended reality device to trigger a first operation on the input box, and then display a virtual keyboard in the three-dimensional environment.

[0074] As Figure 3 shown, a user interaction interface 10 is presented within a three-dimensional environment 1, and an input box 11 is displayed on the user interaction interface 10. For example, the user interaction interface 10 can be a user interaction interface corresponding to a video playback client. In response to a first operation on the input box 11, such as selecting the input box 11 through a virtual cursor 40 to trigger an editing instruction for the input box 11, and then waking up a virtual keyboard 20 to display the virtual keyboard 20 in the three-dimensional environment 1.

[0075] Step 140, display a virtual screen in an adjacent area of the display position of the virtual keyboard, and an extraction box is displayed on the virtual screen.

[0076] Among them, the display position of the virtual screen should be adjacent to the virtual keyboard, which can facilitate the user to refer to and operate during input. The design of the virtual screen can be coordinated with the overall style of the three-dimensional environment, and at the same time, the usability and aesthetics of the user interaction interface should also be considered.

[0077] Among them, an extraction box is set on the virtual screen for displaying the content input by the user or information related to the input content.

[0078] Among them, the extraction box is associated with the input box displayed on the user interaction interface, and the content displayed in the extraction box can be synchronously associated and displayed in the input box. That is, the extraction box should correspond to the input box to ensure that when the user inputs content into the input box, they can intuitively see the display of the input content in the extraction box. The user can conveniently observe and check the input content to avoid input errors or omissions. To ensure the accuracy of synchronous display, the association between the extraction box and the input box should be established on a unified data model. This means that the input box and the extraction box should share the same data source to maintain data consistency during the input process. This association can also be achieved through programming interfaces or data binding. The extended reality device can provide relevant interfaces so that developers can associate the extraction box with the input box. Through these interfaces, developers can implement the synchronous display function of the extraction box and the input box in the application program.

[0079] For example, to enhance the user's interaction experience, the design of the extraction box should also be simple and clear, and easy to use. The size and position of the extraction box can be reasonably arranged according to the input content and the position of the virtual keyboard so that the user can easily observe and edit the input content.

[0080] In addition, to improve the input efficiency, the extraction box can have some intelligent functions. For example, when the user inputs text, the extraction box can update and display the input content in real time so that the user can check and edit at any time. At the same time, the extraction box can also provide corresponding tips and suggestions according to the user's operations to help the user complete the input and extraction tasks faster.

[0081] As Figure 4 shown, the user interaction interface 10 is presented in the three-dimensional environment 1, and an input box 11 is displayed on the user interaction interface 10. For example, the user interaction interface 10 can be the user interaction interface corresponding to the video playback client. In response to a first operation on the input box 11, such as selecting the input box 11 through the virtual cursor 40 to trigger an edit instruction for the input box 11, and then waking up the virtual keyboard 20 to display the virtual keyboard 20 in the three-dimensional environment 1. In addition, a virtual screen 30 is displayed in the adjacent area of the display position of the virtual keyboard 30, and an extraction box 31 is displayed on the virtual screen 30. For example, the virtual screen 30 can be displayed in the adjacent area on the same horizontal plane as the display position of the virtual keyboard 30.

[0082] In some embodiments, the displaying the virtual screen in the adjacent area of the display position of the virtual keyboard includes:

[0083] Displaying the virtual screen at the planar extension of the plane where the display position of the virtual keyboard is located.

[0084] Among them, the planar extension relationship between the virtual keyboard and the virtual screen can be adjusted according to specific application scenarios and user requirements. This means that the virtual screen can be displayed at different positions on the same plane as the virtual keyboard, or be appropriately extended vertically or horizontally with respect to the virtual keyboard.

[0085] For example, if the planar extension is at different positions on the same plane as the virtual keyboard, it is equivalent to mapping the input box to an extraction box in the same plane area as the virtual keyboard, so as to convert the user's input behavior in the three-dimensional environment into an input behavior in the planar space, thereby optimizing the relative position of the input box and the virtual keyboard, so that when the user operates the virtual keyboard with their head down, the specific content entered can more conveniently appear within the user's line of sight, achieving the effect of not having to frequently look up for confirmation during input.

[0086] For example, if the planar extension is at a position where it is appropriately extended vertically with respect to the virtual keyboard, then more space can be allocated vertically to the virtual screen and the extraction box, making the extraction box in the virtual screen more prominent and easier to observe. For instance, when the user is inputting on the virtual keyboard, the user can directly observe the extraction box in the virtual screen above the virtual keyboard without looking up, so as to check the input content or perform other related operations. This layout enables the user not to frequently look up or shift their line of sight, thereby improving the efficiency and accuracy of input.

[0087] By displaying the virtual screen at the planar extension of the virtual keyboard, a larger operation space and richer information display can be provided for the user. For example, when the user needs to input a long text or perform multi-step operations, the virtual screen can be correspondingly extended to the side of the virtual keyboard to provide more input options or operation buttons.

[0088] In addition, this planar extension design can also enhance the relevance between the virtual keyboard and the virtual screen. By placing the virtual screen at the extension of the virtual keyboard, the user can more intuitively understand the relationship between the two and easily switch and operate between them.

[0089] To further improve the interaction efficiency and user satisfaction, the design of the virtual screen at the planar extension can also consider adopting a dynamic adjustment method. This means that the virtual screen can be dynamically expanded and contracted according to the user's operations and requirements to adapt to different interaction scenarios and task needs.

[0090] Setting the virtual screen at the planar extension of the virtual keyboard is a flexible layout method, which can provide a larger operation space and richer information display for the user. Through dynamic adjustment and enhanced relevance, this layout method can also improve the user's interaction efficiency and satisfaction. This will help enhance the practicality and user experience of extended reality devices in actual applications.

[0091] Step 150, in response to a second operation on the virtual keyboard, synchronously display corresponding input content in the extraction box and the input box.

[0092] For example, the second operation includes but is not limited to somatosensory control operations, gesture control operations, eye movement operations, touch operations, voice control commands, or operations on external control devices.

[0093] Among them, in response to the second operation on the virtual keyboard, corresponding operation commands can be generated. Common operation commands include but are not limited to key selection, character input, deletion, copy and paste, etc. These operation commands are used to indicate the generation of corresponding input content so as to synchronously display the corresponding input content in the extraction box and the input box.

[0094] As Figure 4 shown, in response to the second operation on the virtual keyboard 20, synchronously display corresponding input content 50 in the extraction box 31 and the input box 11. For example, the current input content 50 is "I am a kehd ".

[0095] Among them, to achieve this synchronous display, an efficient input processing mechanism is required. The system needs to capture the user's operations on the virtual keyboard in real time and convert them into corresponding text or commands. At the same time, the system also needs to update these input contents to the extraction box and the input box in real time to maintain synchronization.

[0096] To improve the user experience, the process of synchronous display should be as smooth and delay-free as possible. Through fast processing and rendering, users can immediately see the corresponding content in the extraction box and the input box while operating the virtual keyboard, thereby improving the efficiency and accuracy of input.

[0097] In addition, to further enhance the interactive experience between the user and the virtual keyboard, some intelligent elements can be added during the synchronous display process. For example, when the user inputs a specific character or word, the system can automatically predict the next possible character or word and give a prompt in the virtual keyboard. This can help users complete the input task faster and improve the input efficiency.

[0098] As Figure 4As shown, when the user inputs the specific character "kehd", the system can automatically predict the next possible character or word and give a hint in the text display area of the virtual keyboard. For example, the possible characters automatically predicted include "client", "attending a meeting", "optimistic about", "hurry back", "do well in an exam", "good class", "happy". Among them, for the possible characters or words, the system can also display a recommended visual indication to indicate the character or word with the highest matching degree. The recommended visual indication can be text highlighting, background highlighting, font bolding, setting a selection box outside the text, etc. For example, when displaying a recommended visual indication for "client", the recommended visual indication is to set a selection box outside the text.

[0099] For example, the process of synchronous display can also take into account the input habits and preferences of different users. For example, some users may prefer to see the input content updated word by word, while some users may prefer to see the entire sentence or paragraph updated at once. Therefore, the system should provide configurable setting options to allow users to adjust the way of synchronous display according to their own preferences.

[0100] In some embodiments, the synchronously displaying the corresponding input content in the extraction box and the input box in response to a second operation on the virtual keyboard includes:

[0101] In response to a second operation on the virtual keyboard, obtaining the input content input based on the virtual keyboard;

[0102] Based on the input connection interface, sending the input content to the extraction box and the input box for synchronous display.

[0103] Among them, when the user performs a second operation on the virtual keyboard, the system will capture these operations in real time and convert them into corresponding input content. This input content can be characters, words, phrases, or other forms of text input. By recognizing the user's operations, the system can accurately obtain the input content input based on the virtual keyboard.

[0104] Among them, in order to achieve synchronous display between the extraction box and the input box, the system needs to send these input contents to the corresponding display area. This is usually accomplished through the InputConnection interface. InputConnection is an interface used to handle communication between the input method and the application in Android development. It is a bridge in the input method framework for connecting the input method and the application that receives input. Through InputConnection, the input method can interact with the application, sending and receiving user input data. It provides methods to read and modify the text content in the input box, as well as handle cursor positions and raw keyboard events, etc. For example, when a user opens an input method, the InputMethodService will create an InputConnection object to communicate with the application that receives input. The application can interact with the input method through the InputConnection interface, such as obtaining the text input by the user, setting the cursor position, or requesting focus, etc. In the embodiment of the present application, this InputConnection interface is used to connect the virtual keyboard to the input box and the extraction box that receive input, so as to synchronously display the corresponding input content in the extraction box and the input box in response to the second operation on the virtual keyboard.

[0105] During the sending process, the input content can be transmitted and displayed in various forms. For example, the input content can be updated word by word as a text string to the extraction box and the input box, or displayed all at once as an entire sentence or paragraph. In addition, the system can also format or beautify the input content according to the user's preferences and settings to improve the visual effect of the display.

[0106] To implement this process, the extended reality device needs to have an efficient communication mechanism and data processing capabilities. The device needs to be able to quickly process the user's operations, generate the corresponding input content, and send it to the display area through the input connection interface. At the same time, the device also needs to ensure the real-time and accuracy of data transmission to ensure the smoothness and accuracy of synchronous display.

[0107] In some embodiments, the method further includes:

[0108] In response to an update operation on the input content displayed in the extraction box, obtain the updated input content and synchronously display the updated input content in the input box.

[0109] Among them, when the user sees the input content in the extraction box, the user may need to edit or modify this content. To meet this requirement, the system needs to be able to identify and process these update operations. These update operations may include, but are not limited to, editing operations such as selection, deletion, insertion, replacement, etc.

[0110] When the user performs these update operations in the extraction box, the system needs to capture these update operations in real time and convert them into corresponding instructions. These instructions direct the system to make corresponding modifications to the input content. For example, if the user selects a part of the text in the extraction box and deletes it, the system will correspondingly delete the corresponding part of the input content. Once the system receives the user's update operation instruction, it can obtain the updated input content. This can be achieved by directly modifying the corresponding data structure in memory or by communicating with the user's input connection interface to receive the updated input content.

[0111] After obtaining the updated input content, the system needs to synchronously display it in the input box. This means that the content of the input box also needs to be correspondingly modified to reflect the changes made by the user in the extraction box. By maintaining the synchronous display between the input box and the extraction box, the user can intuitively see the final input result.

[0112] In the embodiments of this application, the user can manage the input content through the extraction box in the virtual screen, reducing the difficulty of operations such as adding, deleting, and modifying in the input box in the traditional input method. This method simplifies the operation process of the input content in the input box and improves the user's input efficiency.

[0113] In some embodiments, the synchronously displaying the updated input content in the input box includes:

[0114] Based on the input connection interface, sending the updated input content to the input box for synchronous display.

[0115] Among them, when the user performs editing and modification operations in the extraction box, the system will capture these operations and generate the corresponding updated input content. This content can be changes in text, characters, or cursor positions.

[0116] Next, the system sends the updated input content to the input box through the InputConnection interface. The InputConnection interface acts as a bridge between the extraction box and the input box, responsible for data transmission and processing. Through this interface, the updated input content is sent to the input box in the correct format and protocol for synchronous display. During the sending process, the updated input content may undergo some processing and conversion to ensure its correct display in the input box. For example, changes in the cursor position may require specific protocols and processing logics to ensure its accurate update in the input box.

[0117] In addition, the InputConnection interface also provides other functions, such as reading and modifying the text content in the input box, handling cursor positions, and raw keyboard events. These functions enable the input method to interact more deeply with the application, providing a richer and more flexible user input experience.

[0118] In some embodiments, the method further includes:

[0119] Dynamically adjusting the size of the extraction box according to the input content.

[0120] Among them, when the user inputs content on the virtual keyboard, the extraction box automatically adjusts its size according to the number of input characters. This is because as the input content increases, more space is needed to display more characters. If the size of the extraction box remains unchanged, it may become too crowded, making it difficult for the user to see or edit the input content. Therefore, the system can dynamically adjust the size of the extraction box according to the length of the input content. When the user inputs short text, the extraction box can maintain a relatively small size to save display space and make other interface elements more accessible. However, when the user inputs long text, the system can automatically expand the size of the extraction box to accommodate more characters and improve readability.

[0121] The method of dynamically adjusting the size of the extraction box can be implemented through various technologies. For example, the system can monitor the length of the input content and calculate the optimal size required for the extraction box based on the length value. Once the optimal size is determined, the system can apply it to the display area of the extraction box through graphics rendering technology.

[0122] In addition, to provide a more natural and intuitive experience, the adjustment of the extraction box size can be combined with the user's interaction operations. For example, when the user scrolls or zooms the display area to view more content, the system can dynamically adjust the size of the extraction box according to these operations. This can provide a more consistent and personalized interaction experience for the user.

[0123] In the embodiments of the present application, by dynamically adjusting the size of the extraction frame, the system can automatically optimize the spatial allocation of the display area according to the user's input content, improving readability and user experience. This provides the user with a more flexible and adaptable extended reality environment, enabling them to process and view the input content more efficiently.

[0124] In some embodiments, the method further includes:

[0125] Dynamically adapting the size of the virtual screen according to the size of the dynamically adjusted extraction frame.

[0126] Among them, when the size of the extraction frame is dynamically adjusted according to the input content, the size of the virtual screen also needs to be adjusted accordingly. This is because the virtual screen, as the display area in the extended reality environment, needs to be coordinated with the extraction frame and other interface elements to ensure the clarity and consistency of the entire interface.

[0127] To achieve dynamic adaptation of the size of the virtual screen, the system needs to monitor the current size of the extraction frame. Once the size of the extraction frame changes, the system can calculate the optimal size required for the virtual screen based on the new size value. This calculation process can be based on a certain algorithm or a preset mapping relationship, such as adjusting the size of the virtual screen according to the width and height of the extraction frame.

[0128] When the optimal size of the virtual screen is determined, the system can apply it to the display area through graphics rendering technology. This may involve corresponding adjustments to the boundaries, positions, and proportions of the virtual screen to adapt to the new size.

[0129] In addition, dynamic adaptation of the size of the virtual screen can also be combined with the user's interaction operations. For example, when the user moves, zooms, or rotates the virtual screen through gestures or operations, the system can monitor these operations and dynamically adjust the size of the virtual screen according to the user's intention. This can provide the user with a more natural and intuitive interaction experience.

[0130] Among them, dynamically adapting the size of the virtual screen not only helps to improve the user experience but also brings benefits in terms of resource utilization and performance optimization. By dynamically adjusting the size of the virtual screen, the system can automatically allocate appropriate screen resources according to the current requirements and scenarios, improving the overall performance and response speed.

[0131] In summary, through this dynamic adaptation mechanism, the system can automatically adjust the size of the virtual screen according to the length of the user input content and the operation intention to maintain the clarity and consistency of the interface and improve the user experience and resource utilization efficiency.

[0132] In some embodiments, the method further includes:

[0133] Hide the virtual screen in response to an input completion instruction for the input content.

[0134] Among them, when the user finishes inputting the content, an input completion instruction can be issued to indicate that the content in the input box no longer needs to be modified or added. This instruction can be a specific operation, event, or signal, such as pressing the "Complete" button on the virtual keyboard, not performing an input operation for a long time, etc. When the system receives the input completion instruction, it will hide the virtual screen. This means that the virtual screen will no longer be displayed in the extended reality environment, making the interface elements in the extended reality environment more concise and clear. Hiding the virtual screen can be achieved in different ways, such as fading out gradually, disappearing immediately, or smoothly scaling to transparency, etc.

[0135] One of the benefits of hiding the virtual screen is to reduce interference from unnecessary interface elements. After the user finishes inputting, there is no need to keep the virtual screen displayed to avoid distracting the user's attention or causing confusion. By hiding the virtual screen, the user can focus more on the input content or other related tasks.

[0136] In addition, hiding the virtual screen also helps to improve the simplicity and efficiency of operations. The user does not need to manually close or adjust the virtual screen, and the system will automatically handle this operation. This provides a more intuitive and automated experience for the user, reducing unnecessary operation steps and time costs.

[0137] It should be noted that hiding the virtual screen does not mean completely removing it. In some cases, the user may need to access the virtual screen again for further editing or input. Therefore, the system can be designed to have the function of redisplaying the virtual screen when the user needs it. This can be achieved by providing corresponding operations or trigger conditions, such as pressing the "Complete" button again or executing a specific navigation command. For example, when responding to relevant operations on the input box again, the virtual screen is redisplayed.

[0138] Through this mechanism, the system can automatically adjust the display state of the virtual screen according to the user's operation intention to improve the user experience, reduce interference, and simplify the operation process. This provides a more efficient and convenient extended reality environment for the user, enabling them to focus more on the task and the content itself.

[0139] In some embodiments, the method further includes:

[0140] Adjust the display position of the virtual screen according to the user's line of sight focus.

[0141] Among them, eye tracking technology or similar line-of-sight tracking technology can be used to monitor the user's line-of-sight focus. This technology can determine the area or object they are looking at by analyzing the movement and position of the user's eyeballs. When the user's line-of-sight focus is detected, the position and direction of the line-of-sight focus can be analyzed to determine the content or interface element the user expects to see. For example, if the user is looking at a specific key on the virtual keyboard, the system can adjust the position of the virtual screen accordingly to make it closer to the display position of that key.

[0142] Adjusting the display position of the virtual screen according to the user's line-of-sight focus can improve the user's interaction efficiency and comfort. By placing the virtual screen near the focus of the user's gaze, the user can interact with the virtual screen more easily without having to frequently move their head or eyes to view and operate the virtual screen.

[0143] In addition, this adjustment can also enhance the user's sense of immersion. When the position of the virtual screen matches the user's line-of-sight focus, the user will feel more natural and comfortable, and this sense of immersion can improve the user's engagement and experience quality.

[0144] In addition, the system can also consider the needs and preferences of different users to provide a more personalized and customized interaction experience.

[0145] For example, as Figure 5 shown in the schematic diagram of the input method architecture of the extended reality device, a layout of an extraction area independent of the keyboard area is designed on the input method architecture, which can accommodate the extraction box in the virtual (fake) screen.

[0146] Among them, the entire input method architecture is implemented based on the InputMethodService class. The InputMethodService class is a class used to handle input method logic in Android system development. It provides a standard implementation of the input method and allows developers to customize and extend it. It is mainly used to manage the logic, interaction, and life cycle of the input method, etc. It provides some methods and callback functions that allow developers to easily implement a custom input method.

[0147] Among them, set the Root View. Among them, the Root View, usually a view that places all other views, is the root node in the tree structure and the parent node of all child views.

[0148] Among them, a fullscreen area (FullscreenArea) is set in the root view (Root View). And an extraction area (ExtractArea) is set within the fullscreen area (FullscreenArea). Among them, an extraction frame (ExtractEditTeText) 31 of a virtual (fake) screen 30 as shown in Figure 4 can be set in the extraction area (ExtractArea).

[0149] Among them, the extraction area (ExtractArea) will only be displayed in fullscreen mode (FullscreenMode). Because in an extended reality device, both the client and the input method are independent windows, so the input method of the extended reality device is in the default fullscreen mode (FullscreenMode).

[0150] The extraction area (ExtractArea)

[0151] Among them, an input area (InputArea) is set in the root view (Root View). And an input method root view (InputMethodRootView) is set within the input area (InputArea). The input method root view (InputMethodRootView) generally refers to the root view component used to display the input method interface in the user interface. In Android development, the InputMethodRootView is a layer of view component used to display the input method interface. It usually serves as a bridge between the input method interface and the application interface, manages the display and hiding of the input method interface, and processes the interaction between the user and the input method interface. In the embodiments of this application, a virtual keyboard 20 as shown in Figure 4 can be constructed based on this input method root view (InputMethodRootView).

[0152] Among them, the input content displayed in the extraction frame of the virtual (fake) screen is obtained and updated from the InputConnection interface. InputConnection is an interface used to handle the communication between the input method and the application in Android development. It is a bridge in the input method framework for connecting the input method and the application receiving the input. Through InputConnection, the input method can interact with the application, sending and receiving user input data. It provides methods to read and modify the text content in the input box, as well as handle the cursor position and raw keyboard events, etc. For example, when the user opens an input method, the InputMethodService will create an InputConnection object to communicate with the application receiving the input. The application can interact with the input method through the InputConnection interface, such as obtaining the text input by the user, setting the cursor position, or requesting focus, etc. In the embodiment of the present application, this InputConnection interface is used to connect the virtual keyboard to the input box and the extraction frame receiving the input, so as to synchronously display the corresponding input content in the extraction frame and the input box in response to the second operation on the virtual keyboard.

[0153] For example, in step 150, in response to the second operation on the virtual keyboard, the corresponding input content is displayed in the extraction frame, which can be specifically implemented through the process as Figure 6 shown below:

[0154] S1. Display the input method window; in the embodiment of the present application, this input method window is a virtual keyboard displayed in a three-dimensional environment.

[0155] S2. Update the full-screen mode (updateFullscreenMode); this step can be used to update the full-screen display mode of the application. If you want the application to enter the full-screen mode and hide the status bar and navigation bar, it can be achieved by calling the function of updating the full-screen mode (updateFullscreenMode).

[0156] S3. Evaluate the full-screen mode (onEvaluateFullscreenMode); this step can be used to evaluate whether to enter the full-screen mode. For example, when the user interacts with the interface, they may want to dynamically evaluate whether to enter the full-screen mode. Based on the function of this evaluate full-screen mode (onEvaluateFullscreenMode), it can be decided whether to update the full-screen mode according to the current conditions.

[0157] S4. Whether it is full screen (isFullScreen). This step can be used to check whether the current mode is the full-screen mode (FullscreenMode). For example, by calling the function of whether it is full screen (isFullScreen), it can be checked whether the application has entered the full-screen mode. In the embodiment of the present application, when it is detected that the input method of the extended reality device enters the full-screen mode, step S5 is further executed.

[0158] S5. Create an extraction view (onCreateExtractView); this step can be used to create an extraction view. In the embodiment of the present application, the extraction view is a virtual (fake) screen for displaying an extraction frame.

[0159] S6. Set the extraction view (setExtractView); this step can be used to set or update the extraction view. After an extraction view has been created, this method can be used to set the extraction view to a certain position or associate it with a certain component. In the embodiment of the present application, the extraction view is a virtual (fake) screen for displaying an extraction frame. After a virtual (fake) screen has been created, by calling the function of setting the extraction view (setExtractView), the virtual (fake) screen can be set to a certain position or associated with a certain component, such as setting the virtual (fake) screen in the adjacent area of the display position of the virtual keyboard.

[0160] S7. Set the extracted text (setExtractingText); this step can be used to set or update the text content that is being extracted or clipped. In the embodiment of the present application, when text content is input into the extraction frame based on the virtual keyboard, the function of setting the extracted text (setExtractingText) can be called to set or update the text content being input into the extraction frame.

[0161] S8. Obtain the extraction frame based on the input connection interface (InputConnection.getExtractedText); the input connection (InputConnection) is an interface in Android for handling the communication between the input method and the application. The text content in the extraction frame (getExtractedText) is obtained and updated from the virtual keyboard based on the input connection (InputConnection) interface.

[0162] S9. Update Extraction Frame Visibility (updateExtractFrameVisibility); This step can be used to update the visibility of the extraction frame. When processing the extraction view, it may be necessary to control the visibility of the extraction view. In an embodiment of the present application, after completing text input based on the virtual keyboard, the virtual (fake) screen can be hidden by calling the function of updating the extraction frame visibility (updateExtractFrameVisibility). For example, when text input is performed based on the virtual keyboard, the virtual (fake) screen can be displayed by calling the function of updating the extraction frame visibility (updateExtractFrameVisibility). This method can control the visible state of the virtual (fake) screen.

[0163] S10. Start Input (doStartinput); This step can be used to start some input operations, such as keyboard input or handwriting recognition. In some input scenarios, such as when starting an input method, you can call the doStartinput function to perform some input method initialization operations or prepare the input environment.

[0164] S11. Set the extraction text (setExtractingText): After calling the start input (doStartinput) function to enter the input method initialization operation or prepare the input environment, you can further call the set extraction text (setExtractingText) function to set or update the text content being entered in the extraction box when entering text content in the extraction box based on the virtual keyboard.

[0165] S12. When the input method is closed and / or the input box of the client is changed, the input method process is ended.

[0166] All of the above technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0167] Embodiments of the present application generate a three-dimensional environment through a display extended reality device; present a user interaction interface within the three-dimensional environment, and an input box is displayed on the user interaction interface; in response to a first operation on the input box, display a virtual keyboard in the three-dimensional environment; display a virtual screen in an adjacent area of the display position of the virtual keyboard, and an extraction box is displayed on the virtual screen; in response to a second operation on the virtual keyboard, synchronously display corresponding input content in the extraction box and the input box. Embodiments of the present application display a virtual screen with a prompt box in an adjacent area of the display position of the virtual keyboard, and synchronously display corresponding input content in the extraction box and the input box in response to relevant operations on the virtual keyboard, which is equivalent to mapping the input box to an extraction box in the same plane area as the virtual keyboard, so as to convert the user's input behavior in the three-dimensional environment into an input behavior in the plane space, thereby optimizing the relative positions of the input box and the virtual keyboard, enabling the specific content entered by the user to appear more conveniently within the user's line of sight when the user looks down to operate the virtual keyboard, achieving the effect of not needing to frequently look up and confirm during input, improving the interaction convenience and input efficiency on the extended reality device, and thus providing a more smooth and natural input experience for the user.

[0168] To facilitate better implementation of the interaction method of the embodiments of the present application, embodiments of the present application also provide an interaction device. Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of the interaction device provided by the embodiments of the present application.

[0169] Among them, the interaction device 200 may include:

[0170] A first display unit 210, configured to display a three-dimensional environment generated by an extended reality device;

[0171] A second display unit 220, configured to present a user interaction interface within the three-dimensional environment, and an input box is displayed on the user interaction interface;

[0172] A third display unit 230, configured to display a virtual keyboard in the three-dimensional environment in response to a first operation on the input box;

[0173] A fourth display unit 240, configured to display a virtual screen in an adjacent area of the display position of the virtual keyboard, and an extraction box is displayed on the virtual screen;

[0174] An interaction unit 250, configured to synchronously display corresponding input content in the extraction box and the input box in response to a second operation on the virtual keyboard.

[0175] In some embodiments, the interaction unit 250 is configured to:

[0176] In response to a second operation on the virtual keyboard, obtain the input content entered based on the virtual keyboard;

[0177] Based on the input connection interface, send the input content to the extraction box and the input box for synchronous display.

[0178] In some embodiments, the interaction unit 250 may further be configured to:

[0179] In response to an update operation on the input content displayed in the extraction box, obtain the updated input content and synchronously display the updated input content in the input box.

[0180] In some embodiments, when the interaction unit 250 synchronously displays the updated input content in the input box, it may be configured to:

[0181] Based on the input connection interface, send the updated input content to the input box for synchronous display.

[0182] In some embodiments, the fourth display unit 240 may be configured to:

[0183] Display a virtual screen at a plane extension of the plane where the display position of the virtual keyboard is located.

[0184] In some embodiments, the interaction unit 250 may further be configured to:

[0185] Dynamically adjust the size of the extraction box according to the input content.

[0186] In some embodiments, the interaction unit 250 may further be configured to:

[0187] Dynamically adapt the size of the virtual screen according to the dynamically adjusted size of the extraction box.

[0188] In some embodiments, the interaction unit 250 may further be configured to:

[0189] In response to an input completion instruction for the input content, hide the virtual screen.

[0190] In some embodiments, the interaction unit 250 may further be configured to:

[0191] Adjust the display position of the virtual screen according to the user's line of sight focus.

[0192] Each unit in the above interaction device 200 can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above units can be embedded in the processor of the terminal device in hardware form or be independent of it, or can be stored in the memory of the terminal device in software form, so that the processor can call and execute the operations corresponding to each of the above units.

[0193] The interaction device 200 can be integrated in a terminal or server with a memory and a processor installed and having computing capabilities, or the interaction device 200 is the terminal or server.

[0194] In some embodiments, the present application also provides a terminal device, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0195] As Figure 8 shown, Figure 8 is a schematic structural diagram of the terminal device provided by the embodiment of the present application. The terminal device 300 can generally be provided in the form of glasses, a head-mounted display (HMD), or contact lenses for realizing visual perception and other forms of perception. Of course, the form in which the terminal device is implemented is not limited to this, and it can be further miniaturized or enlarged according to needs. The terminal device 300 may include but is not limited to the following components:

[0196] Detection module 301: Use various sensors to detect the user's operation commands and act on the virtual environment, such as continuously updating the image displayed on the display screen following the user's line of sight to realize the interaction between the user and the virtual and real scenes. For example, continuously update the real content based on the detected rotation direction of the user's head.

[0197] Feedback module 302: Receive data from the sensors and provide real-time feedback to the user. Among them, the feedback module 302 can be used to display a graphical user interface, such as displaying a virtual environment on the graphical user interface. For example, the feedback module 302 may include a display screen, etc.

[0198] Sensor 303: On the one hand, receive the user's operation commands and act on the virtual environment; on the other hand, provide the results generated after the operation to the user in various feedback forms.

[0199] Control module 304: Control the sensors and various input / output devices, including obtaining the user's data (such as actions, voices) and outputting perception data, such as images, vibrations, temperatures, and sounds, and acting on the user, the virtual environment, and the real world.

[0200] Modeling module 305: Construct a three-dimensional model of the virtual environment, which may also include various feedback mechanisms such as sound and touch in the three-dimensional model.

[0201] In the embodiments of the present application, a virtual scene in a three-dimensional environment can be constructed through the modeling module 305; the extended reality device-generated three-dimensional environment can be displayed through the feedback module 302, and a user interface for interaction can be presented within the three-dimensional environment; the first operation on the input box and the second operation on the virtual keyboard can be obtained through the detection module 301 and / or the sensor 303; in response to the first operation on the input box, the virtual keyboard can be displayed in the three-dimensional environment through the feedback module 302 by the control module 304; a virtual screen can be displayed in an adjacent area of the display position of the virtual keyboard through the feedback module 302, and an extraction box is displayed on the virtual screen; in response to the second operation on the virtual keyboard, the corresponding input content can be synchronously displayed in the extraction box and the input box through the feedback module 302 by the control module 304.

[0202] In some embodiments, as Figure 9 shown, Figure 9 Another schematic structural diagram of the terminal device provided by the embodiments of the present application, the terminal device 300 further includes a processor 310 with one or more processing cores, a memory 320 with one or more computer-readable storage media, and a computer program stored on the memory 320 and executable on the processor. Among them, the processor 310 is electrically connected to the memory 320. Those skilled in the art can understand that the structural diagram of the terminal device shown in the figure does not limit the terminal device, and it may include more or fewer components than shown in the figure, or combine certain components, or arrange different components.

[0203] The processor 310 is the control center of the terminal device 300, connecting various parts of the entire terminal device 300 through various interfaces and lines, and by running or loading software programs and / or modules stored in the memory 320, and calling data stored in the memory 320, it executes various functions of the terminal device 300 and processes data, thereby monitoring the terminal device 300 as a whole.

[0204] In the embodiments of the present application, the processor 310 in the terminal device 300 will load the instructions corresponding to the processes of one or more application programs into the memory 320 according to the following steps, and the processor 310 will run the application programs stored in the memory 320 to implement various functions:

[0205] Display a three-dimensional environment generated by an extended reality device; present a user interaction interface within the three-dimensional environment, and an input box is displayed on the user interaction interface; in response to a first operation on the input box, display a virtual keyboard in the three-dimensional environment; display a virtual screen in an adjacent area of the display position of the virtual keyboard, and an extraction box is displayed on the virtual screen; in response to a second operation on the virtual keyboard, synchronously display corresponding input content in the extraction box and the input box.

[0206] For the specific implementation of each of the above operations, reference can be made to the previous embodiments, which will not be elaborated here.

[0207] In some embodiments, the processor 310 may include a detection module 301, a control module 304, and a modeling module 305.

[0208] In some embodiments, as Figure 9 shown, the terminal device 300 further includes: a radio frequency circuit 306, an audio circuit 307, and a power supply 308. Among them, the processor 310 is electrically connected to the memory 320, the feedback module 302, the sensor 303, the radio frequency circuit 306, the audio circuit 307, and the power supply 308 respectively. Those skilled in the art can understand that Figure 8 or Figure 9 the structure of the terminal device shown in

[0209] does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0209] The radio frequency circuit 306 can be used to receive and transmit radio frequency signals to establish wireless communication with a network device or other terminal devices, and receive and transmit signals with the network device or other terminal devices.

[0210] The audio circuit 307 can be used to provide an audio interface between the user and the terminal device through a speaker and a microphone. The audio circuit 307 can transmit the electrical signal after converting the received audio data to the speaker, and the speaker converts it into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 307 and then converted into audio data. After the audio data is output to the processor 310 for processing, it is sent through the radio frequency circuit 306 to, for example, another terminal device, or the audio data is output to the memory for further processing. The audio circuit 307 may also include an earphone jack to provide communication between a peripheral earphone and the terminal device.

[0211] The power supply 308 is used to supply power to each component of the terminal device 300.

[0212] Although Figure 8 or Figure 9Not shown in the figure, the terminal device 300 may further include a camera, a Wi-Fi module, a Bluetooth module, an input module, etc., which will not be elaborated here.

[0213] In some embodiments, the present application further provides a computer-readable storage medium for storing a computer program. The computer-readable storage medium can be applied to the terminal device or the server, and the computer program enables the terminal device or the server to execute the corresponding processes in the interaction method in the embodiments of the present application. For the sake of brevity, it will not be elaborated here.

[0214] In some embodiments, the present application further provides a computer program product. The computer program product includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the terminal device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, enabling the terminal device to execute the corresponding processes in the interaction method in the embodiments of the present application. For the sake of brevity, it will not be elaborated here.

[0215] The present application further provides a computer program. The computer program includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the terminal device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, enabling the terminal device to execute the corresponding processes in the interaction method in the embodiments of the present application. For the sake of brevity, it will not be elaborated here.

[0216] It should be understood that the processor in the embodiments of the present application may be an integrated circuit chip with the ability to process signals. In the implementation process, the steps of the above method embodiments can be completed by the integrated logic circuit in the hardware of the processor or the instructions in the form of software. The above processor may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed by the hardware decoding processor, or executed by the combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0217] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0218] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0219] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0220] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the function of the module or unit.

[0221] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical, or other form.

[0222] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0223] In addition, each functional unit in the embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a terminal device (which can be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disc that can store program codes.

[0224] As described above, it is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An interaction method, characterized in that, The method includes: Displaying a three-dimensional environment generated by an extended reality device; Presenting a user interaction interface within the three-dimensional environment, where an input box is displayed on the user interaction interface; In response to a first operation on the input box, displaying a virtual keyboard in the three-dimensional environment; Displaying a virtual screen in an adjacent area of the display position of the virtual keyboard, where an extraction box is displayed on the virtual screen; In response to a second operation on the virtual keyboard, synchronously displaying corresponding input content in the extraction box and the input box.

2. The interactive method according to claim 1, wherein The step of, in response to a second operation on the virtual keyboard, synchronously displaying corresponding input content in the extraction box and the input box includes: In response to a second operation on the virtual keyboard, obtaining the input content input based on the virtual keyboard; Based on an input connection interface, sending the input content to the extraction box and the input box for synchronous display.

3. The interactive method according to claim 2, wherein The method further includes: In response to an update operation on the input content displayed in the extraction box, obtaining the updated input content and synchronously displaying the updated input content in the input box.

4. The interactive method according to claim 3, wherein The step of synchronously displaying the updated input content in the input box includes: Based on the input connection interface, sending the updated input content to the input box for synchronous display.

5. The interactive method according to claim 1, wherein The step of displaying a virtual screen in an adjacent area of the display position of the virtual keyboard includes: Displaying a virtual screen at a planar extension of the plane where the display position of the virtual keyboard is located.

6. The interactive method according to claim 1, wherein The method further includes: Dynamically adjusting the size of the extraction box according to the input content.

7. The interactive method according to claim 6, wherein The method further includes: Dynamically adapting the size of the virtual screen according to the dynamically adjusted size of the extraction box.

8. The interactive method according to claim 1, characterized in that The method further includes: In response to an input completion instruction for the input content, hiding the virtual screen.

9. The interactive method according to claim 1, characterized in that, The method further includes: Adjusting the display position of the virtual screen according to the user's line of sight focus.

10. An interaction device, characterized in that, The device includes: A first display unit for displaying a three-dimensional environment generated by an extended reality device; A second display unit for presenting a user interaction interface within the three-dimensional environment, where an input box is displayed on the user interaction interface; A third display unit for, in response to a first operation on the input box, displaying a virtual keyboard in the three-dimensional environment; A fourth display unit for displaying a virtual screen in an adjacent area of the display position of the virtual keyboard, where an extraction box is displayed on the virtual screen; An interaction unit for, in response to a second operation on the virtual keyboard, synchronously displaying corresponding input content in the extraction box and the input box.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program is adapted to be loaded by a processor to execute the interaction method according to any one of claims 1-9.

12. A terminal device, characterized in that, The terminal device includes a processor and a memory, and the memory stores a computer program. The processor is configured to execute the interaction method according to any one of claims 1-9 by calling the computer program stored in the memory.

Images