Display method of virtual keyboard and head-mounted display equipment

Abstract

The invention relates to the technical field of virtual interaction, provides a display method of a virtual keyboard and head-mounted display equipment, and is used for solving the problem that the virtual keyboard is shielded. For the condition that a virtual keyboard is used for inputting, two layers are created, and the current depth of the layer corresponding to the virtual keyboard is controlled to be smaller than the minimum depth of the layer corresponding to the 3D application, so that an image of the virtual keyboard is always displayed above a picture of the 3D application, the problem that the virtual keyboard is shielded is solved, and meanwhile, the image of the virtual keyboard is displayed on the 3D application. By rendering the picture of the 3D application and the image of the virtual keyboard to different layers, rendering of the picture of the 3D application and rendering of the image of the virtual keyboard are separated, system-level display of the virtual keyboard is achieved, later maintenance is facilitated, and compared with application layer display integrated in the 3D application, the development difficulty is lowered, and the development efficiency is improved. And the reliability and the flexibility of the virtual keyboard are also improved.

Description

Background Art

[0002] For head-mounted display devices (such as VR devices, AR devices, MR devices, etc.), realizing user interaction in 3D applications is a core technology for immersive experience. As a common interaction method, keyboard input plays an important role in virtual environments. Users can use the keyboard to input text, execute commands, and so on.

[0003] At present, the virtual keyboards of most head-mounted display devices on the market use the application layer rendering method, which integrates the rendering of the virtual keyboard into the 3D application and is rendered by each 3D application separately. Even if some head-mounted display devices use self-developed input methods, the rendering method of the virtual keyboard has not changed. However, unlike 2D applications, since the virtual keyboard is rendered in the 3D application, it is easily partially blocked due to factors such as position placement, thus affecting the input operation. Summary of the invention

[0004] The embodiments of the present application provide a method for displaying a virtual keyboard and a head-mounted display device, which are used to solve the problem of a virtual keyboard being blocked and improve the reliability and flexibility of the virtual keyboard display.

[0005] In a first aspect, an embodiment of the present application provides a method for displaying a virtual keyboard, comprising:

[0006] In response to a start-up operation of the 3D application, a first layer is created, a screen of the 3D application is rendered onto the first layer, and the screen is submitted to the runtime;

[0007] In response to a keyboard interaction operation triggered for the 3D application, starting an input method service and creating a second layer, rendering an image of a virtual keyboard corresponding to the input method service onto the second layer and submitting the image to the runtime;

[0008] According to the current position parameters of the head and the current position parameters of the virtual camera, the current depth of the second layer is controlled to be smaller than the minimum depth of the first layer, so that the second layer is not blocked by the first layer.

[0009] The beneficial effects of the above technical solution are: by creating two layers, the screen of the 3D application and the image of the virtual keyboard are rendered on different layers, and the current depth of the layer corresponding to the virtual keyboard is controlled to be less than the minimum depth of the layer corresponding to the application screen. In this way, the image of the virtual keyboard can be kept always displayed above the screen of the 3D application, thereby solving the problem of the virtual keyboard being blocked. At the same time, the rendering of the 3D application screen and the rendering of the virtual keyboard image are separated through two layers, so as to realize the system-level display of the virtual keyboard, which is convenient for later maintenance. Compared with the application layer display integrated in the 3D application, it not only reduces the development difficulty, but also improves the reliability and flexibility of the virtual keyboard.

[0010] Optionally, the first position data includes at least a position and a field of view of the head, and controlling the current depth of the second layer to be less than the minimum depth of the first layer according to the first position parameter and the second position parameter so that the second layer is not blocked by the first layer includes:

[0011] Determining the front position of the head according to the posture of the head, and determining the line of sight of the head according to the field of view of the head;

[0012] Determine a position at a first preset distance in front of the position and a second preset distance below the line of sight, which is toward the head, as the position of the virtual keyboard in the three-dimensional space;

[0013] Determining the current depth of the virtual keyboard according to a second position parameter of the virtual camera and the position of the virtual keyboard in the three-dimensional space; wherein the second position parameter includes at least the position and the field of view angle of the virtual camera;

[0014] The current depth of the second layer is controlled to be smaller than the minimum depth of the first layer, so that the second layer is not blocked by the first layer.

[0015] The beneficial effects of the above technical solution are: determining the optimal position of the virtual keyboard in three-dimensional space according to the head posture and field of view, which is usually located at a certain distance below the player's line of sight, ensuring that the virtual keyboard is adapted to the screen of the 3D application, which can improve the comfort of user interaction in different modes, improve the display effect of the virtual keyboard, and, by considering the position and field of view of the virtual camera, the display of the virtual keyboard can be adapted to the user's perspective, further improving the display effect of the virtual keyboard.

[0016] Optionally, after controlling the current depth of the second layer to be less than the minimum depth of the first layer, the method further includes:

[0017] In response to an input operation triggered by the 3D application, obtaining a position and a direction of an interaction ray;

[0018] Determining a focus position of the interaction ray on the virtual keyboard according to the position and orientation of the interaction ray and the position information of the virtual keyboard;

[0019] The text at the focus position is acquired, and the text is added to the screen of the 3D application.

[0020] The beneficial effect of the above technical solution is: the focus position is determined by the position and direction of the interaction ray, so that the text at the focus position is input into the 3D application, and the user can interact with the 3D application based on the virtual keyboard.

[0021] Optionally, the method further includes:

[0022] In response to the update operation of the input method service, sending a keyboard resource request to a server;

[0023] receiving a resource package returned by the server according to the keyboard resource request;

[0024] The virtual keyboard updated according to the resource pack is displayed on the second layer.

[0025] The beneficial effect of the above technical solution is: the rendering of the 3D application screen and the rendering of the virtual keyboard image are separated through two layers. In this way, when the input method service needs to be updated, it is only necessary to download the resource package of the virtual keyboard, without compiling and publishing the 3D application, thereby improving the maintainability of the input method service.

[0026] Optionally, when the resource package is a layout resource package of the virtual keyboard, the method further includes:

[0027] Detecting whether the layout resource package of the virtual keyboard is updated;

[0028] If yes, updating the UI controls of the virtual keyboard according to the layout resource package, and the position information of the virtual keyboard in the three-dimensional space determined according to the first position parameter;

[0029] The input text is acquired according to the updated position information and fed back to the 3D application.

[0030] The beneficial effect of the above technical solution is: by requesting the layout resource package of the virtual keyboard from the server, the virtual keyboard layout is automatically updated without manual development.

[0031] Optionally, when the resource package is a font resource package of the virtual keyboard, the method further includes:

[0032] When the input method service is started next time, the font library of the virtual keyboard is updated according to the font library resource package.

[0033] The beneficial effect of the above technical solution is: by requesting the font resource package of the virtual keyboard from the server, the virtual keyboard font library is automatically updated without manual development.

[0034] Optionally, controlling the current depth of the second layer to be smaller than the minimum depth of the first layer so that the second layer is not blocked by the first layer includes:

[0035] The graphics synthesizer is called to overlay the second layer on the first layer in a semi-transparent manner for display.

[0036] The beneficial effects of the above technical solution are: while solving the problem of the virtual keyboard being blocked, the abruptness of the virtual keyboard display is reduced through a semi-transparent display method, thereby improving the display effect.

[0037] Optionally, in response to the start-up operation of the 3D application, creating a first layer, rendering the screen of the 3D application onto the first layer and submitting it to the runtime includes:

[0038] Creating an interface instance and a session for communicating with the runtime, wherein the session is used to manage the life cycle of the 3D application;

[0039] Creating an interaction profile of the 3D application according to the hardware of the device itself, and determining an interaction mode of the 3D application and an interaction button in the interaction profile according to a preset usage scenario; the interaction button is used to receive an interaction operation of the user;

[0040] Create a first swap chain component and a first layer according to the graphics interface and rendering parameters;

[0041] Acquire a rendering carrier of the 3D application through the first exchange chain component, and render a picture of the 3D application onto the carrier to obtain a picture texture;

[0042] The screen texture is rendered onto the first layer for submission to the runtime.

[0043] The beneficial effect of the above technical solution is that the screen of the 3D application is rendered separately, which not only reduces the difficulty of developing the 3D application, but also improves the rendering efficiency of the 3D application compared to integrating the virtual keyboard in the 3D application.

[0044] In a second aspect, an embodiment of the present application provides a display device for a virtual keyboard, comprising:

[0045] An application rendering module, configured to create a first layer in response to a start-up operation of a 3D application, render a picture of the 3D application onto the first layer, and submit the picture to a runtime;

[0046] An input method rendering module, configured to, in response to a keyboard interaction operation triggered for the 3D application, start an input method service and create a second layer, render an image of a virtual keyboard corresponding to the input method service onto the second layer and submit it to the runtime;

[0047] A synthesis module is used to control the current depth of the second layer to be less than the minimum depth of the first layer according to the current position parameters of the head and the current setting parameters of the virtual camera, so that the second layer is not blocked by the first layer; wherein the virtual camera is used to represent the position of the human eye.

[0048] In a third aspect, an embodiment of the present application provides a head mounted display device, comprising a processor, a memory and a display screen, wherein the display screen, the memory and the processor are connected via a bus;

[0049] The display screen is used to display the layer;

[0050] The memory stores a computer program, and the processor executes the steps of any one of the methods for displaying a virtual keyboard provided in the first aspect according to the computer program.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed, the steps of any one of the virtual keyboard display methods provided in the first aspect can be implemented.

[0052] The technical effects brought about by any one of the implementation methods in the second to fourth aspects can refer to the technical effects brought about by the corresponding implementation method in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1A and Figure 1B A schematic diagram of virtual keyboard shielding provided in an embodiment of the present application;

[0055] Figure 2 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0056] Figure 3 A flow chart of a method for displaying a virtual keyboard provided in an embodiment of the present application;

[0057] Figure 4 A flowchart of a rendering method for a 3D application provided in an embodiment of the present application;

[0058] Figure 5 A timing interaction diagram of a 3D application and runtime provided in an embodiment of the present application;

[0059] Figure 6 A flowchart of a rendering method for an input method service provided in an embodiment of the present application;

[0060] Figure 7 A flow chart of a method for inputting using a virtual keyboard provided in an embodiment of the present application;

[0061] Figure 8 An interactive timing diagram of inputting using a virtual keyboard provided in an embodiment of the present application;

[0062] Fig. 9 A display effect diagram of a virtual keyboard provided in an embodiment of the present application;

[0063] Fig.10 An update sequence diagram of the input method service provided in the embodiment of the present application;

[0064] Fig.11 A structural diagram of a display device of a virtual keyboard provided in an embodiment of the present application;

[0065] Fig.12 A structural diagram of a head-mounted display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0067] Based on the exemplary embodiments shown in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete technical solution separately.

[0068] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0069] The terms "first", "second", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the diagrams or descriptions of the embodiments of this application.

[0070] In addition, the terms "include" and "have" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components explicitly listed but may include other components not explicitly listed or inherent to such products or devices.

[0071] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0072] The following is an overview of the design concept of the embodiments of the present application in conjunction with application scenarios.

[0073] Currently, the virtual keyboards in head-mounted display devices are mostly manually coded by developers and integrated into each 3D application. Each application renders it separately. However, the rendering method of the application layer has the following main problems:

[0074] 1. The virtual keyboard in 3D applications requires high precision in placement. If it is not placed properly, the virtual keyboard will be partially blocked by objects in the application screen, which requires manual adjustment by the developer. Figure 1A and Figure 1B In the 3D application, part of the virtual keyboard is blocked by the panel of the 3D application.

[0075] 2. The complexity of integrating virtual keyboards in 3D applications is relatively high, and usually requires developers to perform a lot of manual coding and position adjustments, which not only increases the development difficulty but also prolongs the development cycle;

[0076] 3. The virtual keyboard is updated unchanged. The virtual keyboard is integrated into the 3D application. When the keyboard layout or interface needs to be updated, the 3D application usually needs to be recompiled and republished, which is not only time-consuming and labor-intensive, but may also affect the normal use of users.

[0077] 4. Low utilization of virtual space. In a virtual environment, the virtual keyboard will occupy a certain amount of virtual space, which is not conducive to the design and optimization of the virtual space layout;

[0078] 5. Impact on user experience. Due to the diversity of 3D applications, when a virtual keyboard is integrated into a 3D application, users can easily make changes to the virtual keyboard, which may result in differences in the virtual keyboards of different 3D applications or loss of some functions.

[0079] In view of this, an embodiment of the present application provides a method for displaying a virtual keyboard, which solves the above-mentioned problems existing when integrating a virtual keyboard in a 3D application through a system-level rendering method of a virtual keyboard. For 3D applications and input method services, a first layer and a second layer are created respectively, and the screen of the 3D application is rendered in the first layer, and the image of the virtual keyboard corresponding to the input method service is rendered in the second layer. In this way, the problem of the virtual keyboard being blocked is solved by superimposing the second layer on the first layer for display. In addition, the rendering of the application screen and the rendering of the virtual keyboard image are separated through the first layer and the second layer, which not only reduces the difficulty of developing 3D applications when integrating a virtual keyboard, but also facilitates automatic updating of the virtual keyboard, making maintenance more flexible. On the other hand, since the virtual keyboard image is rendered at the system level through the second layer, the user cannot change the virtual keyboard when using a 3D application, which effectively improves the reliability and security of the virtual keyboard.

[0080] See also Figure 2 , is a schematic diagram of an application scenario provided in an embodiment of the present application. A user wears a head-mounted display device to experience a 3D game. During the experience, the user can start an input method service and input in the 3D game through a virtual keyboard provided by the input method service.

[0081] It should be noted that Figure 2 This is just an example of a scenario. The embodiment of the present application does not impose any restrictive requirements on the interaction mode of the head mounted display device, except for Figure 2 In addition to the handle interaction, it can also be gesture interaction, glove interaction, ring interaction and other modes.

[0082] In addition, the embodiments of the present application do not impose any restrictive requirements on the type of input method service. It can be an input method already available on the market or a self-developed input method.

[0083] See also Figure 3 , is a virtual keyboard display method process provided in an embodiment of the present application, the process is performed by a head mounted display device, and mainly includes the following steps:

[0084] S301: In response to a start-up operation of a 3D application, a first layer is created, and a screen of the 3D application is rendered onto the first layer and submitted to the runtime.

[0085] The user triggers the command to start the 3D application by clicking the icon of the 3D application or confirming the icon of the 3D application. The head-mounted display device responds to the command and renders the screen of the 3D application. For the specific process, see Figure 4 , mainly includes the following steps:

[0086] S3011: Create an interface instance and session for communicating with the runtime.

[0087] Taking OpenXR as an example, the head-mounted display device creates an OpenXR instance of the application itself by calling the xrCreateInstance function. This interface instance can be used to implement the communication process between the 3D application and the runtime (Runtime), as well as to create a session (Session) between the 3D application and the runtime. The session is used to manage the entire life cycle of the 3D application from startup to shutdown.

[0088] S3012: Create an interaction profile of the 3D application according to the hardware of the device itself, and determine the interaction mode of the 3D application and the interaction buttons in the interaction profile according to a preset usage scenario.

[0089] Generally speaking, a head-mounted display device will create a unique interaction profile for a 3D application based on its own hardware, and determine the interaction mode of the 3D application and the interaction buttons in the interaction profile of the 3D application for receiving user interaction operations under the interaction mode based on the preset usage scenario corresponding to the 3D application.

[0090] Optionally, the interaction modes include but are not limited to handle interaction, gesture interaction, ring interaction, glove interaction, head control interaction, etc.

[0091] S3013: Create a first swap chain component and a first layer according to the graphics interface and rendering parameters.

[0092] When developing 3D applications, developers usually set a graphics interface for the 3D application, such as OpenGL and Vulkan, based on factors such as the positioning of the 3D application and product performance. Therefore, when the head-mounted display device starts a 3D application, it will create a swap chain component (Swapchain) and a normal layer (Layer) for the 3D application based on the set graphics interface and the rendering parameters of the 3D application (such as texture, material, lighting, etc.).

[0093] In order to distinguish the swap chain components and layers of the input method service from the 3D application, the swap chain component corresponding to the 3D application is recorded as the first swap chain component, the swap chain component corresponding to the input method service is recorded as the second swap chain component, the layer corresponding to the 3D application is recorded as the second layer, and the layer corresponding to the input method service is recorded as the second layer.

[0094] S3014: Obtain a rendering carrier of the 3D application through the first exchange chain component, and render the screen of the 3D application onto the carrier to obtain a screen texture.

[0095] The rendering carrier Image is created when the 3D application is started. For example, if the graphics interface of the 3D application is OpenGL, the rendering carrier is the texture2D texture of OpenGL, which is blank before the 3D application is rendered.

[0096] When rendering each frame of the 3D application, a rendering carrier is obtained through the first exchange chain component, and the current frame is rendered onto the rendering carrier to obtain a picture texture of the current frame.

[0097] S3015: Render the screen texture onto the first layer to submit to the runtime.

[0098] The image texture of each frame of the 3D application is rendered to the first layer and submitted to the runtime through the first exchange chain component so as to be synthesized with images of other system layers (such as the image of the virtual keyboard, the image of the prompt information, etc.) and displayed on the screen after processing operations such as prediction and anti-distortion.

[0099] See also Figure 5 , which is a timing interaction diagram of 3D application and runtime in the head mounted display device provided in an embodiment of the present application, mainly includes the following steps:

[0100] S1: The 3D application establishes an inter-process communication connection with the runtime.

[0101] S2: The runtime creates a corresponding first layer and a first swap chain component for the 3D application.

[0102] S3: The runtime sets a handle for the first swap chain component and sends it to the 3D application.

[0103] S4: The 3D application renders the image to the first layer according to the handle of the first swap chain.

[0104] S5: The 3D application submits the first layer.

[0105] S6: During the runtime, the first layer is synthesized, predicted, and anti-distorted with other images in the system layer, and then displayed on the screen.

[0106] In the embodiment of the present application, the rendering of the virtual keyboard is separated from the 3D application, and the screen of the 3D application is rendered separately. Compared with the integrated rendering, it not only reduces the difficulty of developing the 3D application, but also improves the rendering efficiency of the 3D application.

[0107] S302: In response to a keyboard interaction operation triggered for a 3D application, start an input method service and create a second layer, render an image of a virtual keyboard corresponding to the input method service onto the second layer and submit it to the runtime.

[0108] When the user wants to input in the 3D application, he moves the focus of the interaction ray to the text box and confirms, triggering an interactive operation using the virtual keyboard. The head-mounted display device responds to the interactive operation, starts the input method service, and establishes an inter-process communication connection between the input method service and the 3D application and between the input method service and the runtime to achieve rendering of the virtual keyboard image in three-dimensional space.

[0109] The rendering process of the virtual keyboard image can be found in Figure 6 , mainly includes the following steps:

[0110] S3021: Create a second exchange chain component and a second layer corresponding to the input method service.

[0111] S3022: Get the current position parameters of the head.

[0112] In a three-dimensional scene, since the head changes dynamically, it is necessary to obtain the current position parameters of the head in real time. The current position parameters of the head include at least the head posture and field of view. Based on the head posture and field of view data, the best display position of the virtual keyboard in the three-dimensional space can be calculated, which is usually located at a certain distance below the user's line of sight.

[0113] S3023: Determine the front position of the head according to the posture of the head, and determine the line of sight of the head according to the field of view of the head.

[0114] S3024: Determine a position at a first preset distance in front of the position and a second preset distance below the line of sight, and toward the head, as the position of the virtual keyboard in the three-dimensional space.

[0115] Taking into account the comfort in different interaction modes, the optimal position of the virtual keyboard is generally set at 0.3m in front of the head and 0.35m below the line of sight, facing the human head.

[0116] S3025: Get the current position parameters of the virtual camera.

[0117] Among them, the virtual camera is a camera created by the head-mounted display device in three-dimensional space to simulate the user's eyes. It is used to represent the position of the human eyes. Therefore, when rendering the virtual keyboard image, it is necessary to obtain the current virtual camera's position and field of view angle and other position parameters in real time.

[0118] S3026: Generate image attributes of the keyboard image according to the current position parameters of the virtual camera and the position of the virtual keyboard in the three-dimensional space.

[0119] Generally, the size of the virtual keyboard corresponding to the input method service is fixed. By combining the current position and field of view of the virtual camera and the position of the virtual keyboard in three-dimensional space, image attributes such as the texture and material of the virtual keyboard can be generated.

[0120] S3027: Setting the image attributes of the virtual keyboard to the second interactive chain component, rendering the virtual keyboard image to the second layer through the second exchange chain component, and submitting it to the runtime.

[0121] Through the handle of the second swap chain component, the second swap chain component corresponding to the input method service can be obtained, and the image attribute of the virtual keyboard is set to the second swap chain component to obtain the virtual keyboard image, which is submitted to the runtime after rendering is completed on the second layer.

[0122] S303: According to the current position parameters of the head and the current position parameters of the virtual camera, control the current depth of the second layer to be smaller than the minimum depth of the first layer, so that the second layer is not blocked by the first layer.

[0123] At the system level, the depth range of the layers corresponding to the application and the input method service is usually defined, wherein the maximum depth of the second layer corresponding to the input method service is less than the minimum depth of the first layer corresponding to the application. At this time, the specific depth of the second layer within the preset depth range can be determined in combination with the position information of the virtual keyboard in three-dimensional space determined based on the current position parameters of the head. Since the second exchange chain component will also submit the current position information of the virtual keyboard in three-dimensional space, the position and field of view of the virtual camera and other parameters to the runtime when submitting the second layer, the current depth of the virtual keyboard can be determined according to the current position and field of view of the virtual camera and the position of the virtual keyboard in three-dimensional space, and the current depth of the second layer can be controlled to be less than the minimum depth of the first layer, thereby realizing the superposition of the second layer and the first layer, and displaying them on the screen after prediction and anti-distortion processing. In specific implementation, the graphics synthesizer (Compositor) is called to superimpose the second layer on the first layer, so that the virtual keyboard image is always displayed above the application screen, thereby solving the problem of virtual keyboard occlusion.

[0124] When the first layer and the second layer are compositely displayed, the embodiment of the present application determines the optimal position information of the virtual keyboard in the three-dimensional space through the posture of the head and the field of view, thereby ensuring that the virtual keyboard is compatible with the screen of the 3D application and improving the display effect of the virtual keyboard. In addition, by considering the position and field of view of the virtual camera, the display of the virtual keyboard can be adapted to the user's perspective, further improving the display effect of the virtual keyboard.

[0125] Optionally, in some embodiments, when the first layer and the second layer are synthesized in a graphics synthesizer, the second layer can be superimposed on the first layer in a semi-transparent manner (such as 90% transparency), so as to avoid the virtual keyboard image being blocked by the application screen, reduce the abruptness of the virtual keyboard image, and improve the synthetic display effect.

[0126] After the virtual keyboard image is displayed, the user can input into the 3D application to achieve interaction. Figure 7 , mainly includes the following steps:

[0127] S304: In response to an input operation triggered by the 3D application, obtain a position and a direction of the interaction ray.

[0128] The user can move the interaction ray over the key range of the virtual keyboard, and when it moves to the target key, the user can trigger the input operation of the 3D application by confirming the indication. The head mounted display device responds to the input operation and obtains the current position and direction of the interaction ray.

[0129] Among them, the embodiment of the present application does not impose any restrictive requirements on the control method of the interactive rays, and can be set according to the actual interaction mode.

[0130] For example, when the interaction mode is head-controlled interaction, the user can control the movement of the interaction ray by moving the head; when the interaction mode is handle interaction, the user can control the movement of the interaction ray by moving the handle.

[0131] S305: Determine the focus position of the interaction ray on the virtual keyboard according to the position and orientation of the interaction ray and the position information of the virtual keyboard.

[0132] When inputting using the virtual keyboard, the interaction ray in the three-dimensional space will fall on the virtual keyboard in the three-dimensional space. By determining the focus position, the text that the user wants to input can be determined.

[0133] S306: Acquire the text at the focus position, and add the text to the screen of the 3D application.

[0134] After submitting the text at the focus position to the application layer, the 3D application renders the text and submits the screen after the text input to the runtime through the first swap chain component.

[0135] In the embodiment of the present application, the focus position is determined by the position and orientation of the interaction ray, so that the text at the focus position is input into the 3D application, thereby enabling the user to interact with the 3D application based on the virtual keyboard.

[0136] In some embodiments, when the user wants to end the interaction, a closing operation on the input method service is triggered. The head-mounted display device responds to the closing operation, ends the input method service, and destroys the virtual keyboard, second layer, second exchange chain component and other resources corresponding to the input method service.

[0137] In an embodiment of the present application, two layers are created, the screen of the 3D application and the image of the virtual keyboard are rendered on different layers, and the image of the virtual keyboard is always displayed above the screen of the 3D application, thereby solving the problem of the virtual keyboard being blocked. At the same time, the rendering of the 3D application screen and the rendering of the virtual keyboard image are separated through two layers to achieve system-level display of the virtual keyboard, which is convenient for later maintenance. Compared with the application layer display integrated in the 3D application, it not only reduces the development difficulty, but also improves the reliability and flexibility of the virtual keyboard.

[0138] See also Figure 8 , which is an interactive timing diagram for inputting using a virtual keyboard provided in an embodiment of the present application, mainly includes the following steps:

[0139] S1: In response to a keyboard interaction operation initiated by a user, the 3D application sends an interaction instruction to an input method service.

[0140] S2: The input method service establishes an inter-process communication connection with the runtime according to the interaction instructions.

[0141] S3: The runtime creates a corresponding second layer and a second swap chain component for the input method service, and returns a handle of the second swap chain component to the input method service.

[0142] S4: The input method service obtains current position parameters of the head and the current position parameters of the virtual camera, and renders the virtual keyboard image on the second layer through the second exchange chain component according to the current position parameters of the head and the current position parameters of the virtual camera.

[0143] S5: The 3D application submits the first layer of the screen to the runtime, and the input method service submits the second layer of the virtual keyboard image to the runtime.

[0144] S6: During operation, the second layer is superimposed on the first layer, and is displayed on the screen after being predicted and anti-distorted.

[0145] S7: In response to the input operation triggered by the user, the 3D application sends an input instruction to the input method service.

[0146] S8: The input method service obtains the position and direction of the interaction ray according to the input instruction.

[0147] S9: The input method service determines the focus position of the interaction ray on the virtual keyboard according to the position and direction of the interaction ray.

[0148] S10: The input method service obtains the text at the focus position and sends the text to the 3D application.

[0149] S11: The 3D application renders the text to the first layer where the screen is located.

[0150] S12: The 3D application submits the first layer to the runtime, and the input method service submits the second layer to the runtime.

[0151] S13: During operation, the second layer is superimposed on the first layer, and the superimposed layer is predicted and anti-distorted before being displayed on the screen.

[0152] S14: In response to the interaction end operation triggered by the user, the 3D application sends an end indication to the input method service.

[0153] S15: The input method service sends a resource destruction message to the runtime according to the end instruction and releases the virtual keyboard.

[0154] S16: The runtime releases the second layer and the second swap chain component according to the resource destruction message.

[0155] In the embodiment of the present application, the rendering of the virtual keyboard of the input method service is separated from the 3D application through the layer at runtime. During the interaction process, the focus calculation is performed according to the interaction ray to obtain the input of the virtual keyboard, and the layer of the virtual keyboard is superimposed on the layer of the 3D application through the field of view angle and position of the virtual camera, as well as the position and size of the virtual keyboard in three-dimensional space in the image synthesizer, so as to ensure that the virtual keyboard is always rendered in the upper layer of the application, thereby solving the problem of partial occlusion of the virtual keyboard. At the same time, since the user will not be exposed to the rendering process of the input method service, the virtual keyboard will not be changed, which improves the security and reliability of the virtual keyboard and ensures the integrity of the virtual keyboard function and the uniformity of the keyboard style. On the other hand, by separating the rendering of the virtual keyboard of the input method service from the rendering process of the 3D application, the developer does not need to manually adjust the position of the virtual keyboard, which reduces the difficulty of 3D application development and ensures the reliability and maintainability of the system.

[0156] like Fig. 9 As shown, this is a diagram of the superimposed display effect of the virtual keyboard image and the application screen provided in an embodiment of the present application. In this way, not only the complexity of integrating input method services in 3D applications is reduced, but also the problem of occlusion of the virtual keyboard during the interaction process is solved, the impact on user interaction is minimized, and the overall stability of the 3D application and user experience are improved.

[0157] In some embodiments, by separating the rendering process of the virtual keyboard of the input method service from the rendering process of the 3D application, a separate update of the input method service can be achieved. In specific implementation, the head mounted display device sends a keyboard resource request to the server of the input method service in response to the update operation of the input method service, and the server sends the resource package required for the input method service update to the head mounted display device according to the request, so that the head mounted display device updates the input method service according to the resource package.

[0158] Optionally, when the resource package requested by the head mounted display device is the layout resource package of the virtual keyboard, the UI resources of the input method service can be automatically updated. Specifically, during the startup of the input method service, it is detected whether the layout resource package of the virtual keyboard is updated. If it is updated, the UI controls of the virtual keyboard (including size, color, style, etc.) are updated according to the layout resource package, and the position information of the virtual keyboard in the three-dimensional space is updated. The input text is obtained according to the updated position information and fed back to the 3D application.

[0159] Optionally, when the resource package requested by the head mounted display device is a font resource package of a virtual keyboard, the font library of the input method service may be automatically updated. Specifically, after reloading the font resource package, the font library of the virtual keyboard is updated according to the font resource package when the input method service is started next time.

[0160] See also Fig.10 , which is an update sequence diagram of the input method service provided in the embodiment of the present application, mainly includes the following steps:

[0161] S1: The input method service sends a layout request to the server in response to a keyboard layout update operation.

[0162] S2: The server sends a layout resource package to the input method service in response to the layout request.

[0163] S3: The input method service updates the UI controls and position information of the virtual keyboard according to the layout resource package.

[0164] S4: The input method service generates text for the virtual keyboard according to the loaded font library.

[0165] S5: The input method service sends a font library request to the server in response to the keyboard font library update operation.

[0166] S6: The server sends a font resource package to the input method service according to the font request.

[0167] S7: When the input method service is started next time, the text of the virtual keyboard is updated according to the font resource package.

[0168] In the embodiment of the present application, since the virtual keyboard is rendered at runtime and displayed superimposed with the 3D application, it is not associated with the rendering process of the 3D application. Therefore, the UI controls and font library of the virtual keyboard can be automatically upgraded based on the user's feedback during the actual interaction process, thereby realizing flexible updating of the input method service.

[0169] Based on the same technical concept, an embodiment of the present application provides a display device for a virtual keyboard, which can implement the steps of the above-mentioned display method for a virtual keyboard and achieve the same technical effect.

[0170] See also Fig.11 The display device includes an application rendering module 1101, an input method rendering module 1102 and a synthesis module 1103, wherein:

[0171] The rendering module 1101 is used to create a first layer in response to a start-up operation of the 3D application, render the screen of the 3D application onto the first layer, and submit it to the runtime;

[0172] An input method rendering module 1102, configured to, in response to a keyboard interaction operation triggered for the 3D application, start an input method service and create a second layer, render an image of a virtual keyboard corresponding to the input method service onto the second layer and submit it to the runtime;

[0173] The synthesis module 1103 is used to control the current depth of the second layer to be less than the minimum depth of the first layer according to the current position parameters of the head and the current setting parameters of the virtual camera, so that the second layer is not blocked by the first layer; wherein the virtual camera is used to represent the position of the human eye.

[0174] Optionally, the current position data of the head includes at least the posture and field of view of the head, and the synthesis module 1103 is specifically used for:

[0175] Determining the front position of the head according to the posture of the head, and determining the line of sight of the head according to the field of view of the head;

[0176] Determine a position at a first preset distance in front of the position and a second preset distance below the line of sight, which is toward the head, as the position of the virtual keyboard in the three-dimensional space;

[0177] Determining the current depth of the virtual keyboard according to the current position parameters of the virtual camera and the position of the virtual keyboard in the three-dimensional space; wherein the current position parameters of the virtual camera at least include the position and the field of view angle of the virtual camera;

[0178] The current depth of the second layer is controlled to be smaller than the minimum depth of the first layer, so that the second layer is not blocked by the first layer.

[0179] Optionally, the display device further includes an input module 1104, which is used to:

[0180] In response to an input operation triggered by the 3D application, obtaining a position and a direction of an interaction ray;

[0181] Determining a focus position of the interaction ray on the virtual keyboard according to the position and orientation of the interaction ray and the position information of the virtual keyboard;

[0182] The text at the focus position is acquired, and the text is added to the screen of the 3D application.

[0183] Optionally, the display device further includes an updating module 1105, which is used to:

[0184] In response to the update operation of the input method service, sending a keyboard resource request to a server;

[0185] receiving a resource package returned by the server according to the keyboard resource request;

[0186] The virtual keyboard updated according to the resource pack is displayed on the second layer.

[0187] Optionally, the updating module 1105 is specifically used for:

[0188] Detecting whether the layout resource package of the virtual keyboard is updated;

[0189] If yes, updating the UI controls of the virtual keyboard according to the layout resource package, and the position information of the virtual keyboard in the three-dimensional space determined according to the first position parameter;

[0190] The input text is acquired according to the updated position information and fed back to the 3D application.

[0191] Optionally, the updating module 1105 is specifically used for:

[0192] When the input method service is started next time, the font library of the virtual keyboard is updated according to the font library resource package.

[0193] Optionally, the synthesis module 1103 is specifically used for:

[0194] The graphics synthesizer is called to overlay the second layer on the first layer in a semi-transparent manner for display.

[0195] Optionally, the application rendering module 1101 is specifically used for:

[0196] Creating an interface instance and a session for communicating with the runtime, wherein the session is used to manage the life cycle of the 3D application;

[0197] Creating an interaction profile of the 3D application according to the hardware of the device itself, and determining an interaction mode of the 3D application and an interaction button in the interaction profile according to a preset usage scenario; the interaction button is used to receive an interaction operation of the user;

[0198] Create a first swap chain component and a first layer according to the graphics interface and rendering parameters;

[0199] Acquire a rendering carrier of the 3D application through the first exchange chain component, and render a picture of the 3D application onto the carrier to obtain a picture texture;

[0200] The screen texture is rendered onto the first layer for submission to the runtime.

[0201] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0202] After introducing the method and apparatus for displaying a virtual keyboard according to an exemplary embodiment of the present application, next, a head mounted display device according to another exemplary embodiment of the present application is introduced.

[0203] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0204] Based on the same inventive concept as the above method embodiment, the structure of the head mounted display device provided in the embodiment of the present application can be as follows: Fig.12 As shown, it includes a processor 1201, a memory 1202 and a display screen 1203;

[0205] The display screen 1203 is used to display images;

[0206] The memory 1202 stores a computer program, and the processor 1201 executes the steps of any one of the methods for displaying a virtual keyboard in the above embodiments according to the computer program.

[0207] In the embodiment of the present application, the memory 1202 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, and programs required to run the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc. The memory 1202 may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory may also be a non-volatile memory (non-volatile memory), such as a read-only memory, a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1202 may be any other medium that can be used to carry or store a desired computer program in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1102 may be a combination of the above memories.

[0208] The processor 1201 may include one or more central processing units (CPU), GPU or a digital processing unit, etc.

[0209] In the embodiment of the present application, the specific connection medium between the display screen 1203, the memory 1202 and the processor 1201 is not limited. In the embodiment of the present application, the bus 1204 between the display screen 1203, the memory 1202 and the processor 1201 is Fig.12 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1204 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.12 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0210] It should be noted that Fig.12 It is only the equipment necessary for the head-mounted display device to implement the display method of the virtual keyboard in the embodiment of the present application. Not shown, the head-mounted display device may also include conventional hardware of the display device such as a multi-eye camera, an IMU, a speaker, a microphone, a power supply, buttons, a handle, etc.

[0211] The embodiment of the present application further provides a computer-readable storage medium for storing some instructions, which, when executed, can complete the steps of any one of the virtual keyboard display methods in the aforementioned embodiments.

[0212] An embodiment of the present application further provides a computer program product for storing a computer program, wherein the computer program is used to execute the steps of any one of the methods for displaying a virtual keyboard in the aforementioned embodiments.

[0213] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0214] The present application is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram, and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart or multiple flows and / or one box or multiple boxes in the block diagram.

[0215] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0216] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0217] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for displaying a virtual keyboard, characterized in that: include: In response to a start-up operation of the 3D application, a first layer is created, a screen of the 3D application is rendered onto the first layer, and the screen is submitted to the runtime; In response to a keyboard interaction operation triggered for the 3D application, starting an input method service and creating a second layer, rendering an image of a virtual keyboard corresponding to the input method service onto the second layer and submitting the image to the runtime; According to the current position parameters of the head and the current setting parameters of the virtual camera, the current depth of the second layer is controlled to be less than the minimum depth of the first layer so that the second layer is not blocked by the first layer; wherein the virtual camera is used to represent the position of the human eye.

2. The method according to claim 1, characterized in that The current position data of the head includes at least the posture and field of view of the head, and the controlling the current depth of the second layer to be less than the minimum depth of the first layer according to the current position parameters and the current position parameters of the virtual camera so that the second layer is not blocked by the first layer includes: Determining the front position of the head according to the posture of the head, and determining the line of sight of the head according to the field of view of the head; Determine a position at a first preset distance in front of the position and a second preset distance below the line of sight, which is toward the head, as the position of the virtual keyboard in the three-dimensional space; Determining the current depth of the virtual keyboard according to the current position parameters of the virtual camera and the position of the virtual keyboard in the three-dimensional space; wherein the current position parameters of the virtual camera at least include the position and the field of view angle of the virtual camera; The current depth of the second layer is controlled to be smaller than the minimum depth of the first layer, so that the second layer is not blocked by the first layer.

3. The method according to claim 1, characterized in that After controlling the current depth of the second layer to be less than the minimum depth of the first layer, the method further includes: In response to an input operation triggered by the 3D application, obtaining a position and a direction of an interaction ray; Determining a focus position of the interaction ray on the virtual keyboard according to the position and orientation of the interaction ray and the position information of the virtual keyboard; The text at the focus position is acquired, and the text is added to the screen of the 3D application.

4. The method according to claim 1, characterized in that The method further comprises: In response to the update operation of the input method service, sending a keyboard resource request to a server; receiving a resource package returned by the server according to the keyboard resource request; The virtual keyboard updated according to the resource pack is displayed on the second layer.

5. The method according to claim 4, characterized in that When the resource package is a layout resource package of the virtual keyboard, the method further includes: Detecting whether the layout resource package of the virtual keyboard is updated; If yes, updating the UI controls of the virtual keyboard according to the layout resource package, and the position information of the virtual keyboard in the three-dimensional space determined according to the first position parameter; The input text is acquired according to the updated position information and fed back to the 3D application.

6. The method according to claim 4, characterized in that When the resource package is a font resource package of the virtual keyboard, the method further includes: When the input method service is started next time, the font library of the virtual keyboard is updated according to the font library resource package.

7. The method according to any one of claims 1 to 6, characterized in that The controlling the current depth of the second layer to be smaller than the minimum depth of the first layer so that the second layer is not blocked by the first layer includes: The graphics synthesizer is called to overlay the second layer on the first layer in a semi-transparent manner for display.

8. The method according to any one of claims 1 to 6, characterized in that The step of creating a first layer in response to a start-up operation of the 3D application, rendering a screen of the 3D application onto the first layer and submitting the screen to the runtime includes: Creating an interface instance and a session for communicating with the runtime, wherein the session is used to manage the life cycle of the 3D application; Creating an interaction profile of the 3D application according to the hardware of the device itself, and determining an interaction mode of the 3D application and an interaction button in the interaction profile according to a preset usage scenario; the interaction button is used to receive an interaction operation of the user; Create a first swap chain component and a first layer according to the graphics interface and rendering parameters; Acquire a rendering carrier of the 3D application through the first exchange chain component, and render a picture of the 3D application onto the carrier to obtain a picture texture; The screen texture is rendered onto the first layer for submission to the runtime.

9. A head mounted display device, characterized in that: It includes a processor, a memory and a display screen, wherein the display screen, the memory and the processor are connected via a bus; The display screen is used to display the layer; The memory stores a computer program, and the processor performs the following operations according to the computer program: In response to a start-up operation of the 3D application, a first layer is created, a screen of the 3D application is rendered onto the first layer, and the screen is submitted to the runtime; In response to a keyboard interaction operation triggered for the 3D application, starting an input method service and creating a second layer, rendering an image of a virtual keyboard corresponding to the input method service onto the second layer and submitting the image to the runtime; According to the current position parameters of the head and the current position parameters of the virtual camera, the current depth of the second layer is controlled to be less than the minimum depth of the first layer so that the second layer is not blocked by the first layer; wherein the virtual camera is used to represent the position of the human eye.

10. The head mounted display device according to claim 9, wherein: The processor is further configured to execute: In response to an input operation triggered by the 3D application, obtaining a position and a direction of an interaction ray; Determining a focus position of the interaction ray on the virtual keyboard according to the position and orientation of the interaction ray and the position information of the virtual keyboard; The text at the focus position is acquired, and the text is added to the screen of the 3D application.

Images