Text input method and device, computer equipment and storage medium

Abstract

The invention relates to a text input method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: in response to an input triggering operation for a virtual program interface, obtaining a real shooting picture shot by a camera; switching the current display picture from the virtual program interface to a real shooting picture, wherein a virtual input interface for input interaction is integrated in the real shooting picture; in response to an input interaction operation for the virtual input interface, obtaining a text input in the virtual input interface based on the first external input device; and switching the current display picture from the real shooting picture to the virtual program interface, and displaying the text to a corresponding text input position in the virtual program interface. According to the embodiment of the invention, the real shot picture and the virtual program interface are separately displayed, and the interactive input interface is displayed in the real shot picture in an overlapping manner as a transmission medium for external character input, so that the character input efficiency is improved.

Description

Technical Field

[0001] This application relates to the technical fields of virtual reality and augmented reality, and particularly to a text input method, apparatus, computer device, storage medium, and computer program product. Background Art

[0002] With the development of computer image technology, technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) have gradually been applied to people's lives. Users enjoy a mixed reality experience by wearing a head-mounted display device.

[0003] The head-mounted display device uses a semi-transparent display medium and a screen with a relatively small field of view (FOV) to display a picture, so that the user can see the physical input device (such as a mobile phone, keyboard, touchpad, etc.) outside the following area. However, this solution significantly sacrifices the FOV of the display picture and reduces the display range, affecting the immersion of the mixed reality device. If the screen FOV is to be increased (generally greater than 90°), a closed structure can be adopted, making it impossible for the user's naked eyes to see the outside world, resulting in the user being unable to find the external input device in the immersive content picture, or being unable to see the content picture after switching to the mixed reality mode, making the input result unknown. Summary of the Invention

[0004] Based on this, it is necessary to provide an accurate text input method, apparatus, computer device, computer-readable storage medium, and computer program product for the above technical problems.

[0005] In a first aspect, this application provides a text input method. It is applied to a head-mounted display device including a camera; the method includes:

[0006] In response to an input trigger operation for a virtual program interface, obtain a real captured picture taken by the camera;

[0007] When it is determined that the head-mounted display device is connected to a first external input device, switch the current display picture from the virtual program interface to the real captured picture, and a virtual input interface for input interaction is integrated in the real captured picture;

[0008] In response to an input interaction operation for the virtual input interface, obtain the text input in the virtual input interface based on the first external input device;

[0009] Switch the current display picture from the real captured picture back to the virtual program interface, and screen the text to the corresponding text input position in the virtual program interface.

[0010] In one embodiment, determining that the head-mounted display device is connected to a first external input device includes:

[0011] Scanning the list of information of removable devices connected to the head-mounted display device;

[0012] When the list of information of removable devices includes the device information of the first external input device, determining that the head-mounted display device is connected to the first external input device.

[0013] In one embodiment, the process of displaying the virtual input interface in the real shooting screen includes:

[0014] Obtaining the optional display positions where the virtual input interface does not block the external input device when the virtual input interface is displayed in the real shooting screen;

[0015] Based on the optional display positions, displaying the virtual input interface in the real shooting screen.

[0016] In one embodiment, the relative position between the optional display position and the real shooting screen is fixed and unchanged.

[0017] In one embodiment, the virtual input interface includes a first interaction element for text input and a second interaction element for confirming or canceling the input; the second interaction element is triggered by a physical button on the first external input device or the second external input device; wherein, the second external input device is connected to the head-mounted display device and is used to control the virtual program, and the triggering mode of the physical button is single-button triggering or combined-button triggering.

[0018] In one embodiment, the first interaction element is a text input box; obtaining the text input in the virtual input interface based on the first external input device includes:

[0019] Obtaining the text input in the text input box based on the first external input device and synchronously displayed in the text input box;

[0020] After the text input is completed, caching the text for uploading to the virtual program interface and clearing the text synchronously displayed in the text input box.

[0021] In a second aspect, the present application also provides a text input device. Applied to a head-mounted display device including a camera; the device includes:

[0022] A first response module, configured to obtain the real shooting screen captured by the camera in response to an input trigger operation for the virtual program interface;

[0023] The first switching module is configured to switch the current display screen from the virtual program interface to the real-world captured screen when it is determined that the head-mounted device is connected to a first external input device. The real-world captured screen integrates a virtual input interface for input interaction;

[0024] The second response module is configured to obtain the text input in the virtual input interface based on the first external input device in response to an input interaction operation on the virtual input interface;

[0025] The second switching module is configured to switch the current display screen back from the real-world captured screen to the virtual program interface and paste the text onto the corresponding text input position in the virtual program interface.

[0026] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0027] In response to an input triggering operation on the virtual program interface, obtain a real-world captured screen captured by a camera;

[0028] When it is determined that the head-mounted device is connected to a first external input device, switch the current display screen from the virtual program interface to the real-world captured screen. The real-world captured screen integrates a virtual input interface for input interaction;

[0029] In response to an input interaction operation on the virtual input interface, obtain the text input in the virtual input interface based on the first external input device;

[0030] Switch the current display screen back from the real-world captured screen to the virtual program interface and paste the text onto the corresponding text input position in the virtual program interface.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0032] In response to an input triggering operation on the virtual program interface, obtain a real-world captured screen captured by a camera;

[0033] When it is determined that the head-mounted device is connected to a first external input device, switch the current display screen from the virtual program interface to the real-world captured screen, and the real-world captured screen integrates a virtual input interface for input interaction;

[0034] In response to an input interaction operation on the virtual input interface, obtain the text input in the virtual input interface based on the first external input device;

[0035] Switch the current display screen from the real captured screen back to the virtual program interface, and upload the text to the corresponding text input position in the virtual program interface.

[0036] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0037] In response to an input trigger operation for the virtual program interface, obtain a real captured screen captured by a camera;

[0038] When it is determined that the head-mounted display device is connected to a first external input device, switch the current display screen from the virtual program interface to the real captured screen, and a virtual input interface for input interaction is integrated in the real captured screen;

[0039] In response to an input interaction operation for the virtual input interface, obtain the text input in the virtual input interface based on the first external input device;

[0040] Switch the current display screen from the real captured screen back to the virtual program interface, and upload the text to the corresponding text input position in the virtual program interface.

[0041] The above text input method, device, computer device, storage medium, and computer program product, in response to an input trigger operation for the virtual program interface, obtain a real captured screen captured by a camera; when it is determined that the head-mounted display device is connected to a first external input device, switch the current display screen from the virtual program interface to the real captured screen, and a virtual input interface for input interaction is integrated in the real captured screen; in response to an input interaction operation for the virtual input interface, obtain the text input in the virtual input interface based on the first external input device; switch the current display screen from the real captured screen back to the virtual program interface, and upload the text to the corresponding text input position in the virtual program interface. By separately displaying the real captured screen and the virtual program interface, and superimposing a virtual interactive input interface on the real captured screen as a transmission medium for external text input, the text input efficiency is improved without changing the field of view angle size of the head-mounted display device. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the usage effect of a mixed reality head-mounted display device in an existing embodiment;

[0043] Figure 2 It is a schematic diagram of the usage effect of a mixed reality head-mounted display device in another existing embodiment;

[0044] Figure 3 It is a schematic diagram of the application environment of the text input method in an embodiment;

[0045] Figure 4 It is a schematic flowchart of a text input method in an embodiment;

[0046] Figure 5 It is a schematic diagram of the current display screen switching process in an embodiment;

[0047] Figure 6 It is a schematic diagram of the current display screen switching process in another embodiment;

[0048] Figure 7 It is a schematic flowchart of a text input method in another embodiment;

[0049] Figure 8 It is a schematic diagram of the position of an optional display position in an embodiment;

[0050] Figure 9a It is a schematic diagram of the position of a virtual input interface in an embodiment;

[0051] Figure 9b It is a schematic diagram of the position of a virtual input interface in another embodiment;

[0052] Figure 10 It is a schematic diagram of a real shooting screen in an embodiment;

[0053] Figure 11 It is a schematic flowchart of a text input method for a mixed reality head-mounted device in an embodiment;

[0054] Figure 12 It is a structural block diagram of a text input device in an embodiment;

[0055] Figure 13 It is an internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0056] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] A head-mounted display device with virtual reality, augmented reality, and mixed reality functions that has a video see-through (VST) function can use a screen mirroring function to virtualize a display screen in front of the user's eyes to achieve the function of a virtual PC. Currently, when interacting with the head-mounted display device, most interactions are carried out using simple interaction methods such as gestures, head movements, voice, eye tracking, and touch (using a touch module set in the head-mounted display device for touch). However, these interaction methods are difficult to meet the efficient and complex interaction requirements such as graphical interface positioning and text input in scenarios such as mobile office, entertainment, and games.

[0058] In order to meet the efficient and complex interaction requirements such as graphical interface positioning and text input in scenarios such as mobile office, entertainment, and games, the prior art has proposed using the means of near-eye display combined with visual recognition to implement a "virtual keyboard" suspended in the air or attached to the surface of an object / human limb. However, this requires the introduction of a visual recognition function, which not only causes high system power consumption but also leads to interaction failures due to the low accuracy of visual recognition. Moreover, when interacting, the line of sight needs to be aligned with the "virtual keyboard" to select and use the "virtual keyboard" for text input, resulting in low interaction efficiency.

[0059] In addition, the head-mounted display device can be connected to a keyboard, mouse, touchpad handle, and other devices through a wired / wireless connection. Then, the head-mounted display device uses the screen mirroring function to virtualize a display screen in front of the user's eyes, and then combines the connected devices such as the handle, touchpad, keyboard, and mouse to achieve interaction. However, in existing head-mounted display devices, in order to increase the screen field of view, a closed structure is adopted, making it impossible for the user's naked eyes to see the outside world. Then, the built-in camera of the device captures the outside image, which is superimposed and rendered with the virtual image and then output to the display screen to achieve the mixed reality effect. However, the mixed reality mode requires support from content developers. If the content itself does not support the mixed reality mode and only works in the virtual reality mode, it will cause the user to be unable to find the external input device in the immersive content screen, as Figure 1 shown, or unable to see the content screen after switching to the mixed reality mode, resulting in an unknown input result, as Figure 2 shown.

[0060] In order to meet the efficient and complex interaction requirements in scenarios such as mobile office, entertainment, and games, improve the interaction efficiency, and improve the portability, the embodiments of the present application provide a text input method, device, computer device, and storage medium, and the specific content can be seen in the following text.

[0061] The text input method provided by the embodiments of the present application can be applied to, for example Figure 3In the application environment shown. It includes a head-mounted device 11 with a camera and at least one external input device 12. The head-mounted device 11 and the external input device 12 can be connected by a wired / wireless connection method. For example, they can be connected through an external input cable of the head-mounted device 11 to the external input device 12. The head-mounted device 11 and the external input device 12 can also be interconnected based on a wireless communication network. Among them, the communication network can be a local area network or a wide area network relayed through a relay device. When the communication network is a local area network, for example, the communication network can be a wifi hotspot network, a zigbee network, or a short-range communication network such as a near field communication (NFC) network. When the communication network is a wide area network, for example, the communication network can be a 3rd-generation wireless telephone technology (3G) network, the 4th generation mobile communication technology (4G) network, the 5th-generation mobile communication technology (5G) network, a future evolved public land mobile network (PLMN), or the Internet, etc. It can be understood that the communication network and service scenario described in the embodiments of this application are for more clearly explaining the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application.

[0062] In one embodiment, as Figure 4 shown, a text input method is provided. Taking the head-mounted device 11 in Figure 3 as an example, the method includes the following steps:

[0063] Step 402, in response to an input trigger operation for the virtual program interface, obtain a real captured image captured by the camera;

[0064] Among them, the virtual program interface refers to a display interface that displays content images in a preset mode on a virtual screen projected by the head-mounted device. For example, a game interface, a work interface, etc. The virtual program interface includes a third interaction element for text input. For example, when the virtual program interface is a game login interface, the third interaction element is an account and password input box; also, for example, when the virtual program interface is a work conference interface, the third interaction element is a dialog box. The virtual program interface also includes a fourth interaction element for determining input and canceling input. Both the third interaction element and the fourth interaction element can be triggered by physical buttons on the second external input device of the head-mounted device.

[0065] The input trigger operation is triggered by a second external input device connected to the head-mounted display device. As Figure 5 shown in the login interface, which includes an account and password input box and a soft button for canceling the login. The second external input device is a handle. The user moves the cursor through the buttons on the handle or by laser, and clicks on the account and password input box or the soft button for canceling the login in the login interface to trigger the input operation. In one embodiment, the second external input device can also be a mobile terminal. A connection is established between the mobile terminal and the head-mounted display device, which can control the virtual program. The user triggers an operation on the mobile terminal, thereby triggering an input operation in the virtual program interface.

[0066] The camera is integrated on the head-mounted display device and is used to capture the real shooting images of the real environment where the head-mounted display device is located. The orientation of the camera can be fixed or variable. Usually, regardless of whether the orientation of the camera is fixed or variable, it can capture the real images of the visual blind area of the user due to wearing the head-mounted display device. It can be understood that the external connection devices of the head-mounted display device, such as external input devices, can be seen through the real shooting images. By superimposing and rendering the real shooting images obtained by the camera with the content images provided by the virtual program and outputting them to the display screen, the effect of mixed reality can be achieved.

[0067] Step 404, when it is determined that the head-mounted display device is connected to a first external input device, switch the current display screen from the virtual program interface to the real shooting screen, and the real shooting screen is integrated with a virtual input interface for input interaction;

[0068] Among them, the first external input device refers to other devices that can achieve text input relative to the head-mounted display device, such as a keyboard, a voice input device, etc. The first external input device is not an input device integrated on the head-mounted display device. The head-mounted display device can achieve information input by connecting the first external input device. The current display screen refers to the screen displayed on the virtual screen projected by the head-mounted display device. After the user wears and turns on the head-mounted display device, the virtual screen projected by the head-mounted display device displays the virtual program interface, that is, the content image of the virtual program.

[0069] The real shooting screen is rendered from the real-time images of the real environment captured by the camera. When switching the current display interface from the virtual program interface to the real shooting screen, the entire virtual program interface displayed on the virtual screen can be switched to the real shooting screen, as Figure 5 shown. It is also possible to divide the display area of the virtual screen and switch the virtual program interface of a part of the display area to the real shooting screen, as Figure 6As shown. In addition, when switching the virtual program interface of a partial display area, it can be directly scaled proportionally. At this time, only a partial picture of the virtual program interface is displayed on the virtual screen. If you want to display it comprehensively, a part of the display area of the virtual screen can be used as the display area of the virtual program interface, and the other part can be used as the display area of the real shooting picture. After reducing the virtual program interface and the real shooting picture to a certain extent, they are completely displayed in the corresponding display areas.

[0070] It can be understood that the external input device in the real environment can be seen through the real shooting picture. Then, a virtual input interface for input interaction is integrated in the real shooting picture. Through the virtual input interface, the input operations of the user on the external input device can be visualized. The input interaction interface at least includes a text input box, an indication cursor, and several soft buttons.

[0071] Step 406, in response to the input interaction operation on the virtual input interface, obtain the text input in the virtual input interface based on the first external input device;

[0072] The input interaction operation on the virtual input interface refers to the input interaction operation triggered by the user on the external input device. For example, the user inputs text through an external keyboard. In response to the input interaction operation on the virtual input interface, obtain the text input in the virtual input interface. For example, the user triggers an interaction operation through an external input device and inputs "Let nature take its course". While inputting, the text "Let nature take its course" is input and displayed in the virtual input interface.

[0073] It should be noted that the virtual input interface also includes an indication cursor, which can delete, line break, modify, etc. the text already input in the virtual input interface in response to the input interaction operation on the virtual input interface. When responding to the input interaction operation on the virtual input interface, the third interaction element for text input in the virtual program interface cannot obtain the input text.

[0074] Step 408, switch the current display screen back to the virtual program interface from the real shooting picture, and screen the text to the corresponding text input position in the virtual program interface.

[0075] Among them, the corresponding text input position in the virtual program interface refers to the third interaction element for text input in the virtual program interface. After obtaining the text input in the virtual input interface, switch the current display screen back to the virtual program interface from the real shooting picture, and screen the text input in the virtual input interface to the corresponding text input position in the virtual program interface.

[0076] Among them, the execution steps of switching the current display screen from the real shooting screen back to the virtual program interface in this step correspond to the execution steps of switching the current display screen from the virtual program interface to the real shooting screen in step 404. If in step 404, all the virtual program interfaces displayed on the virtual screen are switched to the real shooting screen, then in this step, the current display screen of the virtual screen will be switched back to the virtual program interface in its entirety. If only some of the virtual program interfaces are switched in step 404, then in this step, the real shooting screen will be switched back to the corresponding partial virtual program interfaces.

[0077] In addition, the screen switching operation in this step is triggered by the user through an external input device. By responding to the input interaction operation for the virtual input interface, the input control instruction of the user is obtained and the screen is switched. In one embodiment, if the input control instruction indicates the end of input, the current display screen is switched from the real shooting screen back to the virtual program interface, and the text is uploaded to the corresponding text input position in the virtual program interface. If the input control instruction indicates input cancellation or input, the virtual program interface is switched and text input is implemented through the soft keyboard of the virtual program interface.

[0078] In the method provided in the above embodiment, in response to the input trigger operation for the virtual program interface, the real shooting screen captured by the camera is obtained; the current display screen is switched from the virtual program interface to the real shooting screen, and a virtual input interface for input interaction is integrated in the real shooting screen; in response to the input interaction operation for the virtual input interface, the text input in the virtual input interface is obtained; the current display screen is switched from the real shooting screen back to the virtual program interface, and the text is uploaded to the corresponding text input position in the virtual program interface. By separately displaying the real shooting screen and the virtual program interface, and superimposing a virtual interactive input interface on the real shooting screen as the transmission medium for external text input, the text input efficiency is improved without changing the field of view angle size of the head-mounted display device.

[0079] In one of the embodiments, determining that the head-mounted display device is connected to a first external input device includes:

[0080] By scanning the list of movable device information connected to the head-mounted display device;

[0081] In the case that the device information of the first external input device is included in the list of movable device information, it is determined that the head-mounted display device is connected to the first external input device.

[0082] Among them, the movable device refers to a movable device connected to the head-mounted display device other than the head-mounted display device itself. The movable device can be connected to the head-mounted display device or not connected to the head-mounted display device. The list of movable device information includes the uniqueness information of all movable devices already connected to the head-mounted display device. The device information of the external input device includes a Product Identification (PID), a Vendor Identification (VID), and a Device Identification (DID). The connection methods between the first external input device and the head-mounted display device include wired connection and wireless connection. For example, the first external input device can be a Bluetooth keyboard, a USB keyboard, a touch screen connected by Bluetooth, a mobile terminal, etc.

[0083] When the head-mounted display device is connected to the first external input device, it can obtain the device information of the first external input device. In this embodiment, the scanned device information can be all the device information obtained by the head-mounted display device or part of the device information. When scanning the device information of all external devices connected to the head-mounted display device, if the first external input device is identified from the scanned device information, text input can be achieved through the first external input device on the virtual input interface.

[0084] In the method provided in the above embodiment, by inputting text through the first external input device on the virtual input interface, combined with the integrated virtual input interface, the text input efficiency can be effectively improved.

[0085] In one of the embodiments, as Figure 7 shown, the display process of the virtual input interface in the real shooting screen includes:

[0086] Step 702, obtain the optional display positions where the virtual input interface does not block the external input device when the virtual input interface is displayed in the real shooting screen;

[0087] Among them, the optional display position refers to a position in the area where the external input device is not displayed in the real shooting screen, such as Figure 8 the position 810 located in area A in. It can be understood that the shooting range of the real shooting screen may change as the user moves. Therefore, the position of the external input device in the real shooting screen is not fixed, and the specific position of the optional display position in the real shooting screen may change.

[0088] Step 704, based on the optional display position, display the virtual input interface in the real shooting screen.

[0089] According to the setting conditions of the optional display position, displaying the virtual input interface at the optional display position will not block the external input device. Therefore, the virtual input interface is displayed at the optional display position where the shooting screen is shown. In fact, according to the condition of not blocking the external input device, multiple optional display positions can be determined in the actual shooting screen, and any one of the optional display positions can be selected for display when the virtual input interface is displayed. In addition, in the current frame of the actual shooting screen, the relative position of the optional display position with respect to the entire current frame of the actual shooting screen may change or remain unchanged compared to the relative position in the previous frame of the actual shooting screen.

[0090] In the method provided by the above embodiment, the display position of the virtual input interface in the actual shooting screen is determined by the position of the external input device, which can ensure the complete presentation of the external input device in the actual shooting screen, enabling the user to see the external input device completely in the actual shooting screen when using the external input device, and improving the input efficiency of the text.

[0091] In one of the embodiments, the relative position of the optional display position and the actual shooting screen is fixed and unchanged.

[0092] The relative position of the optional display position and the actual shooting screen refers to the position of the optional display position in the screen coordinate system of the actual shooting screen. Combining with the user's habit of using the external input device, a fixed position that will not block the external input device during use can be preset in the actual shooting screen as the optional display area to display the virtual input interface. For example, when the external input device is a keyboard, the user usually uses the keyboard in a posture facing the keyboard directly. Therefore, in the actual shooting screen when the user uses the external input device to input text, the external input device is in the center position of the screen, then the upper left corner position of any frame of the actual shooting screen can be selected as the optional display area to display the virtual input interface. As Figure 9a and Figure 9b shown, the position of the virtual input interface in the actual shooting screen remains unchanged.

[0093] In the method provided by the above embodiment, keeping the relative position of the optional display position and the actual shooting screen fixed and unchanged makes the virtual input interface always fixed at the specified position of the screen in the actual shooting screen, so as to ensure that when the user is looking for the external input device, the virtual input interface will not be lost due to the user's movement, and to ensure the smooth progress of text input.

[0094] In one of the embodiments, as Figure 10As shown in the figure, the virtual input interface includes a first interaction element 101 for text input and a second interaction element 102 for confirming or canceling the input; the second interaction element 102 is triggered by a physical button on the first external input device or the second external input device; wherein, the second external input device is connected to the head-mounted display device and is used to control the virtual program, and the triggering mode of the physical button is single-button triggering or combined-button triggering.

[0095] Among them, the first interaction element is triggered by a physical button on the first external input device; for example, the first external input device is a keyboard, and the first interaction element inputs text in response to the user's input operation on the keyboard. The second interaction element can be triggered by a physical button on the first external input device. For example, the first external input device is a keyboard, and the second interaction element is triggered by a specific button or a combination of buttons on the keyboard; the second interaction element can also be triggered by a button on the second external input device. For example, the second external input device is a gamepad, and the second interaction element is triggered by a specific button or a combination of buttons on the gamepad in the form of a gamepad ray.

[0096] In the method provided in the above embodiment, text input is implemented through the first interaction element, and the text input is determined or canceled through the second interaction element, which improves the content of the virtual input interface and better controls the text input.

[0097] In one of the embodiments, the first interaction element is a text input box; obtaining the text input in the virtual input interface based on the first external input device includes:

[0098] Obtaining the text input in the text input box based on the first external input device and synchronously displayed in the text input box;

[0099] After the text input is completed, the text is cached for being sent to the virtual program interface, and the text synchronously displayed in the text input box is cleared.

[0100] When the first external input device inputs in the text input box, the input text is synchronously displayed in the text input box. After the user determines that the text input is completed through the second interaction element, the text is cached for being sent to the virtual program interface, and the text synchronously displayed in the text input box is cleared. Among them, the caching of the text can be completed by a specific button or a combination of buttons on the first external input device. For example, the user presses Ctrl+Shift+X on the keyboard to cut the text in the text input box, then clears the text in the text input box, and after switching the current display screen from the real-time shooting screen back to the virtual program interface, pastes the cut text into the third interaction element in the virtual program interface.

[0101] In the method provided in the above embodiments, the virtual input interface is used as a transmission medium for external text input, and text input for the virtual program interface is realized through the virtual input interface, which can effectively improve the text input efficiency.

[0102] In one embodiment, a text input method for a mixed reality headset device is provided. As Figure 11 shown, the method includes:

[0103] Step 1102: In response to an input trigger operation for the virtual program interface, obtain a real captured image captured by a camera.

[0104] Step 1104: Switch the current display screen from the virtual program interface to the real captured image. A virtual input interface for input interaction is integrated at a preset fixed position in the real captured image. The virtual input interface includes a first interaction element for text input and a second interaction element for confirming or canceling the input. The second interaction element is triggered by a physical button on a first external input device or a second external input device. Among them, the second external input device is connected to the headset device and is used to control the virtual program, and the triggering method of the physical button is single-button triggering or combined-button triggering.

[0105] Step 1106: When it is determined that the mixed reality headset device is connected to the first external input device by scanning the device information of the first external input device, obtain the text input in the text input box based on the first external input device and synchronously displayed in the text input box.

[0106] Step 1108: After the text input is completed, cache the text and clear the text synchronously displayed in the text input box.

[0107] Step 1110: Switch the current display screen back from the real captured image to the virtual program interface, and upload the cached text to the corresponding text input position in the virtual program interface.

[0108] In the method provided in the above embodiments, by superimposing and displaying a virtual input interface in the real captured image as a transmission medium for external text input, a solution that takes into account both the immersive experience (large FOV display full-closed structure) and the text input efficiency is provided.

[0109] It should be understood that although the steps in the flowcharts involved in the above embodiments are sequentially shown according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0110] Based on the same inventive concept, an embodiment of the present application also provides a text input device for implementing the above-mentioned text input method. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the text input device provided below can refer to the limitations on the text input method in the above, and will not be repeated here.

[0111] In one embodiment, as Figure 12 shown, a text input device is provided, which is applied to a head-mounted device including a camera, and includes: a first response module 1201, a first switching module 1202, a second response module 1203, and a second switching module 1204, where:

[0112] The first response module 1201 is configured to obtain a real captured image captured by the camera in response to an input trigger operation on the virtual program interface;

[0113] The first switching module 1202 is configured to switch the current display screen from the virtual program interface to the real captured image when it is determined that the head-mounted device is connected to a first external input device, and a virtual input interface for input interaction is integrated in the real captured image;

[0114] The second response module 1203 is configured to obtain the text input in the virtual input interface based on the first external input device in response to an input interaction operation on the virtual input interface;

[0115] The second switching module 1204 is configured to switch the current display screen back from the real captured image to the virtual program interface, and upload the text to the corresponding text input position in the virtual program interface.

[0116] In one of the embodiments, the second response module 1203 is further configured to:

[0117] By scanning the list of information of the removable devices connected to the head-mounted device;

[0118] When the device information of the first external input device is included in the list of movable device information, it is determined that the head-mounted device is connected to the first external input device.

[0119] In one embodiment, the first switching module 1202 is further configured to:

[0120] Obtain an optional display position where the virtual input interface does not block the external input device when the virtual input interface is displayed in the real shooting screen;

[0121] Based on the optional display position, display the virtual input interface in the real shooting screen.

[0122] In one embodiment, the first switching module 1202 is further configured to: determine that the relative position between the optional display position and the real shooting screen is fixed and unchanged.

[0123] In one embodiment, the first switching module 1202 is further configured to: determine that the virtual input interface includes a first interaction element for text input and a second interaction element for confirming or canceling the input; the second interaction element is triggered by a physical button on the first external input device or the second external input device; wherein, the second external input device is connected to the head-mounted device and is used to control the virtual program, and the triggering method of the physical button is single-button triggering or combined-button triggering.

[0124] In one embodiment, the second response module 1203 is further configured that the first interaction element is a text input box; obtain the text input in the virtual input interface based on the first external input device, including:

[0125] Obtain the text input in the text input box based on the first external input device and synchronously displayed in the text input box;

[0126] After the text input is completed, cache the text for uploading to the virtual program interface, and clear the text synchronously displayed in the text input box.

[0127] Each module in the above text input device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0128] In one embodiment, a computer device is provided. The computer device can be a terminal, and its internal structure diagram can be as Figure 13As shown. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be implemented through WIFI, mobile cellular networks, NFC (Near Field Communication), or other technologies. The computer program, when executed by the processor, implements a text input method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0129] Those skilled in the art can understand that Figure 13 the structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0130] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the following steps are implemented:

[0131] In response to an input trigger operation for a virtual program interface, obtain a real captured image captured by a camera;

[0132] When it is determined that a first external input device is connected to the head-mounted device, switch the current display screen from the virtual program interface to the real captured image, and a virtual input interface for input interaction is integrated in the real captured image;

[0133] In response to an input interaction operation for the virtual input interface, obtain the text input in the virtual input interface based on the first external input device;

[0134] Switch the current display screen from the real captured image back to the virtual program interface, and upload the text to the corresponding text input position in the virtual program interface.

[0135] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0136] By scanning the list of removable device information connected to the head-mounted device;

[0137] When the device information of the first external input device is included in the list of mobile device information, it is determined that the head-mounted device is connected to the first external input device.

[0138] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0139] Obtain the optional display positions where the virtual input interface is displayed in the real captured image and does not block the external input device.

[0140] Based on the optional display positions, display the virtual input interface in the real captured image.

[0141] In one embodiment, when the processor executes the computer program, the following steps are further implemented: The relative position between the optional display positions and the real captured image is fixed and unchanged.

[0142] In one embodiment, when the processor executes the computer program, the following steps are further implemented: The virtual input interface includes a first interaction element for text input and a second interaction element for confirming or canceling the input; the second interaction element is triggered by a physical button on the first external input device or the second external input device; wherein, the second external input device is connected to the head-mounted device and is used to control the virtual program, and the triggering method of the physical button is single-button triggering or combined-button triggering.

[0143] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0144] Obtain the text input in the text input box based on the first external input device and synchronously displayed in the text input box.

[0145] After the text input is completed, cache the text for uploading to the virtual program interface and clear the text synchronously displayed in the text input box.

[0146] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0147] In response to an input trigger operation for the virtual program interface, obtain the real captured image captured by the camera.

[0148] When it is determined that the head-mounted device is connected to the first external input device, switch the current display screen from the virtual program interface to the real captured image, and the virtual input interface for input interaction is integrated in the real captured image.

[0149] In response to an input interaction operation for the virtual input interface, obtain the text input in the virtual input interface based on the first external input device.

[0150] Switch the current display screen from the real-time captured screen back to the virtual program interface, and display the text on the virtual program interface at the corresponding text input position.

[0151] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0152] Scan the list of information of removable devices connected to the headset device;

[0153] When the list of information of removable devices includes the device information of the first external input device, determine that the headset device is connected to the first external input device.

[0154] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0155] Obtain the optional display positions where the virtual input interface is displayed in the real-time captured screen and does not block the external input device;

[0156] Based on the optional display positions, display the virtual input interface in the real-time captured screen.

[0157] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: The relative position between the optional display positions and the real-time captured screen is fixed and unchanged.

[0158] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: The virtual input interface includes a first interaction element for text input and a second interaction element for confirming or canceling the input; the second interaction element is triggered by a physical button on the first external input device or the second external input device; wherein, the second external input device is connected to the headset device and is used to control the virtual program, and the triggering mode of the physical button is single-button triggering or combined-button triggering.

[0159] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0160] Obtain the text input in the text input box based on the first external input device and synchronously displayed in the text input box;

[0161] After the text input is completed, cache the text for display on the virtual program interface, and clear the text synchronously displayed in the text input box.

[0162] In one embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0163] In response to an input trigger operation for a virtual program interface, obtain a real captured image captured by a camera;

[0164] When it is determined that a first external input device is connected to the head-mounted device, switch the current display screen from the virtual program interface to the real captured image, and a virtual input interface for input interaction is integrated in the real captured image;

[0165] In response to an input interaction operation for the virtual input interface, obtain the text input in the virtual input interface based on the first external input device;

[0166] Switch the current display screen from the real captured image back to the virtual program interface, and input the text to the corresponding text input position in the virtual program interface.

[0167] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0168] By scanning the list of removable device information connected to the head-mounted device;

[0169] When the device information of the first external input device is included in the list of removable device information, determine that the first external input device is connected to the head-mounted device.

[0170] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0171] Obtain an optional display position where the virtual input interface does not block the external input device when the virtual input interface is displayed in the real captured image;

[0172] Based on the optional display position, display the virtual input interface in the real captured image.

[0173] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: The relative position between the optional display position and the real captured image is fixed and unchanged.

[0174] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: The virtual input interface includes a first interaction element for text input and a second interaction element for confirming or canceling the input; the second interaction element is triggered by a physical button on the first external input device or the second external input device; wherein, the second external input device is connected to the head-mounted device and is used to control the virtual program, and the triggering mode of the physical button is single-button triggering or combined-button triggering.

[0175] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0176] Obtain the text input in the text input box based on the first external input device and synchronously displayed in the text input box;

[0177] After the text input is completed, cache the text for being sent to the virtual program interface and clear the text synchronously displayed in the text input box.

[0178] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the various embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the various embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the various embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0179] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0180] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A text input method, characterized in that, Applied to a head-mounted device including a camera; the method includes: In response to an input trigger operation for a virtual program interface, obtaining a real captured image captured by the camera; When it is determined that the head-mounted device is connected to a first external input device, switching the current display screen from the virtual program interface to the real captured image, and a virtual input interface for input interaction is integrated in the real captured image; In response to an input interaction operation for the virtual input interface, obtaining text input by the first external input device in the virtual input interface; Switching the current display screen from the real captured image back to the virtual program interface, and uploading the text to the corresponding text input position in the virtual program interface.

2. The method according to claim 1, characterized in that The determining that the head-mounted device is connected to a first external input device includes: Scanning the list of movable device information connected to the head-mounted device; When the device information of the first external input device is included in the list of movable device information, determining that the head-mounted device is connected to the first external input device.

3. The method according to claim 1, wherein The display process of the virtual input interface in the real captured image includes: Obtaining an optional display position where the virtual input interface does not block the external input device when the virtual input interface is displayed in the real captured image; Based on the optional display position, displaying the virtual input interface in the real captured image.

4. The method according to claim 3, characterized in that The relative position of the optional display position and the real captured image is fixed and unchanged.

5. The method according to claim 1, characterized in that The virtual input interface includes a first interaction element for text input and a second interaction element for confirming input or canceling input; the second interaction element is triggered by a physical button on the first external input device or the second external input device; wherein, the second external input device is connected to the head-mounted device and is used to control the virtual program, and the triggering method of the physical button is single-button triggering or combined-button triggering.

6. The method according to claim 5, characterized in that The first interaction element is a text input box; the obtaining of the text input by the first external input device in the virtual input interface includes: Obtaining the text input by the first external input device in the text input box and synchronously displayed in the text input box; After the text input is completed, caching the text for uploading to the virtual program interface, and clearing the text synchronously displayed in the text input box.

7. A text input device, characterized in that, Applied to a head-mounted device including a camera; the device includes: A first response module, configured to obtain a real captured image captured by the camera in response to an input trigger operation for a virtual program interface; A first switching module, configured to switch the current display screen from the virtual program interface to the real captured image when it is determined that the head-mounted device is connected to a first external input device, and a virtual input interface for input interaction is integrated in the real captured image; A second response module, configured to obtain text input by the first external input device in the virtual input interface in response to an input interaction operation for the virtual input interface; A second switching module, configured to switch the current display screen from the real captured screen back to the virtual program interface, and upload the text to a corresponding text input position in the virtual program interface.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images