Cross-device input method and device of head-mounted display device, head-mounted display device and storage medium

Abstract

The invention provides a cross-device input method and device of head-mounted display equipment, the head-mounted display equipment and a storage medium. According to the application method, a first input method calling instruction is sent to terminal equipment in response to a text input request, and a first input method keyboard of the terminal equipment is called, so that convenient cross-device input is realized; the text input operation is executed through the first input method keyboard of the terminal equipment, so that the interference of a complex display environment of the head display equipment on the input operation is reduced, and the text input accuracy is improved; the input text input by the user is displayed through the target window synchronized with the visual angle of the user, so that the input text is always displayed in the visual field range of the user, the user can conveniently check the content of the input text in real time, interference of switching or positioning of an input interface on text content display is avoided, sight transfer of the user in the input process is reduced, and user experience is improved. Therefore, the accuracy and the input efficiency of text input are improved.

Description

Technical Field

[0001] The present application relates to the field of visual display technology of head-mounted display devices, and in particular to a cross-device input method and apparatus for head-mounted display devices, a head-mounted display device, and a storage medium. Background Art

[0002] With the development of augmented reality technology, head-mounted display devices are becoming an important tool in people's daily lives. As one of the basic functions of head-mounted display devices, the optimization of text input is crucial to improving user experience.

[0003] Existing head-mounted display devices have various input methods, including virtual keyboards, external devices such as rings, or text input through visual tracking of fingertip coordinates, as well as physical keyboards, mobile device input and transmission to the head-mounted display device through wireless data channels.

[0004] However, the 3D display mode of the head-mounted display device and the changes in the display screen caused by the user's head movement make it difficult for the user to focus on the displayed text in the head-mounted display device when inputting text, affecting input efficiency and accuracy. At the same time, the complexity of the display environment of the head-mounted display device is also a challenge.

[0005] Therefore, how to improve the text input efficiency of head-mounted display devices has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The present application provides a cross-device input method and apparatus for a head-mounted display device, a head-mounted display device, and a storage medium, aiming to improve the text input efficiency of the head-mounted display device.

[0007] In a first aspect, the present application provides a cross-device input method for a head-mounted display device, the method comprising:

[0008] Based on the text input request, sending a first input method calling instruction to the terminal device to call a first input method keyboard of the terminal device;

[0009] Receiving input text input by a user based on the first input method keyboard;

[0010] Based on a target window with a degree of freedom of 0 DoF, the input text is displayed so that the input text is displayed within the field of view of the head display device.

[0011] In a second aspect, the present application further provides a cross-device input device for a head-mounted display device, wherein the cross-device input device for the head-mounted display device comprises:

[0012] A keyboard calling module, used for sending a first input method calling instruction to a terminal device based on a text input request, so as to call a first input method keyboard of the terminal device;

[0013] A text receiving module, used for receiving input text input by a user based on the first input method keyboard;

[0014] A text display module is used to display the input text based on a target window with a degree of freedom of 0 DoF, so that the input text is displayed within the field of view of the head display device.

[0015] In a third aspect, the present application also provides a head-mounted display device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the cross-device input method of the head-mounted display device as described above are implemented.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the cross-device input method of the head-mounted display device as described above are implemented.

[0017] The present application provides a cross-device input method, apparatus, head-mounted display device and storage medium for a head-mounted display device. The method of the present application includes sending a first input method call instruction to a terminal device based on a text input request to call the first input method keyboard of the terminal device; receiving input text input by a user based on the first input method keyboard; displaying the input text based on a target window with a degree of freedom of 0 DoF so that the input text is displayed within the field of view of the head-mounted display device. Through the above-mentioned method, the present application sends a first input method call instruction to a terminal device in response to a text input request, calls the first input method keyboard of the terminal device, realizes convenient cross-device input, performs text input operations through the first input method keyboard of the terminal device, reduces the interference of the complex display environment of the head-mounted display device on the input operation, and improves the accuracy of text input; displays the input text input by the user through a target window synchronized with the user's perspective, so that the input text is always displayed within the user's field of view, which is convenient for the user to view the content of the input text in real time, reduces the user's line of sight shift during the input process, and thus improves the accuracy and efficiency of text input. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of a flow chart of a first embodiment of a cross-device input method for a head-mounted display device provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a flow chart of a second embodiment of a cross-device input method for a head-mounted display device provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of a cross-device input process of a head-mounted display device provided in an embodiment of the present application;

[0022] Figure 4 It is a structural schematic diagram of a first embodiment of a cross-device input device for a head-mounted display device provided by the present application;

[0023] Figure 5 It is a schematic block diagram of the structure of a head-mounted display device provided in an embodiment of the present application.

[0024] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0028] Please refer to Figure 1 , Figure 1 A flowchart of a first embodiment of a cross-device input method for a head-mounted display device provided in an embodiment of the present application.

[0029] like Figure 1 As shown, the cross-device input method of the head display device includes steps S101 to S103.

[0030] S101, based on a text input request, sending a first input method calling instruction to a terminal device to call a first input method keyboard of the terminal device;

[0031] In one embodiment, the head display device refers to a smart device that can be worn on the head and used to display images or videos, such as a virtual reality (VR) head display or an augmented reality (AR) head display, etc. The terminal device may include a desktop terminal device (such as a desktop computer), a mobile terminal device (such as a mobile phone, a laptop computer, a smart watch, etc.), etc.

[0032] In one embodiment, the head display device can set a custom input method application, which is used to monitor the text input request of IMMS (Input Method Manager Service) through a broadcast receiver, and control the calling and hiding of the input method.

[0033] Among them, the input method management service inherits from the Input Method Service class and is used to process the logic and interface of the input method, such as switching, displaying and hiding the input method, etc. The broadcast receiver is used to monitor and respond to broadcast messages.

[0034] In one embodiment, when a user clicks an input box of a head-mounted display device, a text input event is triggered; when a broadcast receiver in the head-mounted display device monitors that the text input event is triggered, the text input request is generated.

[0035] In one embodiment, when the user wears the head display device, if the application in the head display device requires the user to input text, or the user needs to perform a text input operation, the user needs to click an input box or a search box in the application. When the user clicks the input box or the search box, a text input event is automatically triggered.

[0036] In one embodiment, when a text input event is triggered, the input method management service IMMS configured in the head-mounted display device initiates a broadcast message of a text input request. When the broadcast receiver component configured in the custom input method application listens to the broadcast message, it generates a first input method call instruction to call the first input method keyboard for the user to perform text input operations.

[0037] Exemplarily, the text input request may include necessary information for executing text input, such as the ID of the input box, the input type (text, number, etc.), the input mode (multi-line, single-line, etc.), and other parameters that may affect the behavior of the input method.

[0038] This embodiment responds to text input requests through a custom input method application, allowing the user to trigger a text input event by clicking on an input box when wearing a head-mounted display device, and then call the first input method keyboard of the mobile terminal to input text, thereby improving the user's text input interaction efficiency in an augmented reality environment and making text input operations more convenient.

[0039] In one embodiment, if Figure 2 As shown, step S101 specifically includes steps S201 to S204:

[0040] S201, based on the text input request, detecting the communication connection status between the head display device and the terminal device;

[0041] In one embodiment, if Figure 3 As shown, when the broadcast receiver in the head display device monitors the broadcast message of the IMMS initiating a text input request, the input method application set in the head display device can check the current communication connection status between the head display device and the terminal device to determine whether text data can be transmitted between the two.

[0042] Exemplarily, the communication method between the head display device and the terminal device may include Bluetooth connection, Wi-Fi connection, local area network and wide area network protocol connection, etc.

[0043] Exemplarily, the head display device and the terminal device can establish a connection via Bluetooth technology. When establishing a connection, the terminal device initiates a Bluetooth broadcast containing first device information (such as the device ID, unique identification code, etc. of the terminal device). The head display device responds to this broadcast, sends an access request, and after successful access, sends the second device information (such as the device ID, unique identification code, verification code, etc. of the head display device) to the terminal device via Bluetooth. After receiving the second device information, the terminal device can compare it with the pre-stored device information library. If the comparison is successful, a communication connection is established.

[0044] In one embodiment, a listener may be registered in the input method application to monitor the status changes of the communication connection between the head display device and the terminal device, and respond to the establishment and disconnection events of the communication connection. For example, when the communication connection is established, the listener may receive an event with a connection status of "connected"; when the communication connection is disconnected, the listener may receive an event with a connection status of "disconnected".

[0045] In another embodiment, when the broadcast receiver in the input method application monitors the broadcast message of the text input request, the head display device can send a small data packet to the terminal device to test whether the head display device and the terminal device are in a communication connection state. If the data packet is successfully received and a confirmation is returned, it can be considered that the head display device and the terminal device are in a communication connection state, and the communication connection state is normal; if the data packet fails to be sent or no confirmation is received within a certain period of time, it can be considered that the head display device and the terminal device are in a communication disconnection state.

[0046] In one embodiment, the input method application can also determine whether the head-mounted display device is in communication connection with the terminal device by querying the device list that the head-mounted display device is currently communicating with, detecting whether there is a device ID of the terminal device in the device list, and checking the current connection status of the terminal device. If the device ID of the terminal device is not in the device list, the two are in a communication disconnection state; if the device ID of the terminal device is in the device list and the current connection status is displayed as normal, the two are in a communication connection state.

[0047] S202, when the head display device and the terminal device are in a communication connection state, generating a first input method calling instruction for the terminal device;

[0048] In one embodiment, when the input method application detects that the head display device and the terminal device are in a communication connection state, the head display device generates a first input method calling instruction for the terminal device. Conversely, if the input method application detects that the head display device and the terminal device are not in a communication connection state, the first input method calling instruction is not generated, that is, the head display device does not call the first input method keyboard of the terminal device.

[0049] S203: Based on the communication connection channel between the head display device and the terminal device, send the first input method calling instruction to the terminal device to call the first input method keyboard of the terminal device.

[0050] In one embodiment, after the first input method calling instruction is generated at the head display device end, the head display device sends the first input method calling instruction to the terminal device through the communication connection data channel between the head display device and the terminal device. The terminal device responds to the first input method calling instruction sent by the head display device, displays the first input method keyboard in the terminal device for user input, and transmits the text data input by the user to the head display device in real time through the communication connection data channel.

[0051] This embodiment ensures that the text input by the user in the head-mounted display device can be accurately transmitted to the terminal device in real time by detecting and verifying the communication connection status between the two, and displays the input method keyboard on the terminal device for user input, thereby improving the reliability and efficiency of cross-device text input.

[0052] S102, receiving input text input by a user based on the first input method keyboard;

[0053] In one embodiment, when the mobile terminal receives the first input method call instruction, the first input method keyboard of the mobile terminal is displayed for the user to perform a text input operation. The user inputs text content through the first input method keyboard of the mobile terminal, and then the text content is transmitted to the head display device through the communication connection data channel between the head display device and the mobile terminal, and the head display device displays the text content input by the user.

[0054] In one embodiment, an Input Connection interface may be established in the head mounted display device to receive input text from the mobile terminal, thereby providing necessary methods and mechanisms for interaction between the input method service and the application.

[0055] The Input Connection interface can serve as a communication channel between the input method service and the application receiving the input. The Input Connection interface allows the input method to read the text around the cursor, submit text to the text box, and submit raw key events to the application.

[0056] In one embodiment, based on the window management service, the target window that needs to perform the text input operation is searched; based on the preset display parameters, the display parameters of the target window are set so that the target window is synchronized with the user's viewing angle.

[0057] In one embodiment, when the input method application detects that the head display device and the terminal device are in a communication connection state, the window management service (WMS) searches for the target window that needs to perform the text input operation based on the input box ID, input box type and other information in the text input request, that is, determines which application or service needs text input.

[0058] Exemplarily, the target window may include a query window, a chat window, a document window, and the like.

[0059] Exemplarily, different types of target windows may adopt different processing methods and display parameters to serve different text input purposes and user interaction modes.

[0060] For example, the query window requires the user to enter the query content and display the search results. The search results may change dynamically, so the query window needs to be able to quickly update the content. And the query window needs to manage the input focus (such as the search box or query input area, etc.). When the user clicks or switches to the input focus through the keyboard, the cursor (or virtual cursor) will appear, prompting the user to start entering the query content.

[0061] The chat window may contain a large number of historical messages, and an effective scrolling mechanism and query mechanism are needed to manage the content display. The chat window usually needs to display an input prompt or cursor so that the user knows where to enter the message.

[0062] The document window needs to correctly display various document formats, such as text, images, tables, etc. The text input of the document window requires more interactive elements, such as toolbars, menus, etc. At the same time, the document window needs to provide different view modes, such as reading mode, editing mode, etc.

[0063] In one embodiment, after determining the target window, the head display device can retrieve the window information of the target window, such as window ID, window type, etc., so as to configure different preset display parameters according to the window information. The preset display parameters may include parameters such as window display coordinates, window display size, rendering priority, and window degrees of freedom.

[0064] For example, the user can adjust the display position of the target window within the field of view by entering coordinates or dragging. Especially when multiple windows are displayed, if the user needs to view multiple windows at the same time, the user can adjust parameters such as the window display coordinates, window display size, and rendering priority of the multiple display windows to make the multiple display windows display side by side, avoiding window overlap between the target window and other display windows, thereby realizing multi-window display.

[0065] Exemplarily, the window display size can be adjusted according to user habits and window properties. The user can proportionally adjust the overall size of the target window, such as enlarging or reducing the entire window; or adjust the width and height of the target window to a fixed ratio, such as a 16:9 window size; the width or height of the window can also be set and adjusted separately.

[0066] In one embodiment, the window level of the target window is set as the highest priority rendering level, so that the target window is displayed above other windows within the field of view.

[0067] For example, during the text input process, in order to prevent the user input text from being blocked by other windows, it is necessary to adjust the target window's hierarchy setting and raise the target window's hierarchy to be above other windows so that the input text is displayed to the user first.

[0068] Exemplarily, in order to ensure that the target window is synchronized with the user's perspective and to ensure that the target window and the input text content displayed in the target window are displayed within the user's field of view, the user can set the degree of freedom of the target window relative to the user's perspective to 0 Dof (Degree of freedom) on the head-mounted display device, so that the target window is fixed in the user's field of view, and no matter how the user moves or turns his head, the target window remains displayed in the same position and direction.

[0069] This embodiment ensures the reliability of cross-device text input by establishing an Input Connection interface; achieves accurate search of the target window through the window management service; and ensures synchronization between the target window and the user's perspective by configuring the display parameters of the target window, so that users can efficiently and conveniently input and view text in different application scenarios.

[0070] S103. Display the input text based on a target window with a degree of freedom of 0 DoF, so that the input text is displayed within the field of view of the head display device.

[0071] In one embodiment, the degree of freedom of the target window relative to the user's viewing angle is 0 Dof, which means that the target window is fixed within the user's field of view. No matter how the user moves or turns his head, the target window remains displayed in the same position and direction in the field of view, presenting a visual display effect of information windows such as a status display window and a small map window.

[0072] In one embodiment, the degrees of freedom of the target window relative to the user's viewing angle is set to 0 Dof. Although the field of view will change synchronously with the posture of the user's head (when wearing a head-mounted display device), the posture of the target window relative to the user's viewing angle will not change. That is, no matter where the user's viewing angle is directed, the target window is always displayed at a fixed position in the field of view.

[0073] In one embodiment, the head-mounted display device can use head tracking technology to determine the user's viewing direction, and dynamically adjust the position and deflection angle of the field of view according to the user's head position and viewing direction. When the degree of freedom of the target window relative to the user's viewing angle is 0 Dof, no matter how the posture (position and deflection angle) of the field of view changes, the target window will always be displayed in the field of view, ensuring that the input text is always displayed within the user's field of view.

[0074] Exemplarily, head tracking technology is a technology that detects the user's head posture and movement trajectory in real time through the device's sensors, and maps these data to a virtual reality or augmented reality scene. Head tracking technology uses sensors installed in the head display device, such as gyroscopes, accelerometers, magnetometers, etc., to detect changes in the posture of the head display device. Using sensor data, combined with mathematical algorithms in three-dimensional space, the position and direction of the head display device, including the pitch angle, yaw angle, and roll angle of the head display device, are calculated and updated in real time. According to the real-time posture of the head display device, the posture of the field of view is transformed (for example, when the posture of the head display device is turned to the right, the page displayed in the field of view will be switched), and no matter how the field of view corresponding to the user's perspective changes, the target window will be displayed in the field of view (when the posture of the head display device is turned to the right, when the page displayed in the field of view is switched, the display position and content of the target window will not change, and it will always be displayed in the field of view), to ensure that the target window is always displayed at a fixed position in the field of view.

[0075] In one embodiment, the head-mounted display device can also cooperate with eye tracking technology to track the user's line of sight focus. According to the intersection of the user's line of sight, while synchronously adjusting the position of the target window and the user's viewing angle (the position of the head-mounted display device), the depth coordinates of the target window can also be adjusted, so that the target window is displayed at the depth position of the user's line of sight focus, so that the user can read the input text in the target window more clearly.

[0076] It is understandable that the head tracking technology and the eye tracking technology are both related existing technologies in the field, and this application will not describe them in detail.

[0077] The present embodiment provides a cross-device input method for a head-mounted display device. The method sends a first input method calling instruction to a terminal device in response to a text input request, calls a first input method keyboard of the terminal device, and realizes convenient cross-device input. The text input operation is performed through the first input method keyboard of the terminal device, thereby reducing the interference of the complex display environment of the head-mounted display device on the input operation and improving the accuracy of text input. The input text entered by the user is displayed through a target window synchronized with the user's perspective, so that the input text is always displayed within the user's field of view, which is convenient for the user to view the content of the input text in real time, avoids the interference of switching or positioning of the input interface on the display of the text content, reduces the user's line of sight shift during the input process, and thus improves the accuracy and efficiency of text input.

[0078] In one embodiment, continue to refer to Figure 2 and Figure 3 The illustrated embodiment, after detecting the communication connection status between the head display device and the terminal device based on the text input request, further includes:

[0079] S204. When the head display device is not communicatively connected with the terminal device, generate a second input method calling instruction for the head display device; based on the second input method calling instruction, call a second input method keyboard of the head display device; based on a target window with a degree of freedom of 0 DoF, display input text input by the user based on the second input method keyboard, so that the input text is displayed within the field of view of the head display device.

[0080] In one embodiment, if the head display device is disconnected from the terminal device, the head display device cannot call the first input method keyboard of the terminal device for the user to use. In this case, the second input method keyboard built into the head display device can be called for the user to use.

[0081] Specifically, when it is detected that the head-mounted display device is not communicatively connected with the terminal device, a second input method calling instruction of the head-mounted display device is generated, and based on the second input method calling instruction, the second input method keyboard built into the head-mounted display device is called, and the second input method keyboard is projected and displayed within the user's field of view for the user to perform text input operations.

[0082] In one embodiment, based on the second input method calling instruction, display parameters of the second input method keyboard are obtained; based on the display parameters, the second input method keyboard is displayed within the user's field of view, so that the user can perform text input operations through the second input method keyboard.

[0083] In one embodiment, the second input method keyboard built into the head display device can be displayed at a preset position when displayed. For example, the depth coordinates and height coordinates of the second input method keyboard displayed within the user's field of view can be preset. The user can also be allowed to customize the display parameters of the second input method keyboard, including the shape, size, keyboard layout, display coordinates, display effects, etc. of the second input method keyboard.

[0084] For example, for the appearance of the second input method keyboard, the user can set the shape of the keyboard, such as rectangle, circle or other creative shapes, and can also configure the keyboard border style, color, transparency and other visual elements, or configure different themes or skin options so that the user can use a variety of keyboard appearances.

[0085] Regarding the size of the second input method keyboard, the user can adjust the overall size of the keyboard to accommodate different viewing angles and comfort levels. At the same time, the user can also independently adjust each element of the keyboard (such as key size, position, etc.).

[0086] For the keyboard layout of the second input method keyboard, the user can set different keyboard layouts, such as QWERTY, AZERTY, Dvorak, etc.

[0087] For the display coordinates of the second input method keyboard, the user can set the position of the keyboard in the virtual space according to his or her own input habits, including horizontal coordinates, vertical coordinates and depth coordinates, and can also adjust the tilt angle of the keyboard.

[0088] For the display effects of the second input method keyboard, users can select different animation effects for the keyboard, such as dynamic feedback when a key is pressed, halo effect, color change, etc.

[0089] In this embodiment, when the communication between the head display device and the terminal device is disconnected, the second input method keyboard built into the head display device is called, and the input text is displayed synchronously with the user's perspective, thereby ensuring that the user can input text in the virtual environment even without the support of external devices, thereby improving the reliability of text input of the head display device.

[0090] Furthermore, based on the fingertip tracking technology, the coordinates of the user's fingertip are identified; based on the coordinate matching algorithm, the target key pointed to by the fingertip coordinates in the second input method keyboard is identified to generate input text.

[0091] In one embodiment, the second input method keyboard of the head-mounted display device can be a virtual keyboard, which can track the user's fingertip coordinates through fingertip tracking technology, and identify the landing point of the user's fingertip coordinates in the virtual keyboard, and then input characters according to the keys corresponding to the landing points to generate input text.

[0092] For example, a camera installed in a head-mounted display device can be used to collect a user's hand image, and a deep learning model (such as YOLOv3) can be used to process the hand image to detect the contours of the hand and fingertips, thereby identifying and tracking the fingertip coordinates. Optionally, the fingertip coordinates can be tracked using algorithms such as a Kalman filter, and the untracked fingertips can be corrected using the K-curvature method to improve the real-time and accuracy of fingertip coordinate tracking.

[0093] Generally, the user may use a single finger to tap the keys of the second input method keyboard, or may use multiple fingers to tap the keys of the second input method keyboard. Therefore, when the user uses the second input method keyboard to input text, the fingertip tracking technology needs to track the fingertip coordinates of all identifiable user fingertips, and identify the keys pointed to by the fingertip coordinates of each fingertip, and then enter the characters corresponding to the keys to generate input text.

[0094] Exemplarily, the display coordinates of the second input method keyboard in the display space of the head display device can be pre-set, and the coordinate information, character information, etc. of each key in the second input method keyboard can be collected. When the user inputs text through the second input method keyboard, the distance between the fingertip coordinates and the coordinate fingertips of each key can be calculated to match them, and the key closest to the fingertip coordinates can be identified as the target key, and the character corresponding to the target key can be input, thereby generating the corresponding input text.

[0095] In one embodiment, the user performs text input operations through the second input method keyboard of the head display device, and the input text is displayed in the target window, similar to when the first input method keyboard of the mobile terminal is called to perform text input operations. Regardless of whether the first input method keyboard of the mobile terminal or the second input method keyboard of the head display device is used, the parameter settings for the target window are the same, that is, the degree of freedom is set to 0 Dof, so that the target window is synchronized with the user's perspective: the posture of the head display device is updated in real time through head tracking technology, and the posture of the target window is synchronously updated according to the posture of the head display device, so that the target window is always displayed at a fixed position in the user's field of view. At the same time, the user's line of sight focus can be tracked through eye tracking technology, and the depth coordinates of the target window can be adjusted so that the target window is displayed at the user's line of sight focus, thereby improving the clarity of the input text display.

[0096] This embodiment tracks the coordinates of the user's fingertips and uses a coordinate matching algorithm to determine the keyboard key pointed to by the fingertips to generate corresponding input text, thereby improving the accuracy of text input on the built-in input method keyboard of the head display device. By setting the degree of freedom of the target window to 0 DoF and using eye tracking technology, it is ensured that the target window is always synchronized with the user's perspective, thereby improving the clarity of text input and user experience.

[0097] See also Figure 4 , Figure 4 It is a structural schematic diagram of a first embodiment of a cross-device input device for a head-mounted display device provided in the present application. The cross-device input device for a head-mounted display device is used to execute the cross-device input method for the head-mounted display device described above.

[0098] like Figure 4 As shown, the cross-device input device 300 of the head display device includes: a keyboard calling module 301, a text receiving module 302 and a text display module 303.

[0099] The keyboard calling module 301 is used to send a first input method calling instruction to the terminal device based on the text input request to call the first input method keyboard of the terminal device;

[0100] A text receiving module 302, used for receiving input text input by a user based on the first input method keyboard;

[0101] The text display module 303 is used to display the input text based on the target window with a degree of freedom of 0 DoF, so that the input text is displayed within the field of view of the head display device.

[0102] In one embodiment, the cross-device input device 300 of the head display device further includes a window parameter setting module, including:

[0103] A target window search unit, used for searching the target window where the text input operation needs to be performed based on the window management service;

[0104] The window parameter setting unit is used to set display parameters for the target window based on preset display parameters so that the target window is synchronized with the user's viewing angle.

[0105] In one embodiment, the cross-device input device 300 of the head display device further includes a request generation module, including:

[0106] An event triggering unit, used to trigger a text input event when the user clicks the input box of the head display device;

[0107] The request generating unit is used to generate the text input request when the broadcast receiver in the head display device monitors that the text input event is triggered.

[0108] In one embodiment, the keyboard calling module 301 includes:

[0109] A communication connection status detection unit, configured to detect a communication connection status between the head display device and the terminal device based on the text input request;

[0110] a first calling instruction generating unit, configured to generate a first input method calling instruction of a terminal device when the head display device and the terminal device are in a communication connection state;

[0111] A calling instruction sending unit is used to send the first input method calling instruction to the terminal device based on the communication connection channel between the head display device and the terminal device, so as to call the first input method keyboard of the terminal device.

[0112] In one embodiment, the calling instruction generation module further includes a second text input submodule, including:

[0113] a second calling instruction generating unit, configured to generate a second input method calling instruction for the head mounted display device when the head mounted display device is not in communication connection with the terminal device;

[0114] A second input method keyboard calling unit, configured to call a second input method keyboard of the head display device based on a second input method calling instruction of the head display device;

[0115] The second text input unit is used to display the input text input by the user based on the second input method keyboard based on the target window with a degree of freedom of 0 DoF, so that the input text is displayed within the field of view of the head display device.

[0116] In one embodiment, the second input method keyboard calling unit includes:

[0117] A display parameter acquisition subunit, configured to acquire display parameters of the second input method keyboard based on the second input method calling instruction;

[0118] The keyboard display subunit is used to display the second input method keyboard within the user's field of view based on the display parameters, so that the user can perform text input operations through the second input method keyboard.

[0119] In one embodiment, the cross-device input device 300 of the head display device further includes a window priority setting module, including:

[0120] The window priority setting unit is used to set the window level of the target window as the highest priority rendering level, so that the target window is displayed above other windows within the field of view.

[0121] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the embodiment of the cross-device input method of the aforementioned head-mounted display device, and will not be repeated here.

[0122] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 5 Running on the headset shown.

[0123] See also Figure 5 , Figure 5 1 is a schematic block diagram of the structure of a head-mounted display device provided in an embodiment of the present application. The head-mounted display device may be a server.

[0124] See also Figure 5 The head display device includes a processor, a memory, and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0125] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any cross-device input method of the head display device.

[0126] The processor is used to provide computing and control capabilities to support the operation of the entire head-mounted display device.

[0127] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any cross-device input method of the head display device.

[0128] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the head-mounted display device to which the solution of the present application is applied. The specific head-mounted display device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0129] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0130] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0131] Based on the text input request, sending a first input method calling instruction to the terminal device to call a first input method keyboard of the terminal device;

[0132] Receiving input text input by a user based on the first input method keyboard;

[0133] Based on a target window with a degree of freedom of 0 DoF, the input text is displayed so that the input text is displayed within the field of view of the head display device.

[0134] In one embodiment, before the processor implements the display of the input text based on the target window with a degree of freedom of 0 DoF so that the input text is displayed within the field of view of the head display device, it is further configured to implement:

[0135] Based on the window management service, searching for the target window where the text input operation needs to be performed;

[0136] Based on preset display parameters, display parameters are set for the target window so that the target window is synchronized with the user's viewing angle.

[0137] In one embodiment, before implementing the sending of the first input method calling instruction to the terminal device based on the text input request to call the first input method keyboard of the terminal device, the processor is further configured to implement:

[0138] When the user clicks the input box of the head display device, a text input event is triggered;

[0139] When the broadcast receiver in the head display device monitors that the text input event is triggered, the text input request is generated.

[0140] In one embodiment, when the processor implements the sending of the first input method calling instruction to the terminal device based on the text input request to call the first input method keyboard of the terminal device, it is configured to implement:

[0141] Based on the text input request, detecting a communication connection status between the head display device and the terminal device;

[0142] When the head display device and the terminal device are in a communication connection state, generating a first input method calling instruction of the terminal device;

[0143] Based on the communication connection channel between the head display device and the terminal device, the first input method calling instruction is sent to the terminal device to call the first input method keyboard of the terminal device.

[0144] In one embodiment, after implementing the detecting the communication connection status between the head display device and the terminal device based on the text input request, the processor is further configured to implement:

[0145] When the head display device is not in communication connection with the terminal device, generating a second input method calling instruction for the head display device;

[0146] Based on the second input method calling instruction of the head display device, calling the second input method keyboard of the head display device;

[0147] Based on the target window with a degree of freedom of 0 DoF, the input text input by the user based on the second input method keyboard is displayed so that the input text is displayed within the field of view of the head display device.

[0148] In one embodiment, when the processor implements the second input method calling instruction based on the head display device and calls the second input method keyboard of the head display device, it is used to implement:

[0149] Based on the second input method calling instruction, obtaining display parameters of the second input method keyboard;

[0150] Based on the display parameters, the second input method keyboard is displayed within a user's field of view, so that the user can perform a text input operation through the second input method keyboard.

[0151] In one embodiment, before the processor implements the display of the input text based on the target window with a degree of freedom of 0 DoF so that the input text is displayed within the field of view of the head display device, it is further configured to implement:

[0152] The window level of the target window is set as the highest priority rendering level, so that the target window is displayed above other windows within the field of view.

[0153] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement a cross-device input method for any head-mounted display device provided in an embodiment of the present application.

[0154] The computer-readable storage medium may be an internal storage unit of the head-mounted display device described in the foregoing embodiment, such as a hard disk or memory of the head-mounted display device. The computer-readable storage medium may also be an external storage device of the head-mounted display device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the head-mounted display device.

[0155] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A cross-device input method for a head-mounted display device, characterized in that: The method comprises: Based on the text input request, sending a first input method calling instruction to the terminal device to call a first input method keyboard of the terminal device; Receiving input text input by a user based on the first input method keyboard; Based on a target window with a degree of freedom of 0 DoF, the input text is displayed so that the input text is displayed within the field of view of the head display device.

2. The cross-device input method of a head display device according to claim 1, characterized in that: Before displaying the input text based on the target window with a degree of freedom of 0 DoF so that the input text is displayed within the field of view of the head display device, the method further includes: Based on the window management service, searching for the target window where the text input operation needs to be performed; Based on preset display parameters, display parameters are set for the target window so that the target window is synchronized with the user's viewing angle.

3. The cross-device input method of a head display device according to claim 1, characterized in that: Before sending a first input method calling instruction to the terminal device based on the text input request to call the first input method keyboard of the terminal device, the method further includes: When the user clicks the input box of the head display device, a text input event is triggered; When the broadcast receiver in the head display device monitors that the text input event is triggered, the text input request is generated.

4. The cross-device input method of a head display device according to claim 1, characterized in that: The step of sending a first input method calling instruction to a terminal device based on a text input request to call a first input method keyboard of the terminal device includes: Based on the text input request, detecting a communication connection status between the head display device and the terminal device; When the head display device and the terminal device are in a communication connection state, generating a first input method calling instruction of the terminal device; Based on the communication connection channel between the head display device and the terminal device, the first input method calling instruction is sent to the terminal device to call the first input method keyboard of the terminal device.

5. The cross-device input method of the head display device according to claim 4, characterized in that: After detecting the communication connection status between the head display device and the terminal device based on the text input request, the method further includes: When the head display device is not in communication connection with the terminal device, generating a second input method calling instruction for the head display device; Calling a second input method keyboard of the head display device based on the second input method calling instruction; Based on the target window with a degree of freedom of 0 DoF, the input text input by the user based on the second input method keyboard is displayed so that the input text is displayed within the field of view of the head display device.

6. The cross-device input method of a head display device according to claim 5, characterized in that: The calling the second input method keyboard of the head display device based on the second input method calling instruction includes: Based on the second input method calling instruction, obtaining display parameters of the second input method keyboard; Based on the display parameters, the second input method keyboard is displayed within a user's field of view, so that the user can perform a text input operation through the second input method keyboard.

7. The cross-device input method of a head display device according to claim 1, characterized in that: Before displaying the input text based on the target window with a degree of freedom of 0 DoF so that the input text is displayed within the field of view of the head display device, the method further includes: The window level of the target window is set as the highest priority rendering level, so that the target window is displayed above other windows within the field of view.

8. A cross-device input device for a head-mounted display device, characterized in that: The cross-device input device of the head display device includes: A keyboard calling module, used for sending a first input method calling instruction to a terminal device based on a text input request, so as to call a first input method keyboard of the terminal device; A text receiving module, used for receiving input text input by a user based on the first input method keyboard; A text display module is used to display the input text based on a target window with a degree of freedom of 0 DoF, so that the input text is displayed within the field of view of the head display device.

9. A head mounted display device, characterized in that: The head display device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the cross-device input method of the head display device according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by the processor, the steps of the cross-device input method of the head display device according to any one of claims 1 to 7 are implemented.

Images