Input interaction method of virtual keyboard, head-mounted display device, medium and product

By displaying the virtual keyboard in the head-mounted display device and responding to user input operations, the problem of users frequently switching keys is solved, and input efficiency and user experience is improved.

CN118915918APending Publication Date: 2024-11-08HANGZHOU LINGBAN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410956252.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

When users frequently enter characters in different states, they need to frequently click the switch key to switch keys, resulting in low input efficiency, complex operation and poor user experience.

Method used

Display the virtual keyboard in the display space of the head-mounted display device, respond to the user's input operations, determine the status of the character input keys, and display the associative word information on the virtual keyboard. Users can directly enter the content by selecting the associative word to reduce switching Key operation.

Benefits of technology

It improves the efficiency of users entering characters, simplifies input operations, and improves user experience. Users can directly input characters in different states through different input operations without frequently clicking to switch keys.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915918A_ABST
    Figure CN118915918A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses an input interaction method of a virtual keyboard, head-mounted display equipment, a medium and a product. A specific embodiment of the method comprises the steps of displaying a virtual keyboard in a display space in response to detection of an input trigger operation corresponding to a target input box; in response to detection of a first input operation acting on the character input key position, determining a character of the character input key position in a first input state as an auxiliary input character; in response to detection of a second input operation corresponding to the character input key position, determining a character of the character input key position in a second input state as an auxiliary input character; displaying at least one associational word information corresponding to the determined at least one auxiliary input character above the virtual keyboard; and in response to detection of a selection operation acting on the displayed arbitrary associational word information, displaying the arbitrary associational word information in the target input box. According to the embodiment, the character input efficiency of the user is improved, the input operation is simplified, and the user experience is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to an input interaction method of a virtual keyboard, a head-mounted display device, a medium, and a product. Background Art

[0002] The physical keyboard and the software keyboard displayed on the screen can be used to input characters and perform various operations. At present, when inputting characters through the keyboard, the usual method is: click the keyboard to input characters, and when you need to input characters in other states indicated by the key position, you need to click a physical key or a key position on the screen to switch.

[0003] However, the inventors have found that when the above method is used, the following technical problems often occur: when users frequently input characters in different states, they need to frequently click the switch key to switch the characters indicated by the key position, resulting in low input efficiency and more complicated input operations, thereby resulting in a poor user experience.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose an input interaction method of a virtual keyboard, a head-mounted display device, a computer-readable medium, and a computer program product to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide an input interaction method for a virtual keyboard, the method comprising: in response to detecting an input trigger operation corresponding to a target input box, displaying a virtual keyboard in the display space of the head-mounted display device; in response to detecting a first input operation acting on a character input key position of the virtual keyboard, determining a character at the character input key position in a first input state as an auxiliary input character; in response to detecting a second input operation corresponding to the character input key position of the virtual keyboard, determining a character at the character input key position in a second input state as an auxiliary input character; displaying at least one associative word information corresponding to at least one determined auxiliary input character above the virtual keyboard; and in response to detecting a selection operation acting on any displayed associative word information, displaying the any associative word information in the target input box.

[0008] Optionally, the method also includes: in response to detecting a touch operation acting on an associated touch screen, and the indication position corresponding to the above touch operation is in the character input key position, and the touch mode corresponding to the above touch operation satisfies the first input condition, determining that a first input operation acting on the character input key position of the above virtual keyboard is detected.

[0009] Optionally, the method also includes: in response to detecting a touch operation acting on an associated touch screen, and the indication position corresponding to the above touch operation is in the character input key position, and the touch mode corresponding to the above touch operation satisfies the second input condition, determining that a second input operation acting on the character input key position of the above virtual keyboard is detected.

[0010] Optionally, the method also includes: in response to detecting a sliding operation acting on the above-mentioned touch screen, mapping the touch coordinate sequence corresponding to the above-mentioned sliding operation to the display space of the above-mentioned head-mounted display device to obtain a mapping coordinate sequence; according to the obtained mapping coordinate sequence, moving the indicated position of the indicating element in the display space of the above-mentioned head-mounted display device.

[0011] Optionally, the method also includes: in response to detecting a waving operation acting on the terminal device corresponding to the above-mentioned touch screen, mapping the spatial coordinate sequence corresponding to the above-mentioned waving operation to the display space of the above-mentioned head-mounted display device to obtain a mapping coordinate sequence; according to the obtained mapping coordinate sequence, moving the indicated position of the indicating element in the display space of the above-mentioned head-mounted display device.

[0012] Optionally, the method also includes: in response to detecting a first sliding input operation acting on each character input key position of the above-mentioned virtual keyboard, determining each character of each character input key position in the first input state as an auxiliary input character sequence, wherein each of the above-mentioned character input key positions corresponds to a sliding order; displaying at least one sliding associative word information corresponding to the auxiliary input character sequence above the above-mentioned virtual keyboard; in response to detecting a selection operation acting on any sliding associative word information displayed, displaying the above-mentioned arbitrary sliding associative word information in the above-mentioned target input box.

[0013] Optionally, the method also includes: in response to detecting a second sliding input operation acting on each character input key position of the above-mentioned virtual keyboard, determining each character of each character input key position in the second input state as an auxiliary input character sequence, wherein each character input key position corresponds to a sliding order; displaying at least one sliding associative word information corresponding to the determined auxiliary input character sequence above the above-mentioned virtual keyboard; in response to detecting a selection operation acting on any sliding associative word information displayed, displaying the above-mentioned arbitrary sliding associative word information in the above-mentioned target input box.

[0014] In a second aspect, some embodiments of the present disclosure provide a head-mounted display device, comprising: one or more processors; a display module for imaging in front of the user's eyes; and a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0015] In a third aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0016] In a fourth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which implements the method described in any implementation manner of the above-mentioned first aspect when executed by a processor.

[0017] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the input interaction method of the virtual keyboard of some embodiments of the present disclosure, the efficiency of user input of characters is improved, the input operation is simplified, and the user experience is improved. Specifically, the reason for the poor user experience is that when the user frequently inputs characters in different states, it is necessary to frequently click the switch key to switch the characters indicated by the key position, resulting in low input efficiency and complex input operations, thereby resulting in poor user experience. Based on this, the input interaction method of the virtual keyboard of some embodiments of the present disclosure, first, in response to detecting the input trigger operation corresponding to the target input box, the virtual keyboard is displayed in the display space of the head-mounted display device. Thus, the virtual keyboard can be displayed in the display space of the head-mounted display device for the user to watch. Then, in response to detecting the first input operation acting on the character input key position of the above-mentioned virtual keyboard, the character of the character input key position in the first input state is determined as an auxiliary input character. Thus, when the user performs the first input operation, the character input in the first input state can be determined as the character the user wants to input. Secondly, in response to detecting the second input operation corresponding to the character input key position of the above-mentioned virtual keyboard, the character of the character input key position in the second input state is determined as an auxiliary input character. Thus, when the user performs the second input operation, the character input in the second input state can be determined as the character that the user wants to input. Then, at least one associative word information corresponding to at least one determined auxiliary input character is displayed above the above-mentioned virtual keyboard. Thus, the corresponding associative word information can be determined according to at least one character input by the user. Finally, in response to detecting a selection operation acting on any associative word information displayed, the above-mentioned any associative word information is displayed in the above-mentioned target input box. Thus, after the user selects the associative word information, the selected associative word information can be used as the input content of the target input box. Also because the user can directly perform different input operations on the character input key position to input characters in different input states, there is no need to click the switch key every time the character state is switched, thereby improving the efficiency of the user inputting characters, simplifying the input operation, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0019] Figure 1 is a flow chart of some embodiments of the input interaction method of the virtual keyboard according to the present disclosure;

[0020] Figure 2is a schematic diagram of an application scenario of a first input operation of an input interaction method of a virtual keyboard according to some embodiments of the present disclosure;

[0021] Figure 3 is a schematic diagram of another application scenario of a first input operation of the input interaction method of a virtual keyboard according to some embodiments of the present disclosure;

[0022] Figure 4 is a schematic diagram of an application scenario of a first sliding input operation of an input interaction method of a virtual keyboard according to some embodiments of the present disclosure;

[0023] Figure 5 is a schematic diagram of another application scenario of the first sliding input operation of the input interaction method of the virtual keyboard according to some embodiments of the present disclosure;

[0024] Figure 6 It is a schematic diagram of the structure of a head-mounted display device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0026] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0027] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0028] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0030] With regard to the collection, storage, and use of user personal information (such as associated word information and sliding associated word information) involved in this disclosure, before performing the corresponding operations, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, fulfilling the obligation to inform the personal information subject, and obtaining the authorization and consent of the personal information subject in advance.

[0031] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0032] Figure 1 The process 100 of some embodiments of the input interaction method of the virtual keyboard according to the present disclosure is shown. The input interaction method of the virtual keyboard includes the following steps:

[0033] Step 101: In response to detecting an input triggering operation corresponding to a target input box, displaying a virtual keyboard in a display space of a head mounted display device.

[0034] In some embodiments, the execution subject (e.g., head-mounted display device) of the input interaction method of the virtual keyboard may display the virtual keyboard in the display space of the head-mounted display device in response to detecting the input trigger operation of the corresponding target input box. Among them, the head-mounted display device may be a display device for wearing on the head to form an image in front of the user's eyes. The head-mounted display device may be a split head-mounted display device or an integrated head-mounted display device. The split head-mounted display device may include, but is not limited to, head-mounted glasses and smart terminals. The smart terminal may provide computing power and / or electricity for the content playback of the head-mounted glasses. The head-mounted glasses and the integrated head-mounted display device may include, but are not limited to, AR glasses, MR glasses, and VR glasses. The target input box may be a control in which the user needs to enter content. Here, the input content may include, but is not limited to, text, emoticons, and emoticons. The input trigger operation may be a selection operation acting on the target input box. Here, the selection operation may include, but is not limited to, at least one of the following: click, hover. It should be noted that the user can interact through at least one of the following interaction methods: touch interaction, gesture interaction, head control interaction, and ray interaction. For example, the user can slide the pointer or cursor in the display space into the above-mentioned target input box through the touch screen and then click to perform an input trigger operation. For another example, the user can point the multi-degree-of-freedom ray in the display space to the above-mentioned target input box through the smart terminal, and then click on the touch pad of the smart terminal to perform an input trigger operation. The multi-degree-of-freedom ray can be a 3DOF ray. The ray can be understood as a ray with depth used to indicate the operation position in the display space. The multi-degree-of-freedom ray can be controlled by gestures or by a smart terminal used to control head-mounted glasses.

[0035] The display space may be a three-dimensional space in a head-mounted display device for users to view displayed content. The virtual keyboard may be a pre-configured software keyboard, which is displayed in the display space of the head-mounted display device to assist users in inputting content. The virtual keyboard may include, but is not limited to, the following types of keys: character input type, emoticon input type, emoticon input type, confirmation type, deletion type, and voice input type. The character input type may include, but is not limited to, symbol input type, number input type, and word input type. The key positions of the word input type can be used to input characters or words of the corresponding language type. The language type may include, but is not limited to, Chinese and English. In practice, the above-mentioned execution entity may display the above-mentioned virtual keyboard below the page where the above-mentioned target input box is located or below the above-mentioned target input box in the page.

[0036] Step 102: In response to detecting a first input operation acting on a character input key of the virtual keyboard, determining a character of the character input key in a first input state as an auxiliary input character.

[0037] In some embodiments, the execution subject may determine the character of the character input key in the first input state as an auxiliary input character in response to detecting the first input operation acting on the character input key of the virtual keyboard. Among them, the character input key may be a key corresponding to the character input type. In the virtual keyboard, a character of a symbol input type and a numeric input type may be set in one key, and an uppercase letter and a corresponding lowercase letter of a word input type may be set in one key, so that the user can view or select more optional characters in the currently displayed virtual keyboard, thereby reducing the switching of the keys displayed in the virtual keyboard. The first input operation may be an operation for selecting the character of the character input key in the first input state. For example, the character in the first input state may be the main character corresponding to the character input key. The main character may be a character displayed on the left side of the character input key, or a character displayed in a larger size in the character input key. The first input operation may be a single-finger click operation. For example, the user may slide the pointer in the display space into the character input key through the touch screen and then click to perform the first input operation. For another example, the user can point the multi-degree-of-freedom ray in the display space to the above-mentioned character input key position through the smart terminal, and then click on the touch pad of the smart terminal to perform the first input operation. Thus, the main character corresponding to the character input key position can be selected through the first input operation. The auxiliary input character can be a character selected by the user during the input process.

[0038] Step 103: in response to detecting a second input operation of a character input key corresponding to the virtual keyboard, determining a character of the character input key in the second input state as an auxiliary input character.

[0039] In some embodiments, the execution subject may determine the character of the character input key in the second input state as an auxiliary input character in response to detecting a second input operation corresponding to the character input key of the virtual keyboard. Among them, the second input operation may be an operation for selecting the character of the character input key in the second input state. For example, the character in the second input state may be the secondary character corresponding to the character input key. The secondary character may be the character displayed on the right side of the character input key, or the character displayed in a smaller size in the character input key. It should be noted that the main character may also be the only character displayed in the current character input key, and the corresponding secondary character may be hidden in the character input key. When the user performs the second input operation, the secondary character may be displayed in the form of an icon on either side of the character input key. The second input operation may be a double-finger click operation or a long press operation. For example, the user may slide the pointer in the display space into the character input key through the touch screen and then double-click or long press for a preset time to perform the second input operation. For another example, the user can point the multi-degree-of-freedom ray in the display space of the smart terminal to the above-mentioned character input key position, and then double-click or long-press for a preset time on the touch pad of the smart terminal to perform the second input operation. Thus, the secondary character corresponding to the character input key position can be selected through the second input operation.

[0040] Step 104: display at least one associative word information corresponding to the determined at least one auxiliary input character above the virtual keyboard.

[0041] In some embodiments, the execution subject may display at least one associative word information corresponding to at least one determined auxiliary input character above the virtual keyboard. The at least one associative word information may be information related to each candidate word determined based on all selected auxiliary input characters. For example, the associative word information may be, but is not limited to, characters, words, emoticons, and emoticons. For example, when inputting Chinese, the character selected by the user for the first time may be "w", the character selected for the second time may be "s", the character selected for the third time may be "s", and the character selected for the fourth time may be "d". After the user selects the character "w" for the first time, the associative word information corresponding to "w" may include, but is not limited to, "I", "ten thousand", "position", "no", and "for". After the user selects the character "s" for the second time, the associative word information corresponding to "w" and "s" may include, but is not limited to, "I say", "perfect", "hygiene", "night", "why", and "why" emoticons. After the user selects the character "s" for the third time, the associative word information corresponding to "w", "s", and "s" may include, but is not limited to, "I try", "I calculate", "I mean", "vitamins", and "on my hand". After the user selects the character "d" for the fourth time, the associative word information corresponding to "w", "s", "s" and "d" may include but is not limited to: ten o'clock in the evening, vitamin D, Supreme God Emperor, I am Shandong. It should be noted that for each additional auxiliary input character, the corresponding at least one associative word information will be updated accordingly. Each associative word information can be determined by the candidate word screening method in the input method, and the candidate words, emoticons and emoticons corresponding to the candidate words can be screened according to the user's input habits.

[0042] As an example, the user can select the character input key "n" 204 in the virtual keyboard 203 by single-finger click operation 202 on the touch screen 201, and each associated word information 205 can be displayed above the virtual keyboard 203. When the user inputs content through the virtual keyboard 203, the touch screen 201 does not display the software keyboard.

[0043] As another example, the user can wave the terminal device 301 to anchor the character input key "n" 204 in the virtual keyboard 203 in the ray 302, and then click the touch screen 303 of the terminal device 301, and each associated word information 205 can be displayed above the virtual keyboard 203. When the user inputs content through the virtual keyboard 203, the touch screen 303 does not display the software keyboard.

[0044] Step 105 , in response to detecting a selection operation acting on any displayed associative word information, displaying the any associative word information in a target input box.

[0045] In some embodiments, the execution subject may display the arbitrary associative word information in the target input box in response to detecting a selection operation acting on the displayed arbitrary associative word information. The selection operation may include but is not limited to: clicking, hovering, and dragging. For example, after the user selects the associative word information "10 o'clock in the evening", "10 o'clock in the evening" may be displayed in the target input box.

[0046] Optionally, the execution subject may also detect the first input operation acting on the character input key of the virtual keyboard in response to the detection of a touch operation acting on the associated touch screen, and the indicated position corresponding to the touch operation is in the character input key position, and the touch mode corresponding to the touch operation satisfies the first input condition. The touch screen may be a touch screen on an electronic device with a touch function. For example, the electronic device may include but is not limited to: a touch pad, a mobile phone, a tablet computer, and a smart terminal. The smart terminal may be a portable host of a head-mounted display device to provide computing power and / or power support for it. The first input condition may be that the touch mode is a single-finger click. Here, the specific setting of the first input condition is not limited. The indicated position corresponding to the touch operation may be a pointer indicating the operation position in the display space or a position of a multi-degree-of-freedom ray in the display space. Here, the indicated position may be represented by three-dimensional coordinates. When the indicated position corresponding to the touch operation is in the character input key position, the end of the pointer or the multi-degree-of-freedom ray is displayed in the character input key position. Thus, the user can perform the first input operation through the touch screen.

[0047] Optionally, the execution subject may determine that a second input operation acting on the character input key of the virtual keyboard is detected in response to detecting a touch operation acting on the associated touch screen, and the indicated position corresponding to the touch operation is in the character input key position, and the touch mode corresponding to the touch operation satisfies the second input condition. The second input condition may be that the touch mode is a double-finger click or a long press for a preset time. Here, the specific setting of the second input condition is not limited. Thus, the user can perform the second input operation through the touch screen.

[0048] Optionally, the execution subject may map the touch coordinate sequence corresponding to the sliding operation to the display space of the head-mounted display device in response to detecting the sliding operation acting on the touch screen, and obtain the mapping coordinate sequence. The sliding operation may be an operation of sliding with a finger on the touch screen. When the user performs the sliding operation, a single finger slides on the touch screen, and the touch coordinates may be the coordinates of the center point of the contact surface. When the user performs the sliding operation, two fingers slide on the touch screen, and the touch coordinates may be the coordinates of the center points of the two contact surfaces. The touch coordinate sequence may be each touch coordinate at a continuous time collected at a preset frequency. In practice, the execution subject may map the touch coordinate sequence corresponding to the sliding operation to the display space of the head-mounted display device by using a coordinate transformation method, and obtain the mapping coordinate sequence. Then, the indicated position of the indicating element may be moved in the display space of the head-mounted display device according to the obtained mapping coordinate sequence. The indicating element may be a pointer, a cursor, or a multi-degree-of-freedom ray. Thus, the user may control the operation position in the display space by sliding on the touch screen.

[0049] Optionally, the execution subject may map the spatial coordinate sequence corresponding to the waving operation to the display space of the head-mounted display device in response to detecting the waving operation of the terminal device corresponding to the touch screen, and obtain a mapping coordinate sequence. The terminal device may be, but is not limited to, a touchpad, a mobile phone, a tablet computer, and a smart terminal. An inertial measurement unit may be provided in the terminal device. The waving operation may be an operation of waving the terminal device in the air. The spatial coordinates may be relative coordinates relative to the head-mounted glasses of the head-mounted display device. The spatial coordinate sequence may be each spatial coordinate in a continuous time collected at a preset frequency. In practice, the execution subject may map the mapping coordinate sequence to the world coordinate system of the head-mounted display device by a coordinate transformation method, and obtain a mapping coordinate sequence. The world coordinate system may be constructed by a SLAM algorithm. Then, the indicated position of the indicating element may be moved in the display space of the head-mounted display device according to the obtained mapping coordinate sequence. Here, the indicating element may be a multi-degree-of-freedom ray. Thus, the user may control the operation position in the display space by waving the terminal device.

[0050] Optionally, the execution subject may determine each character of each character input key position in the first input state as an auxiliary input character sequence in response to detecting a first sliding input operation acting on each character input key position of the virtual keyboard. Among them, each character input key position corresponds to a sliding order. For example, the user can control the indicating element to successively stroke each character input key position. The first sliding input operation may be an operation of sliding a single finger on the touch screen so that the indicating element strokes each character input key position. During the single-finger sliding process, the indicating element may be displayed as a pointer or a cursor. The first sliding input operation may also be an operation of contacting the touch screen of a terminal device with a single finger and waving the terminal device so that the indicating element strokes each character input key position. During the waving process, the indicating element may be displayed as a multi-degree-of-freedom ray. For example, the multi-degree-of-freedom ray may be a 3DOF ray.

[0051] Then, the execution entity may display at least one sliding associative word information corresponding to the auxiliary input character sequence above the virtual keyboard. Each sliding associative word information may be determined by a candidate word screening method in the input method, and the candidate words, emoticons and emoticons corresponding to the candidate words may be screened according to the user's input habits. Finally, in response to detecting a selection operation acting on any sliding associative word information displayed, the arbitrary sliding associative word information may be displayed in the target input box. Thus, the user may directly select multiple characters in the first state continuously in a sliding manner, and select the input content to be input into the target input box according to the recommended sliding associative word information corresponding to each selected character.

[0052] As an example, a schematic diagram of sliding a single finger on a touch screen to perform a first sliding input operation can be referred to as follows: Figure 4 . Figure 4 In the embodiment, the user continuously slides on the touch screen 201 in a certain sliding manner to perform a first sliding input operation. The virtual keyboard 203 may display a sliding path 401 corresponding to the first sliding input operation, and determine the character input keys crossed by the sliding path as the character input keys acted upon by the first sliding input operation. For example, the sliding path 401 may cross the characters "w", "s", "z", "x", "c", "f", "g", "f", and "r". Then, the sliding association word information 402 may be displayed above the virtual keyboard 203. The sliding path of the user on the touch screen 201 may be 403, Figure 4 The sliding path 403 in FIG. 2 only indicates the sliding method of the user and is not displayed on the touch screen 201 .

[0053] As another example, a schematic diagram of touching the touch screen of the terminal device with a single finger and waving the terminal device to perform the first sliding input operation can refer to Figure 5 . Figure 5In the embodiment, the user can touch the touch screen 303 of the terminal device 301 with a single finger, and wave the terminal device 301 to continuously anchor the ray 302 to each character input key position to perform the first sliding input operation. The virtual keyboard 203 can display the waving path 501 corresponding to the first sliding input operation, and determine the various character input keys drawn by the waving path 501 as the various character input keys acted on by the first sliding input operation. For example, the waving path 501 can draw the characters "w", "s", "z", "x", "c", "f", "g", "f" and "r" in the first input state. Then, each sliding association word information 502 can be displayed above the virtual keyboard 203.

[0054] Optionally, the execution subject may determine each character of each character input key position in the second input state as an auxiliary input character sequence in response to detecting a second sliding input operation acting on each character input key position of the virtual keyboard, wherein each character input key position corresponds to a sliding order. The second sliding input operation may be an operation of sliding two fingers on the touch screen so that the indicating element strokes each character input key position. During the two-finger sliding process, the indicating element may be displayed as a pointer or a cursor. The second sliding input operation may also be an operation of contacting the touch screen of a terminal device with two fingers and waving the terminal device so that the indicating element strokes each character input key position. During the waving process, the indicating element may be displayed as a multi-degree-of-freedom ray. For example, the multi-degree-of-freedom ray may be a 3DOF ray.

[0055] Then, the execution subject may display at least one sliding associative word information corresponding to the determined auxiliary input character sequence above the virtual keyboard. In response to detecting a selection operation acting on any sliding associative word information displayed, the any sliding associative word information is displayed in the target input box. Thus, the user can directly select multiple characters in the second state continuously in a sliding manner, and select the input content to be input into the target input box according to the recommended sliding associative word information corresponding to each selected character.

[0056] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the input interaction method of the virtual keyboard of some embodiments of the present disclosure, the efficiency of user input of characters is improved, the input operation is simplified, and the user experience is improved. Specifically, the reason for the poor user experience is that when the user frequently inputs characters in different states, it is necessary to frequently click the switch key to switch the characters indicated by the key position, resulting in low input efficiency and complex input operations, thereby resulting in poor user experience. Based on this, the input interaction method of the virtual keyboard of some embodiments of the present disclosure, first, in response to detecting the input trigger operation corresponding to the target input box, the virtual keyboard is displayed in the display space of the head-mounted display device. Thus, the virtual keyboard can be displayed in the display space of the head-mounted display device for the user to watch. Then, in response to detecting the first input operation acting on the character input key position of the above-mentioned virtual keyboard, the character of the character input key position in the first input state is determined as an auxiliary input character. Thus, when the user performs the first input operation, the character input in the first input state can be determined as the character the user wants to input. Secondly, in response to detecting the second input operation corresponding to the character input key position of the above-mentioned virtual keyboard, the character of the character input key position in the second input state is determined as an auxiliary input character. Thus, when the user performs the second input operation, the character input in the second input state can be determined as the character that the user wants to input. Then, at least one associative word information corresponding to at least one determined auxiliary input character is displayed above the above-mentioned virtual keyboard. Thus, the corresponding associative word information can be determined according to at least one character input by the user. Finally, in response to detecting a selection operation acting on any associative word information displayed, the above-mentioned any associative word information is displayed in the above-mentioned target input box. Thus, after the user selects the associative word information, the selected associative word information can be used as the input content of the target input box. Also because the user can directly perform different input operations on the character input key position to input characters in different input states, there is no need to click the switch key every time the character state is switched, thereby improving the efficiency of the user inputting characters, simplifying the input operation, and improving the user experience.

[0057] Reference below Figure 6 , which shows a schematic structural diagram of a head-mounted display device 600 suitable for implementing some embodiments of the present disclosure. Figure 6 The head mounted display device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0058] like Figure 6As shown, the head mounted display device 600 may include a processing device 601 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the head mounted display device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0059] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; and a communication device 609. The communication device 609 may allow the head mounted display device 600 to communicate with other devices wirelessly or by wire to exchange data. Figure 6 The head mounted display device 600 having various devices is shown, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0060] The head mounted display device 600 further includes a display module for imaging in front of the user's eyes. The display module may include an optical element and a micro display. The optical element is used to image the content displayed in the micro display in front of the user's eyes.

[0061] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0062] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0063] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0064] The computer-readable medium may be included in the head-mounted display device; or it may exist independently without being assembled into the head-mounted display device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the head-mounted display device, the head-mounted display device: in response to detecting an input trigger operation corresponding to a target input box, displays a virtual keyboard in the display space of the head-mounted display device; in response to detecting a first input operation acting on a character input key position of the virtual keyboard, determines the character of the character input key position in the first input state as an auxiliary input character; in response to detecting a second input operation corresponding to the character input key position of the virtual keyboard, determines the character of the character input key position in the second input state as an auxiliary input character; displays at least one associative word information corresponding to at least one determined auxiliary input character above the virtual keyboard; in response to detecting a selection operation acting on any associative word information displayed, displays the above arbitrary associative word information in the above target input box.

[0065] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0066] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0067] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0068] Some embodiments of the present disclosure further provide a computer program product, including a computer program, which implements any of the above-mentioned virtual keyboard input interaction methods when executed by a processor.

[0069] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A virtual keyboard input interaction method, applied to a head mounted display device, include: In response to detecting an input triggering operation corresponding to a target input box, displaying a virtual keyboard in a display space of the head mounted display device; In response to detecting a first input operation acting on a character input key of the virtual keyboard, determining a character at the character input key in a first input state as an auxiliary input character; In response to detecting a second input operation corresponding to the character input key position of the virtual keyboard, determining the character of the character input key position in the second input state as an auxiliary input character; Displaying at least one associative word information corresponding to the determined at least one auxiliary input character above the virtual keyboard; In response to detecting a selection operation acting on any displayed associative word information, the any associative word information is displayed in the target input box.

2. The method according to claim 1, in, The method further comprises: In response to detecting a touch operation acting on an associated touch screen, and the indicated position corresponding to the touch operation is in a character input key position, and the touch mode corresponding to the touch operation satisfies a first input condition, it is determined that a first input operation acting on a character input key position of the virtual keyboard is detected.

3. The method according to claim 1, in, The method further comprises: In response to detecting a touch operation acting on the associated touch screen, and the indicated position corresponding to the touch operation is in the character input key position, and the touch mode corresponding to the touch operation satisfies the second input condition, it is determined that a second input operation acting on the character input key position of the virtual keyboard is detected.

4. The method according to claim 2 or 3, in, The method further comprises: In response to detecting a sliding operation on the touch screen, mapping a touch coordinate sequence corresponding to the sliding operation to a display space of the head mounted display device to obtain a mapping coordinate sequence; According to the obtained mapping coordinate sequence, the indicated position of the indicating element is moved in the display space of the head mounted display device.

5. The method according to claim 2 or 3, in, The method further comprises: In response to detecting a waving operation of a terminal device corresponding to the touch screen, mapping a spatial coordinate sequence corresponding to the waving operation to a display space of the head mounted display device to obtain a mapping coordinate sequence; According to the obtained mapping coordinate sequence, the indicated position of the indicating element is moved in the display space of the head mounted display device.

6. The method according to claim 1, in, The method further comprises: In response to detecting a first sliding input operation acting on each character input key position of the virtual keyboard, determining each character of each character input key position in a first input state as an auxiliary input character sequence, wherein each character input key position corresponds to a sliding order; Displaying at least one sliding associative word information corresponding to the auxiliary input character sequence above the virtual keyboard; In response to detecting a selection operation acting on the displayed arbitrary sliding associative word information, the arbitrary sliding associative word information is displayed in the target input box.

7. The method according to claim 1, in, The method further comprises: In response to detecting a second sliding input operation acting on each character input key position of the virtual keyboard, determining each character of each character input key position in the second input state as an auxiliary input character sequence, wherein each character input key position corresponds to a sliding sequence; Displaying at least one sliding associative word information corresponding to the determined auxiliary input character sequence above the virtual keyboard; In response to detecting a selection operation acting on the displayed arbitrary sliding associative word information, the arbitrary sliding associative word information is displayed in the target input box.

8. A head mounted display device, include: one or more processors; A display module, used to form an image in front of the user's eyes; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

9. A computer readable medium having a computer program stored thereon, in, When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.