An efficient text input method for virtual reality

Abstract

The present invention relates to an efficient virtual reality text input method, a single-handle text input method based on a newly designed flower-shaped keyboard, using 3D hand displacement interaction for letter selection, and conducting user studies to optimize the keyboard design and the mapping between interaction and selection, and evaluating the method of the present invention. The results show that compared with the most advanced handle-based text input method, the method of the present invention has higher typing speed, lower error rate, and is very friendly to novices. After a short training, the typing speed of the novice group reached 17.65WPM, and the typing speed of the potential expert group reached 22.97WPM. The highest typing speed was 30.80WPM achieved by a participant in the potential expert group.

Description

Technical Field

[0001] The invention relates to an efficient virtual reality text input method, belonging to the technical field of human-computer interaction. Background Art

[0002] Text input is a common and important task in many virtual reality (VR) scenarios, such as collaborative communication between multiple users and labeling scene information.

[0003] Many researchers have explored how to perform text input efficiently and conveniently in the context of VR applications. The most direct method is to connect a physical keyboard. This method uses MR technology to visualize the physical keyboard and the user's hands to the user in a virtual or real way, which can achieve a very high typing speed. However, this method introduces external devices and cannot be used in mobile scenarios.

[0004] In order to solve the text input problem in mobile scenarios, touch screen-based technology has been proposed. This method usually requires access to a touch screen, such as a tablet, mobile phone, or smart watch. However, the typing efficiency of this method is not high and it is difficult to integrate into existing VR systems.

[0005] Another solution is a voice-based method, which can achieve a very high typing speed and can be easily accepted by users. However, its problem is that it is difficult to correct errors, and the accuracy of voice recognition will be affected in noisy outdoor environments. At the same time, the user's personal privacy cannot be guaranteed.

[0006] Mid-air typing and head-based technologies have also been proposed. Mid-air typing usually uses the recognition of hand movements for input, which requires relatively expensive sensor equipment. Head-based technology usually uses hand movements to make selections, which can easily cause dizziness in users in a VR environment.

[0007] Many of the existing text input techniques mentioned above require additional equipment that is complex and expensive. The handle controller is currently the most common VR interaction device, so it is a potential candidate input device for text input in virtual environments (VEs). Many researchers have explored the efficiency, learnability, and usability of handle-based text input techniques. PizzaText is a two-handed joystick-based text input technique with a circular keyboard layout. After two hours of training, expert users can type at a speed of 15.85WPM. The drum-like keyboard is also two-handed, borrowing the idea of ​​ray selection, with a total error rate of 7.2%. Compared with two-handed, one-handed technology can free up one hand to do other interactive tasks. HiPad is a one-handed text input technique that uses a handle with a touchpad for text input on a circular keyboard. Novices can reach a typing speed of 13.57WPM after training on 60 phrases. However, these methods have high error rates (drum-like keyboard), require devices with joysticks (PizzaText), and require devices with touchpads (HiPad). Summary of the invention

[0008] The technical problem solved by the present invention is to overcome the deficiencies of the prior art and provide an efficient virtual reality text input method to achieve high typing speed, low error rate and good learnability.

[0009] The technical solution adopted by the present invention is: an efficient virtual reality text input method, comprising the following steps:

[0010] Step (1), designing a fancy text input keyboard layout, specifically implemented as follows: dividing the classic QWERTY keyboard layout into 6 parts, the upper two parts are "QWERT" and "YUIOP", the middle two parts are "ASDF" and "GHJKL", and the lower two parts are "ZXCV" and "BNM", and retaining the form of one key for each letter; rotating the "QWERT" and "ZXCV" parts 60 degrees upward and downward around the right endpoint, rotating the "YUIOP" and "BNM" parts 60 degrees upward and downward around the left endpoint, and finally placing the space bar in the middle of the "ASDF" and "GHJKL" parts to form a "flower-shaped" keyboard layout; this layout design retains the advantages of a circular keyboard, minimizes the displacement required when selecting letters, and is a completely new keyboard layout form;

[0011] Step (2) is to design a fancy text input keyboard interaction, and use the 3D displacement of the hand and a single key to make a selection. The specific implementation is as follows: for the detection of the 3D displacement of the hand, first use the Grib button on the right handle of the Vive Focus Controller to establish an interactive coordinate system. Pressing the Grib button means taking the 3D spatial position of the current handle as the origin, the positive forward direction as the negative direction of the z-axis, the vertical upward direction as the positive direction of the y-axis, and the vertical right direction as the positive direction of the x-axis to establish an interactive coordinate system. In this coordinate system, the user moves the handle to interact. In each frame of the program execution, the vector from the origin of the coordinate system to the user's hand position is converted into Split into displacement distance and direction vector The displacement distance is a vector The Euclidean norm of Direction vector is vector Two-dimensional vector projected onto the XOY plane The direction vector is used to guide the selection of the six partial keyboards in (1), where the direction range is [-30°, 30°] to select letters in "GHJKL", the direction range is [30°, 90°] to select letters in "YUIOP", the direction range is [90°, 150°] to select letters in "QWERTY", the direction range is [150°, -150°] to select letters in "ASDF", the direction range is [-150°, -90°] to select letters in "ZXCV", and the direction range is [-90°, -30°] to select letters in "BNM". The displacement distance is mapped by a piecewise linear function in six directions. The function on [-30°, 30°] is: The function on [30°, 90°] is The function on [90°, 150°] is The function on [150°, -150°] is The function on [-150°, -90°] is The function on [-90°, -30°] is Where x is the displacement distance f(x) is the distance on the keyboard in the virtual scene, and the length of each key on the keyboard in the virtual scene is 0.16.

[0012] During the interaction process, the fancy text input keyboard also sets four optimization options: key highlighting, mapping position visualization, word prediction and error correction, and sound and handle vibration feedback; in terms of button selection, a single ViveFocus Controller right handle is used, where the B button is used to confirm the selection, the A button is used to backspace, the Trigger button is used to select the predicted word, and the Grib button is used to establish the interactive coordinate system.

[0013] The advantages of the present invention compared with the prior art are:

[0014] (1) The present invention splits the classic QWERTY keyboard into 6 parts, rotates the upper and lower parts, and places the space bar in the center of the keyboard, thereby forming a flower-shaped keyboard. In order to improve the user's sensitivity and familiarity with the keyboard, the present invention applies four optimization options: key highlighting, mapping position visualization, word prediction and error correction, and sound and handle vibration feedback. By analyzing the user's hand displacement vector, an optimized and reasonable mapping function is designed to map them to the keyboard for the user to perform the next selection operation.

[0015] (2) The method of the present invention is a single-handle text input technology, which takes into account the following aspects in design: fast typing speed, low error rate, using only simple handles and buttons (no joystick or touchpad), and users can learn more naturally and easily. It can not only meet the needs of efficient text input, but also free up the other hand to do other interactive tasks. It can also be easily integrated into the existing VR system without the introduction of external equipment, and the learning cost is very low.

[0016] (3) Compared with the existing methods, the typing speed of the present invention is faster and the error rate is lower: the typing speed of the novice group of the present invention is 17.65WPM, and the typing speed of the potential expert group is 22.97WPM. The error rates NCER and TER of the novice group are 0.09% and 2.50%, respectively, and the error rates NCER and TER of the potential expert group are 0.04% and 3.42%, respectively. The current latest two-hand handle method PizzaText has a novice group speed of 8.59WPM, a potential expert group speed of 15.85WPM, and the error rates NCER and TER of the novice group are 1.56% and 5.91%, respectively, and the error rates NCER and TER of the potential expert group are 1.59% and 5.08%, respectively; the current latest one-hand handle method HiPad has a novice group speed of 13.57WPM, and the error rate NCER of the novice group is 0.22%. It can be seen that our method has a faster typing speed and a lower error rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flow chart for realizing the method of the present invention;

[0018] Figure 2 It is the effect diagram of the typing process of the present invention;

[0019] Figure 3 A diagram of the process of inputting the letter 'a' for the present invention;

[0020] Figure 4 A diagram showing the process of inputting the letter 'r' for the present invention;

[0021] Figure 5 generating a schematic diagram for a layout of the present invention;

[0022] Figure 6 A schematic diagram of a function button provided on a handle of the present invention;

[0023] Figure 7 is a schematic diagram of the mapping method;

[0024] Figure 8 Schematic diagram for different optimization options;

[0025] Fig. 9 A schematic diagram of the user's landing data;

[0026] Fig.10 It is a schematic diagram of the landing point data in the center area of ​​the keyboard;

[0027] Fig.11 A table for recalculating mapping coefficients for each key;

[0028] Fig.12 The performance test result diagram of the present invention. The left figure is a graph showing the change of typing speed over time for 10 participants in a 6-day learning experiment; the middle figure is a graph showing the change of average typing speed over time for the two groups after the 10 participants were divided into a novice group and an expert group; the right figure is a graph showing the change of NCER and TER over time for the two groups. DETAILED DESCRIPTION

[0029] like Figure 1 As shown, the implementation process of the method of the present invention is as follows:

[0030] Step (1), layout design, divide the classic QWERTY keyboard layout into 6 parts, the upper two parts are "QWERT" and "YUIOP", the middle two parts are "ASDF" and "GHJKL", the lower two parts are "ZXCV" and "BNM", and retain the form of one key for each letter; rotate the "QWERT" and "ZXCV" parts 60 degrees upward and downward around the right end point, rotate the "YUIOP" and "BNM" parts 60 degrees upward and downward around the left end point, and finally place the space bar in the middle of the "ASDF" and "GHJKL" parts to form a "flower-shaped" keyboard layout. The characteristics of the keyboard layout are: one key for each letter; the order of letter arrangement remains unchanged; the keyboard is divided into two left and right areas; this layout design retains the advantages of a circular keyboard: minimizing the displacement required when selecting letters, and this is a new keyboard layout form.

[0031] Step (2), interaction design, we use 3D hand displacement and single button to make selections. First, use the Grib button on the right handle of the ViveFocus Controller to establish an interactive coordinate system. Pressing the Grib button means that the 3D spatial position of the current handle is the origin, the positive direction is the negative direction of the z-axis, the vertical upward direction is the positive direction of the y-axis, and the vertical right direction is the positive direction of the x-axis. An interactive coordinate system is established. In this coordinate system, the user moves the handle to interact. In each frame of the program execution, the vector from the origin of the coordinate system to the user's hand position is converted to Split into displacement distance and direction vector The displacement distance is a vector The Euclidean norm of Direction vector is vector Two-dimensional vector projected onto the XOY plane The direction vector is used to guide the selection of the six partial keyboards in (1), where the direction range is [-30°, 30°] to select letters in "GHJKL", the direction range is [30°, 90°] to select letters in "YUIOP", the direction range is [90°, 150°] to select letters in "QWERTY", the direction range is [150°, -150°] to select letters in "ASDF", the direction range is [-150°, -90°] to select letters in "ZXCV", and the direction range is [-90°, -30°] to select letters in "BNM"; in terms of distance mapping, the user's landing point data during typing was collected and analyzed, and it was found that the user's movements in each direction are not completely linear. Therefore, based on the information obtained from the data analysis, we designed a piecewise linear function in each direction. The function on [-30°, 30°] is The function on [30°, 90°] is The function on [90°, 150°] is The function on [150°, -150°] is The function on [-150°, -90°] is The function on [-90°, -30°] is Where x is the displacement distance f(x) is the distance on the keyboard in the virtual scene, and the length of each key on the keyboard in the virtual scene is 0.16. During the interaction process, four optimization options are set: key highlighting, mapping position visualization, word prediction and error correction, and sound and handle vibration feedback. The above are all innovative interaction designs. After testing, these optimization options can significantly increase typing speed and reduce workload.

[0032] Step (3), speed and learning test, a 6-day learning experiment was conducted, and the experimental participants were divided into a novice group and a potential expert group, wherein the novice group participants had not used our invention, while the potential expert group participants had used our invention. The experimental results showed that the typing speed of the novice group reached 17.65WPM on the last day, an increase of 96.99% compared to the first day, and the typing speed of the potential expert group reached 22.97WPM on the last day, an increase of 23.56% compared to the first day. These data show that the method of the present invention is very efficient, has good learning properties, and is very friendly to novice users.

[0033] like Figure 2As shown. In the VR scene, the user can see a flower-shaped keyboard with three keys of different colors. They are distinguished according to the frequency of use of English letters. They are three different colors from high frequency to low frequency, which is used to speed up the user's letter search efficiency; during the typing process, the user can see a highlighted point representing the current hand mapping to the position on the keyboard to assist the user in making a selection; the method will also make predictions based on the currently input letter string, and the user can complete and correct errors by selecting recommended words, thereby improving the typing speed and reducing the error rate; when the user presses the selection key on the handle, a sound similar to keyboard tapping in reality will be heard, and when the delete key is pressed, a Windows-style delete sound will be heard, and when the user controls the handle to move on the keyboard, the vibration of the handle can be felt corresponding to the switching of the candidate keys on the keyboard. These feedbacks can improve the accuracy of selection and optimize the user experience. In terms of the mapping of 3D hand displacement, the present invention collects and analyzes the user's landing point data, optimizes it, proposes an optimized mapping function, and achieves good results. The main innovations of the present invention are: (1) a new flower-shaped keyboard is designed. (2) 3D hand displacement interaction is introduced into the handle-based text input technology. (3) The mapping between interaction and selection is optimized according to user behavior, and a piecewise linear mapping function that distinguishes directions is proposed.

[0034] Figure 3 and Figure 4 The process diagram of inputting letters 'a' and 'r' is given. When the user holds the handle and moves it from the center origin position to the left end position, the letter 'a' is selected for input ( Figure 3 ), then move the handle to the upper left position and select the letter "r" to enter ( Figure 4 ).

[0035] The present invention is further described below in conjunction with other drawings and specific embodiments.

[0036] 1. Keyboard layout design

[0037] like Figure 5 As shown in the figure, the layout of the flower keyboard combines the common features of the QWERTY keyboard and the circular keyboard, dividing the classic QWERTY keyboard into 6 parts, among which the 'G' key originally belonged to the left keyboard part, but in order to balance the number of keys in the left and right directions, it is divided into the right keyboard part, and then the upper and lower parts are rotated, and the space bar is placed in the center of the entire keyboard, thus forming a flower-shaped keyboard.

[0038] 2. Interactive implementation

[0039] Figure 6The function keys used on a single handle are shown. We introduce 3D hand movement interaction into the method of the present invention. Before the typing task begins, we first need to establish an interaction coordinate system, such as Figure 7 As shown, the hand displacement vector Project the hand displacement vector (from the origin O of the interactive coordinate system to the handle position P) onto the XY plane Then calculate the vector The length of the hand displacement Where ‖·‖2 represents the Euclidean norm. Then, the hand displacement binary is used For the selection of keyboard keys, where Directly used to indicate direction, The current displacement of the hand is projected onto the keyboard through a mapping function, and the mapping distance p is calculated according to formula (1), where k represents the mapping coefficient, and then a selection is completed through the selection key on the handle.

[0040]

[0041] The typing speed, typing error rate and workload are used for measurement. The typing speed is calculated using formula (2), where T is the target phrase, S is the time between the first and last press in each phrase (in seconds); the error rate is Total Error Rate (TER) = Not Corrected Error Rate (NCER) + Corrected Error Rate (CER); and the workload is scored using the NASA-TLX questionnaire.

[0042]

[0043] Four optimization options have been added to the keyboard: key highlighting, mapping position visualization, word prediction and error correction, and sound and controller vibration feedback (such as Figure 8 After experimentation, it was found that they all significantly increased typing speed and reduced workload.

[0044] 3. Collection and analysis of landing point data

[0045] Five different linear mapping functions were tested, and the landing point data during the experiment were collected. The data of the mapping function with the best speed performance were analyzed, and the data were visualized using formula (3) (e.g. Fig. 9As shown in the figure, the semi-transparent ellipses of different colors in the figure are confidence ellipses with a confidence level of 95% that each key contains data. The data in the figure also includes points that were incorrectly selected. For example, when a user wants to select 'H', they may mistakenly select the adjacent 'G' key, and then have to delete 'G' and reselect 'H'. Such incorrect selection points are added to the set of 'H'. The above type of data can better reflect user behavior.

[0046]

[0047] like Fig.10 As shown in FIG. 1 , it is found that the center of the space bar deviates from the expected coordinate O(0, 0) and becomes O'(0.0003, 0.0012), which indicates that the user's perception of the center is somewhat offset. Therefore, the mapping coefficient is recalculated for each letter key according to formula (4), where is the new keyboard center O' to the confidence ellipse center C of each key i Len is the distance from the center of the keyboard to the center of each key in the virtual scene. The present invention does not consider the outermost keys in each direction, because people usually do not consider the distance when pressing the outer keys, and map the distance greater than a certain range to the outermost keys. The calculation result is as follows Fig.11 As shown in the figure, we can see that the mapping coefficients k of the keys around the center of the keyboard are all less than 4.5, while the k values ​​of the keys outside this range are all greater than or equal to 4.5. Therefore, in each direction, two k values ​​are used to scale the hand displacement, that is, within the range of the space bar and the surrounding keys in the virtual scene, two k values ​​are used to scale the hand displacement. Fig.11 The k value in the first column is used as the mapping coefficient. If it exceeds this range, use Fig.11 The k value in the last column of is used as the mapping coefficient. This optimized piecewise linear mapping was tested, and the experiment showed that it can significantly improve the user's typing speed and reduce the error rate.

[0048]

[0049] 4. Evaluation of the performance of the present invention

[0050] A 6-day user experiment was designed to evaluate the performance of our invention. Five novice participants were invited to form a novice group, and five participants who performed well in previous experiments were invited to form a potential expert group. Every day, they were required to conduct a speed test experiment with a length of 10 phrases. The experimental results are shown in the figure below. Fig.12As shown, the typing speed of the novice group reached 17.65WPM on the last day, an increase of 96.99% compared to the first day, while the typing speed of the potential expert group reached 22.97WPM on the last day, an increase of 23.56% compared to the first day, and the NCER and TER of the novice group were 0.09% and 2.50%, respectively, and the NCER and TER of the potential expert group were 0.04% and 3.42%, respectively. The current latest two-hand handle method PizzaText has a maximum speed of 15.85WPM, and the NCER and TER of the novice group are 1.56% and 5.91%, respectively, and the NCER and TER of the potential expert group are 1.59% and 5.08%, respectively; the current latest one-hand handle method HiPad has a maximum speed of 13.57WPM, of which the NCER of the novice group is 0.22%. After comparison, it can be found that the method of the present invention can achieve efficient typing, low error rate, and good learnability.

[0051] The hardware configuration used in the experiment is Intel i7-10700KF processor, Nvidia GeForce 3080Ti graphics card, 16G memory. It runs on Windows 10 64-bit system.

[0052] The technical contents not elaborated in detail in the present invention belong to the common knowledge of those skilled in the art.

[0053] Although the above describes the illustrative specific embodiments of the present invention to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. An efficient virtual reality text input method, characterized in that The following steps are involved: Step (1), designing a fancy text input keyboard layout, specifically implemented as follows: dividing the classic QWERTY keyboard layout into 6 parts, the upper two parts are "QWERT" and "YUIOP", the middle two parts are "ASDF" and "GHJKL", and the lower two parts are "ZXCV" and "BNM", and retaining the form of one key for each letter; rotating the "QWERT" and "ZXCV" parts 60 degrees upward and downward around the right endpoint, respectively, rotating the "YUIOP" and "BNM" parts 60 degrees upward and downward around the left endpoint, respectively, and finally placing the space bar in the middle of the "ASDF" and "GHJKL" parts to form a "flower-shaped" keyboard layout; Step (2) is to design a fancy text input keyboard interaction, using the hand 3D displacement and single button press to make selections. The specific implementation is as follows: for the detection of the hand 3D displacement, first use the Grib button on the right handle of the Vive Focus Controller to establish an interactive coordinate system. Pressing the Grib button means that the 3D spatial position of the current handle is taken as the origin, the positive forward direction is the negative direction of the z-axis, the vertical upward direction is the positive direction of the y-axis, and the vertical right direction is the positive direction of the x-axis. An interactive coordinate system is established. In this coordinate system, the user moves the handle to interact. In each frame of the program execution, the vector from the origin of the coordinate system to the user's hand position is converted into Split into displacement distance and direction vector , where the displacement distance is a vector The Euclidean norm of , the direction vector is the vector Two-dimensional vector projected onto the XOY plane , the direction vector is used to guide the selection of the 6 partial keyboards in step (1), where the direction range is [-30°, 30°] to select letters in "GHJKL", the direction range is [30°, 90°] to select letters in "YUIOP", the direction range is [90°, 150°] to select letters in "QWERTY", the direction range is [150°, -150°] to select letters in "ASDF", the direction range is [-150°, -90°] to select letters in "ZXCV", and the direction range is [-90°, -30°] to select letters in "BNM"; the displacement distance is mapped by a different piecewise linear function in the 6 directions, and the function on [-30°, 30°] is , the function on [30°, 90°] is , the function on [90°, 150°] is , the function on [150°, -150°] is , the function on [-150°, -90°] is , the function on [-90°, -30°] is , where x is the displacement distance , f(x) is the distance on the keyboard in the virtual scene, and the length of each key on the keyboard in the virtual scene is 0.

16.

2. The efficient virtual reality text input method according to claim 1, characterized in that: During the interaction process, the fancy text input keyboard also sets four optimization options: key highlighting, mapping position visualization, word prediction and error correction, and sound and handle vibration feedback; in terms of button selection, a single Vive Focus Controller right handle is used, where the B button is used to confirm the selection, the A button is used to backspace, the Trigger button is used to select the predicted word, and the Grib button is used to establish the interaction coordinate system.

Images