Intelligent text input system based on virtual reality

Abstract

The invention belongs to the technical field of virtual reality, and particularly relates to an intelligent text input system based on virtual reality, which comprises a user input module used for acquiring keystroke details of a user when the user clicks a target key by using a virtual finger; the force description module is used for carrying out force calculation according to the Newton second law when the user clicks the key; and the force driving and compensation mechanism module is used for performing visual feedback of automatic compensation and force driving according to the force calculated by the force description module. According to the method, the user experience of text input in the virtual reality equipment can be improved to a great extent; in the VR environment, if the visual feedback can change according to the knocking force, the physical behavior of the real world can be simulated, so that the user feels like using a real keyboard.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, in particular to an intelligent text input system based on virtual reality. Background Art

[0002] In recent years, the ability of VR systems to track physical hands has made great progress, which has made bare-hand VR interaction more popular. That is, it is possible to interact with virtual objects bare-handed without additional hardware support. Bare-hand VR interaction is a promising interaction paradigm due to its convenience and directness. However, due to the lack of tactile feedback, bare-hand VR interaction in most VR systems is still not natural and realistic enough.

[0003] In virtual reality (VR) systems, users often cannot feel the physical restrictions or resistance when touching virtual objects, which affects the naturalness and intuitiveness of the interaction. However, tactile feedback mainly relies on additional hardware to generate. These hardware devices are very cumbersome or expensive to integrate into VR systems. Some researchers apply the concept of pseudo-tactile to compensate for the lack of kinesthetic feedback without the need for additional hardware. For example, using a virtual hand to touch a virtual object, when the user's real hand enters the inner space of the virtual object, the pseudo-tactile system will visually constrain the virtual hand to the surface of the virtual object. Although the concept of pseudo-tactile can induce the illusion of kinesthetic feedback. However, due to the obvious difference in the position and posture of the virtual hand and the corresponding real hand, this concept may reduce the user's interaction performance.

[0004] In this regard, Kine-Appendage provides an innovative solution, which provides visual motion feedback by adding virtual appendages to the user's virtual hand and showing visual transformation when touching the virtual object, so that the hand can produce a sense of restriction or resistance when touching the object. BrittleStylus technology, as an application of Kine-Appendage technology, provides the special effect of virtual pen tip breaking when the user is typing, which can significantly reduce the typing error rate without sacrificing typing speed. In addition, it can also reduce the possibility of the user's hand penetrating the virtual keyboard and improve the realism of the interaction. However, when users use BrittleStylus technology, the real feedback is not strong enough and the immersion is insufficient. Because in the real world, we type with different strengths, which must be accompanied by different tactile feedback, so the same broken visual feedback does not conform to people's intuitive feelings. At the same time, the hand tracking system of VR devices cannot completely and accurately capture every movement of the user, especially when typing quickly. In the VR environment, a slight error or delay in the hand position may make it impossible to accurately recognize each key press. Therefore, the device often misses keys, which causes the VR device to be unable to recognize every input of the user, which affects the user's input experience. Therefore, an intelligent text input system based on virtual reality is invented. Summary of the invention

[0005] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0006] An intelligent text input system based on virtual reality, comprising:

[0007] A user input module, used to obtain the keystroke details of the user when the user clicks the target key with a virtual finger;

[0008] The force characterization module is used to calculate the force according to Newton's second law when the user presses the key;

[0009] A force driving and compensation mechanism module, used for automatic compensation and force-driven visual feedback according to the force calculated by the force characterization module;

[0010] The force driving and compensation mechanism module includes:

[0011] An automatic compensation mechanism module, used for performing force threshold detection, missing key search and compensation input execution in sequence according to the force calculated by the force characterization module;

[0012] The force-driven visual feedback module is used to perform dynamic visual feedback and custom adjustment coefficients in turn according to the force calculated by the force characterization module.

[0013] As a preferred solution of the virtual reality-based intelligent text input system of the present invention, the specific steps of the force characterization module are as follows:

[0014] S21: Calculate the typing force F=ma according to Newton's second law, where m is the mass of the finger and a is the acceleration; the calculation formula of a is as follows:

[0015] a=(v2 2 -v1 2 ) / (2lenth)

[0016] Among them, the distance lenth is the distance between the keyboard and a vertical board in front of the keyboard with the same size as the keyboard, v1 is the speed of passing through the vertical board, and v2 is the speed of passing through the keyboard. The calculation formulas of v1 and v2 are as follows:

[0017] v1=d1 / t1

[0018] v2=d2 / t2

[0019] Wherein, t1 and t2 are the time to pass through the vertical plate and the time to pass through the keyboard, respectively; d1 and d2 are the thickness of the vertical plate and the thickness of the keyboard, respectively;

[0020] S22: approximate the acceleration a to the instantaneous acceleration, that is, the force depicted is the instantaneous force at the time of typing;

[0021] S23: In order to avoid abnormal reactions caused by extreme values, the sigmoid function is used to map F to between 0 and 1. The specific formula is as follows:

[0022] F N =1 / (1+e (-F) ).

[0023] As a preferred solution of the virtual reality-based intelligent text input system of the present invention, the specific steps of the automatic compensation mechanism module are as follows:

[0024] S31: force threshold detection;

[0025] S32: Missing key search;

[0026] S33: Compensation input execution.

[0027] As a preferred solution of the virtual reality-based intelligent text input system of the present invention, the specific steps of the force threshold detection are: when the user performs a typing operation, the system monitors the force F of the finger hitting the virtual keyboard in real time. N , if the force F N Greater than the preset force threshold F t , the user's input is considered valid and the next step is to proceed.

[0028] As a preferred solution of the virtual reality-based intelligent text input system of the present invention, the specific steps of the missed key search are: after confirming that the input is valid, the system will check whether the finger collides with the keyboard normally. If the device fails to detect the collision, there is no input character in the input box, that is, missed keys occur. At this time, the intelligent text input system will start the automatic missed key compensation mechanism to calculate the Euclidean distance D between the finger and all keys. ij ;

[0029]

[0030] Where i represents the finger; j represents the jth key; (x i ,y i ,z i ) is the finger coordinate, (x j ,y j ,z j ) is the jth key coordinate; set a variable D min Used to store the minimum distance, the initial value is D i1 ; if D ij <D min, then update D min =D ij ; Select the key with the smallest distance as the key the user is most likely to want to enter.

[0031] As a preferred solution of the virtual reality-based intelligent text input system described in the present invention, the specific steps of executing the compensation input are: after determining the most likely key press, the system automatically enters the character corresponding to the key press into the text box to complete the missed key compensation; the compensation mechanism will not work when the user is inputting normally, and will only be activated when a missed key occurs to avoid unnecessary interference.

[0032] As a preferred solution of the virtual reality-based intelligent text input system of the present invention, the specific steps of the force-driven visual feedback module are as follows:

[0033] S41: dynamic visual feedback;

[0034] S42: Custom adjustment factor.

[0035] As a preferred solution of the intelligent text input system based on virtual reality described in the present invention, the specific steps of the dynamic visual feedback are: setting two correlation coefficients, namely, the correlation coefficient K1 between the execution speed and strength of the broken recovery animation, and the correlation coefficient K2 between the brightness and the strength; adjusting the two set coefficients through multiple rounds of tests to achieve the best effect, and finally determining that the K1 value is 2 and the K2 value is 3; at this time, according to F N , calculate the brightness L around the virtual appendage and the speed S of the broken recovery animation execution;

[0036] S=1+2F N

[0037] L=3F N

[0038] Because the shattering recovery animation execution speed is 1 before dynamic visual feedback is adopted, a basic shattering recovery animation execution time 1 needs to be added during dynamic adjustment; the brightness L affects the visual effect of the virtual appendage, making it brighter when knocked, enhancing the visual feedback; the shattering recovery animation execution speed S determines the speed of shattering and recovery of the virtual appendage. The greater the force, the faster the shattering and recovery speed, the shorter the shattering recovery time, and the faster the feedback.

[0039] As a preferred solution of the intelligent text input system based on virtual reality described in the present invention, the specific steps of customizing the adjustment coefficient are as follows: based on the different visual feedback required by each user, the user can dynamically adjust the correlation coefficients K1 and K2 values ​​according to their own needs when using the intelligent text input system; wherein:

[0040]

[0041] Compared with existing technologies:

[0042] Compared with traditional pseudo-tactile feedback, the present invention can greatly enhance the user's experience of text input in virtual reality devices; in a VR environment, if the visual feedback can change according to the tapping force, this will simulate the physical behavior of the real world, making the user feel like using a real keyboard; dynamic visual feedback provides users with an intuitive signal indicating that their operation has been recognized and responded to by the system; this instant feedback enables users to better control their input, which not only enhances user participation, but also makes the operation process smoother and reduces the possibility of misoperation; force-driven visual feedback not only reduces the error rate and penetration rate during text input, but also provides more realistic tactile feedback compared to BrittleStylus technology; this adaptive feedback mechanism can adjust visual assistance according to the user's real-time performance, optimize the user's immersion and interactivity; at the same time, the automatic compensation mechanism also reduces the number of missed keys during text input on the basis of force-driven visual feedback, further improving the user's experience of text input using bare hands in a VR environment.

[0043] In addition, the present invention also has the following effects:

[0044] More accurate text input: Compared with freehand typing and using the BrittleStylus technology, the text input error rate of the present invention has been significantly reduced;

[0045] Scalability: The present invention uses the development framework of Unity+XR interaction toolkit. Although Pico 4Ultra is used for development, Unity's own XR Plug-in Management can support a variety of XR devices and platforms, so the present invention can be widely used in different platforms and devices;

[0046] More realistic user experience: Compared with bare-hand typing and using the BrittleStylus technology, the finger penetration rate of the present invention is also significantly reduced, reflecting a more realistic interaction state; the evaluation results of the NASA Task Load Index questionnaire show that the intelligent text input system has a more comfortable user experience and the user's fatigue burden is significantly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0049] The present invention provides an intelligent text input system based on virtual reality, please refer to Figure 1 ,include:

[0050] The user input module is used to obtain the user's keystroke details when the user uses a virtual finger to click the target key; the force characterization module is used to calculate the force according to Newton's second law when the user presses the key; the force drive and compensation mechanism module is used to automatically compensate and provide visual feedback of the force drive according to the force calculated by the force characterization module;

[0051] The force driving and compensation mechanism module includes: an automatic compensation mechanism module, which is used to perform force threshold detection, missed key search and compensation input execution in sequence according to the force calculated by the force characterization module; a force driving visual feedback module, which is used to perform dynamic visual feedback and customized adjustment coefficient in sequence according to the force calculated by the force characterization module.

[0052] The specific steps of the force characterization module are as follows:

[0053] S21: Calculate the typing force F=ma according to Newton's second law, where m is the mass of the finger and a is the acceleration; the calculation formula of a is as follows:

[0054] a=(v2 2 -v1 2 ) / (2lenth)

[0055] Among them, the distance lenth is the distance between the keyboard and a vertical board in front of the keyboard with the same size as the keyboard, v1 is the speed of passing through the vertical board, and v2 is the speed of passing through the keyboard. The calculation formulas of v1 and v2 are as follows:

[0056] v1=d1 / t1

[0057] v2=d2 / t2

[0058] Wherein, t1 and t2 are the time to pass through the vertical plate and the time to pass through the keyboard, respectively; d1 and d2 are the thickness of the vertical plate and the thickness of the keyboard, respectively;

[0059] S22: approximate the acceleration a to the instantaneous acceleration, that is, the force depicted is the instantaneous force at the time of typing;

[0060] S23: In order to avoid abnormal reactions caused by extreme values, the sigmoid function is used to map F to between 0 and 1. The specific formula is as follows:

[0061] F N =1 / (1+e(-F) ).

[0062] The specific steps of the automatic compensation mechanism module are as follows:

[0063] S31: Force threshold detection: When the user is typing, the system monitors the force F of the finger hitting the virtual keyboard in real time. N , if the force F N Greater than the preset force threshold F t , the user's input is considered valid and the next step is to proceed;

[0064] S32: Missed key search: After confirming that the input is valid, the system will check whether the finger collides with the keyboard normally. If the device fails to detect the collision and no characters are entered in the input box, it means that a missed key has occurred. At this time, the intelligent text input system will start the automatic missed key compensation mechanism to calculate the Euclidean distance D between the finger and all keys. ij ;

[0065]

[0066] Where i represents the finger; j represents the jth key; (x i ,y i ,z i ) is the finger coordinate, (x j ,y j ,z j ) is the jth key coordinate; set a variable D min Used to store the minimum distance, the initial value is D i1 ; if D ij <D min , then update D min =D ij ; Select the key with the smallest distance as the key the user is most likely to want to input;

[0067] S33: Compensation input execution: After determining the most likely key, the system automatically enters the character corresponding to the key into the text box to complete missed key compensation; the compensation mechanism will not work when the user inputs normally.

[0068] It will only start when a missed key occurs to avoid unnecessary interference.

[0069] The specific steps of the force-driven visual feedback module are as follows:

[0070] S41: Dynamic visual feedback: Set two correlation coefficients, namely, the correlation coefficient K1 between the speed and strength of the broken recovery animation, and the correlation coefficient K2 between the brightness and strength; adjust the two coefficients through multiple rounds of testing to achieve the best effect, and finally determine that the K1 value is 2 and the K2 value is 3; at this time, the FN , calculate the brightness L around the virtual appendage and the speed S of the broken recovery animation execution;

[0071] S=1+2F N

[0072] L=3F N

[0073] Before dynamic visual feedback is adopted, the execution speed of the shattering recovery animation is 1, so a basic shattering recovery animation execution time of 1 needs to be added during dynamic adjustment; the brightness L affects the visual effect of the virtual appendage, making it brighter when struck and enhancing the visual feedback; the shattering recovery animation execution speed S determines the speed of the virtual appendage breaking and recovering. The greater the force, the faster the breaking and recovery speed, the shorter the shattering recovery time, and the faster the feedback.

[0074] S42: Customized adjustment coefficient: Based on the different visual feedback required by each user, the user can dynamically adjust the correlation coefficients K1 and K2 according to their own needs when using the intelligent text input system; where:

[0075]

[0076] In specific use, the system implementation steps are as follows:

[0077] When a user uses the smart text input system in a virtual reality device, each keystroke goes through the following steps:

[0078] Step 1: The user uses a virtual finger to click the target key, and the system obtains the user's keystroke details;

[0079] Step 2: After the system obtains the keystroke details of the user, the keystroke details are described in the force characterization module of the system. According to Newton's second law, the typing force F=ma is calculated, where m is the finger mass (set as a constant 1 in the system) and a is the acceleration. The calculation formula of a is as follows:

[0080] a=(v2 2 -v1 2 ) / (2lenth)

[0081] In the system, the distance lenth is the distance between the keyboard and a vertical board in front of the keyboard of the same size as the keyboard. The value of lenth is set very small; v1 is the speed of passing through the vertical board, and v2 is the speed of passing through the keyboard. The calculation formulas of v1 and v2 are as follows:

[0082] v1=d1 / t1

[0083] v2=d2 / t2

[0084] Among them, t1 and t2 are the time to pass through the vertical board and the time to pass through the keyboard respectively, d1 and d2 are the thickness of the vertical board and the thickness of the keyboard respectively. Because the thickness of the vertical board and the keyboard is very small, v1 and v2 can be approximated as the instantaneous speed;

[0085] Through the above calculations, the acceleration a is approximated as the instantaneous acceleration, that is, the force depicted is the instantaneous force at the time of typing;

[0086] Step 3: After calculating the keystroke force through the mechanical model, in order to avoid abnormal reactions caused by extreme values, use the sigmoid function to map F to between 0 and 1;

[0087] F N =1 / (1+e (-F) )

[0088] Step 4: The system monitors the force F of the finger tapping the virtual keyboard in real time N If the force F N Greater than the preset force threshold F t (After multiple rounds of testing, it was finally determined that F t The value is 0.6), then the user's input is considered valid and the next step is to proceed;

[0089] Step 5: When the intelligent text input system determines that the user input is valid, it checks whether the device detects the collision between the finger and the keyboard normally; if the device fails to detect the collision, no characters are input in the input box, that is, missed keys occur. At this time, the intelligent text input system will start the automatic missed key compensation mechanism and calculate the Euclidean distance D between the finger and all keys. ij ;

[0090]

[0091] Where i represents the finger, j represents the jth key; (x i ,y i ,z i ) is the finger coordinate, (x j ,y j ,z j ) is the jth key coordinate; set a variable D min Used to store the minimum distance, the initial value is D i1 ; if D ij <D min , then update D min =D ij ; Select the key with the smallest distance as the key the user is most likely to want to input;

[0092] Step 6: After the system predicts the key that the user is most likely to enter, it will perform a compensation operation; the system automatically enters the character corresponding to the key into the text box to complete the missed key compensation; the compensation mechanism will not work when the user is inputting normally, and will only be activated when a missed key occurs to avoid unnecessary interference;

[0093] Step 7: While the automatic compensation mechanism is working, the force-driven visual feedback will also change dynamically; according to F N , calculate the brightness L around the virtual appendage and the speed S of the broken recovery animation execution;

[0094] S=1+2F N

[0095] L=3F N

[0096] Two correlation coefficients are set: the correlation coefficient K1 between the speed of the shattering recovery animation and the intensity, and the correlation coefficient K2 between the brightness and the intensity. These two coefficients are adjusted through multiple rounds of testing to achieve the best effect, and finally the K1 value is determined to be 2 and the K2 value is determined to be 3. Because the speed of the shattering recovery animation is 1 before dynamic visual feedback is adopted, a basic shattering recovery animation execution time 1 needs to be added during dynamic adjustment. The brightness L affects the visual effect of the virtual appendage, making it brighter when struck, and enhancing the visual feedback. The shattering recovery animation execution speed S determines the speed of the virtual appendage to be broken and recovered. The greater the intensity, the greater the speed of breaking and recovery, the shorter the shattering recovery time, and the faster the feedback.

[0097] Step 8: The system dynamically adjusts the corresponding visual feedback according to the user's keystroke force. At this point, the user completes a complete keystroke process and waits for the next keystroke;

[0098] Step 9: After the user completes multiple keystrokes, the correlation coefficient can be dynamically adjusted according to the different visual feedback required by the user. When using the intelligent text input system, the user can dynamically adjust the correlation coefficients K1 and K2 values ​​according to their own needs; wherein:

[0099]

[0100] Based on the above, setting a range for the correlation coefficient is mainly to avoid system instability or deterioration of user experience; for example, when the K1 value is too large, the broken recovery animation may be incomplete;

[0101] Step 10: At this point, the complete implementation process of the intelligent text input system is completed.

[0102] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced by equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed in the present invention may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. An intelligent text input system based on virtual reality, characterized in that: include: A user input module, used to obtain the keystroke details of the user when the user clicks the target key with a virtual finger; The force characterization module is used to calculate the force according to Newton's second law when the user presses the key; A force driving and compensation mechanism module, used for automatic compensation and force-driven visual feedback according to the force calculated by the force characterization module; The force driving and compensation mechanism module includes: An automatic compensation mechanism module, used for performing force threshold detection, missing key search and compensation input execution in sequence according to the force calculated by the force characterization module; The force-driven visual feedback module is used to perform dynamic visual feedback and custom adjustment coefficients in turn according to the force calculated by the force characterization module.

2. The intelligent text input system based on virtual reality according to claim 1, characterized in that: The specific steps of the force characterization module are as follows: S21: Calculate the typing force F=ma according to Newton's second law, where m is the mass of the finger and a is the acceleration; the calculation formula of a is as follows: a=(v2 2 -v1 2 ) / (2lenth) Among them, the distance lenth is the distance between the keyboard and a vertical board in front of the keyboard with the same size as the keyboard, v1 is the speed of passing through the vertical board, and v2 is the speed of passing through the keyboard. The calculation formulas of v1 and v2 are as follows: v1=d1 / t1 v2=d2 / t2 Wherein, t1 and t2 are the time to pass through the vertical plate and the time to pass through the keyboard, respectively; d1 and d2 are the thickness of the vertical plate and the thickness of the keyboard, respectively; S22: approximate the acceleration a to the instantaneous acceleration, that is, the force depicted is the instantaneous force at the time of typing; S23: In order to avoid abnormal reactions caused by extreme values, the sigmoid function is used to map F to between 0 and 1. The specific formula is as follows: F N =1 / (1+e (-F) )。 3. The intelligent text input system based on virtual reality according to claim 1, characterized in that: The specific steps of the automatic compensation mechanism module are as follows: S31: force threshold detection; S32: Missing key search; S33: Compensation input execution.

4. The intelligent text input system based on virtual reality according to claim 3 is characterized in that: The specific steps of the force threshold detection are as follows: when the user is typing, the system monitors the force F of the finger hitting the virtual keyboard in real time. N , if the force F N Greater than the preset force threshold F t , the user's input is considered valid and the next step is to proceed.

5. The intelligent text input system based on virtual reality according to claim 4 is characterized in that: The specific steps of the missed key search are as follows: after confirming that the input is valid, the system will check whether the finger collides with the keyboard normally. If the device fails to detect the collision, no characters are input in the input box, that is, a missed key occurs. At this time, the intelligent text input system will start the automatic missed key compensation mechanism to calculate the Euclidean distance D between the finger and all the keys. ij ; Where i represents the finger; j represents the jth key; (x i ,y i ,z i ) is the finger coordinate, (x j ,y j ,z j ) is the jth key coordinate; set a variable D min Used to store the minimum distance, the initial value is D i1 ; if D ij <D min , then update D min =D ij ; Select the key with the smallest distance as the key the user is most likely to want to enter.

6. The intelligent text input system based on virtual reality according to claim 5, characterized in that: The specific steps of executing the compensation input are: after determining the most likely key, the system automatically enters the character corresponding to the key into the text box to complete the missed key compensation; the compensation mechanism will not work when the user is inputting normally, and will only be activated when a missed key occurs to avoid unnecessary interference.

7. The intelligent text input system based on virtual reality according to claim 1, characterized in that: The specific steps of the force-driven visual feedback module are as follows: S41: dynamic visual feedback; S42: Custom adjustment factor.

8. The intelligent text input system based on virtual reality according to claim 7, characterized in that: The specific steps of the dynamic visual feedback are: setting two correlation coefficients, namely, the correlation coefficient K1 between the execution speed and strength of the broken recovery animation, and the correlation coefficient K2 between the brightness and strength; adjusting the two set coefficients through multiple rounds of testing to achieve the best effect, and finally determining that the K1 value is 2 and the K2 value is 3; at this time, the F N , calculate the brightness L around the virtual appendage and the speed S of the broken recovery animation execution; S=1+2F N L=3F N Because the shattering recovery animation execution speed is 1 before dynamic visual feedback is adopted, a basic shattering recovery animation execution time 1 needs to be added during dynamic adjustment; the brightness L affects the visual effect of the virtual appendage, making it brighter when knocked, enhancing the visual feedback; the shattering recovery animation execution speed S determines the speed of shattering and recovery of the virtual appendage. The greater the force, the faster the shattering and recovery speed, the shorter the shattering recovery time, and the faster the feedback.

9. The intelligent text input system based on virtual reality according to claim 8, characterized in that: The specific steps of customizing the adjustment coefficients are as follows: based on the different visual feedback required by each user, the user can dynamically adjust the correlation coefficients K1 and K2 values ​​according to their own needs when using the intelligent text input system; wherein:

Images