Multi-input system for recognizing gestures, writing symbols and virtual keyboard on touch screen

Abstract

The invention discloses a multi-input system for recognizing gestures, writing symbols and a virtual keyboard on a touch screen. The multi-input system comprises the touch screen; the touch IC receives a touch signal from the touch screen and converts the touch signal into a touch data frame; a processor group receives the touch data frame to perform touch data processing, wherein the touch data processing applies an algorithm which can be directly executed by an OS and an algorithm processed by an AI; when the touch data frame is larger than a specific number of inputs, transmitting the touch data frame to an AI identification module connected with the processor group for AI identification, and transmitting an identification result to the OS for processing; wherein when the identification result corresponds to the key input or the written symbol, language grammar correction or symbol correction is performed; the touch screen is used for displaying a virtual keyboard and comprises a plurality of function keys, a number key, a symbol key and a plurality of character keys.

Description

Technical Field

[0001] The present invention relates to input recognition of a touch screen, and more particularly to a multi-input system for recognizing gestures, written symbols and a virtual keyboard on a touch screen. Background Art

[0002] At present, many electronic products use touch screens as input devices, which are widely used in various industries, such as computers, PADs, mobile phones, automobiles, medical equipment, industrial control equipment, etc., and input is performed on the touch screen through gestures, clicking virtual keys or stylus. However, the current virtual keyboard of the touch screen lacks the feel of typing on the traditional physical keyboard, and the position is fixed, which is quite inconvenient. Although there are virtual keyboards with special key arrangements, it is necessary to keep visual contact with the virtual keyboard to click its key position during input, so the input speed is still relatively slow and prone to misoperation.

[0003] Generally, electronic information devices have three main input methods. One is gesture operation, which is mainly used to operate the display of various function panels or call related applications (APP); one is key input, in which a virtual keyboard is displayed on the display screen, and the user inputs characters through the virtual keyboard; the other is complex graphic input, which may be handwritten characters or mathematical equations.

[0004] Among them, the operation gestures are supported by the operating systems of electronic information devices and are easier to recognize.

[0005] As for key input, since most touch screens use a simplified virtual keyboard, the key arrangement is very different from that of a traditional physical keyboard. Due to the control method of a traditional touch screen and its operating system, these operations must be input in sequence through multiple consecutive actions, and the input speed is slow and cannot be compared with the input speed of a traditional physical keyboard. Moreover, the virtual keyboard of the touch screen cannot use multiple fingers to type quickly like a traditional physical keyboard.

[0006] Handwritten characters such as general English letters such as A, B, C, or Chinese characters can be recognized, but the virtual keyboard input mode must be switched before handwriting recognition can be performed. However, handwritten mathematical equations cannot be recognized through traditional touch screens.

[0007] Therefore, the present invention hopes to propose a new multi-input system for recognizing gestures, written symbols and virtual keyboards on a touch screen. Multiple continuous inputs on the touch screen can be recognized through AI recognition, allowing users to operate the touch screen as smoothly and quickly as using physical buttons. It can also recognize handwritten characters and mathematical equations to solve the above-mentioned defects in the prior art. Summary of the invention

[0008] Therefore, the purpose of the present invention is to solve the above-mentioned problems in the conventional technology. The present invention proposes a multi-input system for identifying gestures, written symbols and virtual keyboards on a touch screen. For the stylus input, gesture action or virtual keyboard input on the touch screen, it is pre-determined whether it can be supported by the operating system of the device installed on the touch screen. If it is, the original operating system is applied to identify the gesture or key. If not, the AI ​​recognition module is applied to predict and identify complex and complicated gestures and stylus input. Therefore, when the user operates the touch screen, it is as smooth and fast as using physical keys. The present invention uses an AI recognition module to simultaneously identify multi-key input, operation gestures and written symbols (including characters and mathematical equations), and also performs grammar correction and symbol correction for key input and written symbols, improves the accuracy of gestures, key input and stylus input on the touch screen, and the reaction speed of touch input, which can improve the typing speed of the user on the touch screen; and it is not easy to make mistakes when typing and operating on the touch screen, thereby improving the user's operating experience.

[0009] In order to achieve the above-mentioned purpose, the present invention proposes a multi-input system for identifying gestures, written symbols and virtual keyboards on a touch screen, comprising a touch screen for receiving a user's touch action and generating a corresponding touch signal; wherein the touch screen is a touch panel of an electronic device; a touch IC connected to the touch screen for receiving a touch signal from the touch screen, the touch IC converting the touch signal into a plurality of touch data, the plurality of touch data forming a two-dimensional touch data frame; a processor group connected to the touch IC and an OS (Operating System); System, operating system), receives the touch data frame from the touch IC to perform a touch data processing, and the touch data processing applies the algorithm that can be directly executed by the OS and the algorithm processed by AI; wherein in the touch data processing, when the touch data frame corresponds to a plurality of inputs greater than a specific number, indicating that the touch data frame is not data that can be directly processed by the OS, the touch data frame is transmitted to an AI recognition module connected to the processor group for AI recognition, and an AI recognition result output by the AI ​​recognition module is transmitted back to the processor group; when the AI ​​recognition result corresponds to a key input, the processor group performs language grammar correction on the key input, and transmits the result of the language grammar correction to the OS for processing; when the AI ​​recognition result corresponds to a written symbol including a string or an equation, the processor group transmits the written symbol to a symbol corrector for correction and transmits it to the OS for processing; when the AI ​​recognition result corresponds to an operation gesture, the processor group transmits the operation gesture to the OS for processing; wherein in the touch data processing, the touch IC Used to perform an initial processing, the initial processing is that the touch IC converts the touch signal of the touch screen into the touch data frame; a pre-processing module is connected to the touch IC, wherein in the touch data processing, the pre-processing module is used to receive the touch data frame from the touch IC and perform noise filtering, normalization and segmentation on the touch data frame; wherein the touch screen is used to display a virtual keyboard or a part of the virtual keyboard, so that the user can visually view the virtual keyboard and input on the virtual keyboard.

[0010] Furthermore, the virtual keyboard includes multiple function keys, a number key and a symbol key, which are respectively placed on the left and right sides of the virtual keyboard, and multiple character keys are placed in the center of the virtual keyboard; according to the position, the virtual keyboard is divided into an upper row of key groups and a lower row of key groups.

[0011] Furthermore, it also includes an input identifier connected to the preprocessing module; wherein in touch data processing, when the touch screen displays the virtual keyboard, the input identifier determines whether the touch data frame corresponds to a key input or a non-key input, and the key input is a character, number or symbol; when the touch data frame corresponds to the key input, multiple continuous key inputs are transmitted to a corpus for corpus matching.

[0012] Furthermore, it also includes a press classifier connected to the input identifier, which is used to determine whether the key input matched by the corpus of the touch data frame is supported by the OS. If so, it is transmitted to the OS for processing, and the touch IC is allowed to perform the initial processing again to process the next touch data frame.

[0013] Furthermore, an AI classifier is connected to the press classifier, wherein in the touch data processing, when the press classifier determines that the touch data frame is not supported by the OS, or the input identifier determines that the touch data frame is a non-key input, or the input identifier determines that the virtual keyboard is not displayed when receiving the touch data frame or cannot determine whether the virtual keyboard is displayed, the press classifier inputs the touch data frame into the AI ​​classifier for touch classification; In the touch classification, the AI ​​classifier sends the touch data frame from the press classifier or the input identifier to an AI recognition module located in the AI ​​classifier for recognition, and the AI ​​recognition module outputs the AI ​​recognition result after recognition, and the AI ​​recognition result is stored in the touch data frame to determine whether the touch data frame corresponds to a specific key input, operation gesture or written symbol; In the training phase of the AI ​​recognition module for writing symbol recognition, the input of the AI ​​recognition module is an equation or a character written by the user on the touch screen using a finger or a stylus, and the output of the AI ​​recognition module corresponds to a specific program code; In the AI ​​classifier, when the AI ​​recognition result corresponds to a specific operation gesture, the touch data frame is sent to the OS and processed by the OS, and the touch IC is made to perform the initial processing again to process the next touch data frame.

[0014] Furthermore, a symbol corrector is connected to the AI ​​classifier, in which, when the AI ​​recognition result corresponds to the written symbol, the touch data frame including the AI ​​recognition result is transmitted to the symbol corrector for symbol correction; wherein the AI ​​recognition result is regarded as a series of written symbols; the symbol corrector applies a pre-trained symbol model library to perform AI writing symbol prediction and error correction on the AI ​​recognition result; wherein the written symbol is a character or a mathematical equation; The symbol corrector transmits the corrected message to the OS for processing, and enables the touch IC to perform the initial processing again to process the next touch data frame.

[0015] Furthermore, when the corrected written symbol is a character or a string, the symbol corrector applies a character editing unit to edit the character or string; when the corrected written symbol is a mathematical equation and it is a string of specific program codes, the symbol corrector applies a mathematical code editing language to compile the program code corresponding to the mathematical equation; when the written symbol corresponding to the AI ​​recognition result is not supported by any known application, the program code corresponding to the written symbol is directly output.

[0016] Furthermore, a grammar corrector is connected to the AI ​​classifier, in which, when the AI ​​recognition result corresponds to the key input, the touch data frame including the AI ​​recognition result is transmitted to the grammar corrector for grammar correction; wherein the AI ​​recognition result is regarded as a series of input results of the virtual keyboard; the grammar corrector uses a pre-trained language grammar model library to perform AI language grammar prediction and error correction on the AI ​​recognition result; The syntax corrector transmits the corrected message to the OS for processing, and enables the touch IC to perform the initial processing again to process the next touch data frame.

[0017] Furthermore, the AI ​​recognition result output by the AI ​​recognition module after recognition is stored in the touch data frame to determine whether the touch data frame corresponds to a specific key input, operation gesture or written symbol; During the training phase of the AI ​​recognition module for writing symbol recognition, the input of the AI ​​recognition module is an equation or a character written by the user on the touch screen using a finger or a stylus, and the output of the AI ​​recognition module corresponds to a specific program code.

[0018] Furthermore, the virtual keyboard also includes a category key for controlling the touch screen to display a symbol input area; the category key is also used to control the touch screen to display a key sequence same as the physical keyboard, or not to display the virtual keyboard or the symbol input area, so as to receive operation gestures input by the user.

[0019] Furthermore, it also includes a keyboard display controller connected to the pre-processing module, which is used to receive the touch data frame, and when the touch data frame corresponds to a keyboard display gesture, the touch screen is driven to display the virtual keyboard, and the touch data frame is subsequently processed by the pre-processing module; When the touch screen displays the virtual keyboard, the pre-processing module can simultaneously accept the user's key input on the virtual keyboard and the written symbols input on the touch screen. The present invention also proposes a fast multi-input keyboard system used on a touch screen, including a touch screen for receiving the user's touch action and generating a corresponding touch signal; wherein the touch screen is a touch panel of an electronic device; wherein the touch screen displays a virtual keyboard; a touch IC connected to the touch screen, for receiving the touch signal from the touch screen, the touch IC converts the touch signal into a plurality of touch data, the plurality of touch data forming a two-dimensional touch data frame; a processor group connected to the touch IC and an OS (Operating System), receiving the touch data frame from the touch IC to perform a touch data processing, the touch data processing using an algorithm that can be directly executed by the OS; wherein the virtual keyboard also includes a plurality of function keys, a number key and a symbol key respectively placed on the left and right sides of the virtual keyboard, and a plurality of character keys placed in the center of the virtual keyboard. According to the position, the virtual keyboard is divided into an upper key group and a lower key group.

[0020] Furthermore, the function keys include a Caps Lock key, an Enter key, a first shift key, a Control key, a Space key, an Alt key, and a second shift key; the upper row of keys is slightly symmetrically M-shaped to accommodate the posture of pressing keys with both hands, and the lower row of keys is slightly concave in shape, the Control key, the Space key, and the Alt key belong to the lower row of keys, and are positioned downward.

[0021] Furthermore, the upper key group has 12 keys, from left to right, namely the Caps Lock key, nine character keys, the symbol key and the enter key; each character key is used to input two or three different English letters; the lower key group includes 6 keys, from left to right, namely the first shift key, the Control key, the space key, the Alt key, the number key and the second shift key.

[0022] Furthermore, the nine character keys on the virtual keyboard are, from left to right, an ABC key corresponding to the letters A, B, C, a DEF key corresponding to the letters D, E, F, a GHI key corresponding to the letters G, H, I, a JKL key corresponding to the letters J, K, L, a MNO key corresponding to the letters M, N, O, a PQR key corresponding to the letters P, Q, R, a STU key corresponding to the letters S, T, U, a VWX key corresponding to the letters V, W, X, a YZ key corresponding to the letters Y, Z; or the nine character keys are, from left to right on the virtual keyboard, a QAZ key corresponding to the letters Q, A, Z, a WSX key corresponding to the letters W, S, X, an EDC key corresponding to the letters E, D, C, a RFV key corresponding to the letters R, F, V, a TGB key corresponding to the letters T, G, B, a YHN key corresponding to the letters Y, H, N, a UJM key corresponding to the letters U, J, M, an IK key corresponding to the letters I, K, and an OLP key corresponding to the letters O, L, P.

[0023] Furthermore, the multiple character keys include seven upper character keys, a lower left character key and a lower right character key; each character key is used to input two or three different English letters; wherein the upper row of key groups includes 10 keys, from left to right are the Caps Lock key, the seven upper character keys, the symbol key and the enter key; the lower row of key groups includes 8 keys, from left to right are the first shift key, the Control key, the lower left character key, the space key, the lower right character key, the Alt key, the number key and the second shift key; wherein the lower left character key is an MNO key corresponding to the letters M, N, O, or a TGB key corresponding to the letters T, G, B; wherein the lower right character key is a PQR key corresponding to the letters P, Q, R, or a YHN key corresponding to the letters Y, H, N.

[0024] Further, the seven upper character keys in the upper row of keys are, from left to right, an ABC key corresponding to letters A, B, C, a DEF key corresponding to letters D, E, F, a GHI key corresponding to letters G, H, I, a JKL key corresponding to letters J, K, L, a STU key corresponding to letters S, T, U, a VWX key corresponding to letters V, W, X, and a YZ key corresponding to letters Y, Z; or the seven upper character keys in the upper row of keys are, from left to right, a QAZ key corresponding to letters Q, A, Z, a WSX key corresponding to letters W, S, X, an EDC key corresponding to letters E, D, C, an RFV key corresponding to letters R, F, V, a UJM key corresponding to letters U, J, M, an IK key corresponding to letters I, K, and an OLP key corresponding to letters O, L, P. The features and advantages of the present invention can be further understood from the following description, and please refer to the accompanying drawings when reading. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Shows the architecture diagram of the present invention; Figure 2 Showing the operation flow chart of the present invention; Figure 3 Shows a block diagram of the components of the present invention; Figure 4 An operational flow chart showing touch data classification by only using an AI classifier according to the present invention; Figure 5 A block diagram showing the components of the present invention for classifying touch data using only an AI classifier; Figure 6 A schematic diagram showing a touch data frame of the present invention; Figure 7 Displaying a first embodiment of the virtual keyboard of the present invention; Figure 8 Displaying a second embodiment of the virtual keyboard of the present invention; Fig. 9 A third embodiment of the virtual keyboard of the present invention is shown; Fig.10 A fourth embodiment of the virtual keyboard of the present invention is shown; Fig.11 A schematic diagram showing the operation of the keyboard display controller of the present invention is shown. DETAILED DESCRIPTION

[0026] The structural composition of the present invention, as well as the effects and advantages it can produce, are described in detail below with reference to the accompanying drawings, with a preferred embodiment of the present invention.

[0027] Please refer to Figures 1 to 11 As shown, the multi-input system of the present invention for recognizing gestures, writing symbols and a virtual keyboard on a touch screen is shown, and the user can directly write writing symbols with characters or mathematical equations on the touch screen. Figure 1 As shown, the multi-input system includes the following elements: A touch screen 27 is used to receive a user's touch action and generate a corresponding touch signal, which is an electrical signal generated by each pixel of the touch screen 27 due to the electrical change (such as a change in capacitance) caused by the touch action. The touch screen 27 refers to a touch panel of various electronic devices, such as a mobile phone, a tablet, and other electronic devices.

[0028] A touch IC 26 is connected to the touch screen 27 and is used to receive a touch signal from the touch screen 27. The touch IC 26 converts the touch signal into a plurality of touch data 110. The plurality of touch data 110 form a two-dimensional touch data frame 100. Figure 6As shown, the touch data frame 100 includes multiple touch areas 120, wherein each touch area 120 is a pressing area of ​​the touch screen 27 pressed by the touch action. During processing, a representative coordinate point 122 in the touch area 120 and the lengths of the major axis 124 and the minor axis 126 of the touch area 120 (for example, two mutually perpendicular axes in the smallest rectangle including the touch area 120) are recorded.

[0029] A processor group 25 is connected to the touch IC 26 and an OS (Operating System) 22 to receive the touch data frame 100 from the touch IC 26 to perform a touch data process. The touch data process applies algorithms that can be directly executed by the OS 22 and algorithms processed by AI.

[0030] In the touch data processing, when the touch data frame 100 corresponds to a plurality of inputs less than a specific number, it means that the touch data frame 100 is data that can be directly processed by the OS 22 , and the touch data frame 100 is transmitted to the OS 22 for processing. When the touch data frame 100 is greater than a specific number of inputs, indicating that the touch data frame 100 is not data that can be directly processed by the OS22, the touch data frame 100 is passed to an AI recognition module 200 connected to the processor group 25 for AI recognition, and an AI recognition result output by the AI ​​recognition module 200 is transmitted back to the processor group 25. When the AI ​​recognition result corresponds to a key input, the processor group 25 performs further language grammar correction on the key input, and transmits the result of the language grammar recognition to the OS22 for processing; when the AI ​​recognition result corresponds to a written symbol (including a string and an equation), the processor group 25 transmits the written symbol to a symbol corrector to correct the written symbol and then transmits it to the OS22 for processing; when the AI ​​recognition result corresponds to an operation gesture, the processor group 25 transmits the operation gesture to the OS22 for operation gesture processing.

[0031] When the OS 22 completes processing of the input data, the OS 22 outputs the relevant data to a corresponding application 21 for use. The OS 22 is connected to the processor group 25 and the application 21 .

[0032] Figure 2 and Figure 3 The flow and related elements of the touch data processing of the processor group 25 are shown, wherein the touch data processing is completed by applying the touch IC 26, a pre-processing module 500, an input identifier 570, a press classifier 510, an AI classifier 540, a symbol corrector 550 and a grammar corrector 560.

[0033] The touch IC 26 is used to perform an initial processing, and the initial processing is that the touch IC 26 converts the touch signal of the touch screen 27 into the touch data frame 100 (step 1) for subsequent processing.

[0034] The pre-processing module 500 is connected to the touch IC 26. The pre-processing module 500 receives the touch data frame 100 from the touch IC 26, and performs noise filtering on the touch data frame 100 to remove unnecessary noise, normalizes the value of each touch data 110 so that it is within a limited range, and performs segmentation and other operations for subsequent processing (step 2). The plurality of touch data 110 are recorded in a memory 20 of the processor group 25.

[0035] The touch screen 27 is used to display a virtual keyboard 150 so that the user can visually view the virtual keyboard 150 and input in the virtual keyboard 150. The touch screen 27 may also display only a portion of the virtual keyboard 150, for example, only displaying keys of a certain column of the virtual keyboard 150.

[0036] The virtual keyboard 150 includes a plurality of function keys 80, a number key 90 and a symbol key 60 respectively disposed on the left and right sides of the virtual keyboard 150, and a plurality of character keys 70 disposed in the center of the virtual keyboard 150. The virtual keyboard 150 is divided into an upper row key group 160 and a lower row key group 180 according to the position. The function keys 80 include: a Caps Lock key 82, an Enter key 84, a first Shift key 81, a Control key 83, a Space key 85, an Alt key 87, and a second Shift key 89. The upper key group 160 may be slightly symmetrically M-shaped to accommodate the gesture of pressing keys with both hands, and the lower key group 180 may be slightly concavely arc-shaped, and the Control key 83, the Space key 85, and the Alt key 87 belong to the lower key group 180, and their positions may be slightly downward.

[0037] like Figure 7 and Figure 8 As shown, in the first and second embodiments, the upper key group 160 has 12 keys, which are respectively the Caps Lock key 82, nine character keys 70, the symbol key 60 and the enter key 84 from left to right; each character key 70 is used to input two or three different English letters. The lower key group 180 includes 6 keys, which are respectively the first shift key 81, the Control key 83, the space key 85, the Alt key 87, the number key 90 and the second shift key 89 from left to right.

[0038] The nine character keys 70 are in the first embodiment (eg Figure 7 As shown in the figure, the virtual keyboard 150 is classified according to the order of the traditional English alphabet, which are as follows from left to right: an ABC key 71 corresponds to the letters "A", "B", and "C", a DEF key 72 corresponds to the letters "D", "E", and "F", a GHI key 73 corresponds to the letters "G", "H", and "I", a JKL key 74 corresponds to the letters "J", "K", and "L", a MNO key 75 corresponds to the letters "M", "N", and "O", a PQR key 76 corresponds to the letters "P", "Q", and "R", a STU key 77 corresponds to the letters "S", "T", and "U", a VWX key 78 corresponds to the letters "V", "W", and "X", and a YZ key 79 corresponds to the letters "Y" and "Z". In the second embodiment (as shown in the figure, the virtual keyboard 150 is classified according to the order of the traditional English alphabet, which are as follows from left to right: an ABC key 71 corresponds to the letters "A", "B", and "C", a DEF key 72 corresponds to the letters "D", "E", and "F", a GHI key 73 corresponds to the letters "G", "H", and "I", a JKL key 74 corresponds to the letters "J", "K", and "L", a MNO key 75 corresponds to the letters "M", "N", and "O", a PQR key 76 corresponds to the letters "P", "Q", and "R", a STU key 77 corresponds to the letters "S", "T", and "U", a VWX key 78 corresponds to the letters "V", "W", and "X", and a YZ key 79 corresponds to the letters "Y" and "Z". Figure 8 (as shown) the nine character keys 70 are classified according to the key sequence from top to bottom in the QWERTY keyboard layout, and are arranged from left to right in the virtual keyboard 150 as follows: a QAZ key 71' corresponds to the letters "Q", "A", and "Z", a WSX key 72' corresponds to the letters "W", "S", and "X", an EDC key 73' corresponds to the letters "E", "D", and "C", an RFV key 74' corresponds to the letters "R", "F", and "V", a TGB key 75' corresponds to the letters "T", "G", and "B", a YHN key 76' corresponds to the letters "Y", "H", and "N", a UJM key 77' corresponds to the letters "U", "J", and "M", an IK key 78' corresponds to the letters "I", and "K", and an OLP key 79' corresponds to the letters "O", "L", and "P".

[0039] like Fig. 9 and Fig.10As shown, in the third and fourth embodiments, the plurality of character keys 70 include seven upper character keys 701, a lower left character key 702 and a lower right character key 702'. Each character key 70 is used to input two or three different English letters. The upper key group 160 includes 10 keys, which are respectively the Caps Lock key 82, the seven upper character keys 701, the symbol key 60 and the enter key 84 from left to right; the lower key group 180 includes 8 keys, which are respectively the first shift key 81, the Control key 83, the lower left character key 702, the space key 85, the lower right character key 702', the Alt key 87, the numeric key 90 and the second shift key 89 from left to right. The lower left character key 702 is an MNO key 75 corresponding to the letters "M", "N", "O", or a TGB key 75' corresponding to the letters "T", "G", "B". The lower right character key 702' is a PQR key 76 corresponding to the letters "P", "Q", and "R", or a YHN key 76' corresponding to the letters "Y", "H", and "N".

[0040] In the third embodiment, the seven upper character keys 701 (such as Fig. 9 ) is classified according to the order of the traditional alphabet, and the upper key group 160 is sequentially as follows from left to right: an ABC key 71 corresponds to the letters "A", "B", and "C", a DEF key 72 corresponds to the letters "D", "E", and "F", a GHI key 73 corresponds to the letters "G", "H", and "I", a JKL key 74 corresponds to the letters "J", "K", and "L", a STU key 77 corresponds to the letters "S", "T", and "U", a VWX key 78 corresponds to the letters "V", "W", and "X", and a YZ key 79 corresponds to the letters "Y" and "Z". In the fourth embodiment (as shown in FIG. Fig.10 ), the seven upper character keys 701 are classified according to the key sequence from top to bottom of the QWERTY keyboard layout, and are arranged from left to right in the upper key group 160 as follows: a QAZ key 71' corresponding to the letters "Q", "A", and "Z", a WSX key 72' corresponding to the letters "W", "S", and "X", an EDC key 73' corresponding to the letters "E", "D", and "C", an RFV key 74' corresponding to the letters "R", "F", and "V", a UJM key 77' corresponding to the letters "U", "J", and "M", an IK key 78' corresponding to the letters "I", and "K", and an OLP key 79' corresponding to the letters "O", "L", and "P".

[0041] The present invention may include a keyboard display controller 530 connected to the pre-processing module 500, for receiving the touch data frame 100, and when the touch data frame 100 corresponds to a keyboard display gesture, the touch screen 27 is driven to display the virtual keyboard 150, and the touch data frame 100 is subsequently processed by the pre-processing module 500 ( Figure 3 or Figure 5 Step 2). Fig.11 As shown, for example, the touch data frame 100 corresponds to an input action of the user touching the touch screen 27 with both palms and fixed indexes, and the input action corresponds to the keyboard display gesture. The virtual keyboard 150 is displayed on the touch screen 27 at a position corresponding to the keyboard display gesture.

[0042] In the present invention, when the touch screen 27 displays the virtual keyboard 150, the pre-processing module 500 simultaneously accepts the key input of the user on the virtual keyboard 150 and the written symbols input on the touch screen 27, that is, the user can still input key input and written symbols at the same time. The following describes the method of identifying key input or written symbol input.

[0043] An input identifier 570 is connected to the preprocessing module 500. When the touch screen 27 displays the virtual keyboard 150, the input identifier 570 determines whether the touch data frame 100 corresponds to a key input or a non-key input (such as a written symbol input). The determination method can be determined according to the actual usage habits. For example, the touch data frame 100 may correspond to the user's input on the virtual keyboard 150, and the input only includes a click action or a click action greater than a fixed number, then the touch data frame 100 is considered to correspond to a key input (step 3). The key input may represent a character, a number or a symbol. Therefore, if the touch data frame 100 corresponds to the key input, a traditional method can be used to transmit multiple continuous key inputs to a corpus for corpus matching. For the accuracy of the operation, when the user needs to input a single character, number or symbol on the virtual keyboard 150, an auxiliary operation can be applied to achieve it. The auxiliary operation is such as: when the key is pressed, sliding in different directions, or long pressing to display a selection screen, allowing the user to select, etc. (step 4).

[0044] The press classifier 510 is connected to the input identifier 570 to determine whether the key input matched by the corpus of the touch data frame 100 is supported by the OS22 (that is, the OS22 can directly process it) (step 5). If so, it is transmitted to the OS22 for processing, and the touch IC26 is instructed to perform the initial processing again (step 1) to process the next touch data frame 100.

[0045] An AI classifier 540 is connected to the press classifier 510. In step 5, when the press classifier 510 determines that the touch data frame 100 is not supported by the OS 22, or in step 3 the input identifier 570 determines that the touch data frame 100 is a non-key input, or in step 3 the input identifier 570 determines that the virtual keyboard 150 is not displayed when receiving the touch data frame 100 or cannot determine whether the virtual keyboard 150 is displayed, the press classifier 510 inputs the touch data frame 100 into the AI ​​classifier 540 for touch classification (step 6).

[0046] In the touch classification, the AI ​​classifier 540 sends the touch data frame 100 from the press classifier 510 or the input identifier 570 to an AI recognition module 200 located in the AI ​​classifier 540 for recognition. The AI ​​recognition module 200 outputs the AI ​​recognition result after recognition. The AI ​​recognition result is stored in the touch data frame 100 to determine whether the touch data frame 100 corresponds to a specific key input, an operation gesture or a written symbol (the written symbol includes characters and mathematical equations) (step 7).

[0047] The AI ​​recognition module 200 is a recognition module that has been trained and tested using training and test sets. For example, different training and test touch data of the same person and different people are used as the input of the AI ​​recognition module 200, and the key input, operation gesture or writing symbol corresponding to the training and test touch data is used as the expected output of the AI ​​recognition module 200. The input and output are used to train and test the AI ​​recognition module 200, so that the AI ​​recognition module 200 has the function of recognizing the touch data 110. That is, the AI ​​recognition module 200 can recognize the corresponding key input, operation gesture or writing symbol for various forms of touch data 110. The AI ​​recognition module 200 is, for example, a neural network.

[0048] During the training phase of the AI ​​recognition module 200 for writing symbol recognition, the input of the AI ​​recognition module 200 is an equation or a character written by the user using a finger or a stylus on the touch screen 27, and the output of the AI ​​recognition module 200 corresponds to a specific program code.

[0049] In step 7, in the AI ​​classifier 540, when the AI ​​recognition result corresponds to a specific operation gesture, the touch data frame 100 is sent to the OS22 for processing by the OS22, and the touch IC26 is instructed to perform the initial processing (step 1) again to process the next touch data frame 100.

[0050] A symbol corrector 550 is connected to the AI ​​classifier 540. In step 7, in the AI ​​classifier 540, when the AI ​​recognition result corresponds to the written symbol, the touch data frame 100 including the AI ​​recognition result is transmitted to the symbol corrector 550 for symbol correction (step 9). The AI ​​recognition result is regarded as a series of written symbols. The symbol corrector 550 applies a pre-trained symbol model library 16 to perform AI writing symbol prediction and error correction functions on the AI ​​recognition result to improve input accuracy and input speed. The written symbol may be a character or a mathematical equation.

[0051] The symbol model library 16 has pre-established AI symbol models of writing symbol rules, which represent correct input and correction rules that comply with the symbol rules. The correction rules are used to correct user writing errors, grammatical errors of characters or strings, rule errors of mathematical equations, or errors generated by the recognition process of the AI ​​recognition module. The AI ​​recognition results are corrected according to these correction rules (i.e., a series of characters or mathematical equations).

[0052] The symbol corrector 550 transmits the corrected message to the OS 22 for processing, and enables the touch IC 26 to perform the initial processing (step 1) again to process the next touch data frame 100 .

[0053] When the corrected written symbol is a character or a string, the symbol corrector 550 uses a character editing unit 551 (which can be processed using Unicode or ASCII) to edit the character or string.

[0054] When the corrected written symbol is a mathematical equation (i.e., a string of specific program codes), the symbol corrector 550 applies a mathematical program code editing language to compile the program code corresponding to the mathematical equation. The mathematical program code editing language is, for example, LaTeX or MathML (Mathematical Markup Language). In fact, the present invention can use any mathematical program code editing language. Therefore, the program code corresponding to the mathematical equation can be converted into the corresponding mathematical equation after compilation by the mathematical program code editing language, and the mathematical equation can be displayed.

[0055] The mathematical program code editing language takes LaTeX as an example. For example, the program code of a mathematical equation is "\\frac { 1}{ 3}", which can be obtained after LaTeX editing to form a mathematical formula ⅓.

[0056] When the writing symbol corresponding to the AI ​​recognition result is not supported by any known application program, the program code corresponding to the writing symbol is directly output.

[0057] The grammar corrector 560 is connected to the AI ​​classifier 540. In step 7, in the AI ​​classifier 540, when the AI ​​recognition result corresponds to the key input, the touch data frame 100 including the AI ​​recognition result is transmitted to the grammar corrector 560 for grammar correction (step 8). The AI ​​recognition result is regarded as a series of input results of the virtual keyboard 150. The grammar corrector 560 uses a pre-trained language grammar model library 14 to perform AI language grammar prediction and error correction functions on the AI ​​recognition result to improve input accuracy and input speed.

[0058] The language grammar model library 14 is an AI grammar model with pre-established grammar rules, which represent grammatically correct inputs and correction rules in the virtual keyboard 150. The correction rules are used to correct errors caused by the user's finger touching the wrong key, or errors generated by the recognition process of the AI ​​recognition module. The AI ​​recognition result is corrected according to these correction rules (i.e., a series of inputs of the virtual keyboard 150).

[0059] The grammar corrector 560 transmits the corrected message (mainly human inter device (HID) message, i.e., the output of the virtual keyboard 150) to the OS22 for processing, and enables the touch IC26 to perform the initial processing (step 1) again to process the next touch data frame 100.

[0060] The virtual keyboard 150 further includes a classification key 152, which is used to control the touch screen 27 to display a symbol input area 153 to receive symbols (the symbols are characters or mathematical equations, i.e., written symbols) input by the user using a finger or a stylus. The symbols input in the symbol input area 153 will correspond to written symbols in the touch data frame 100 and will be recognized by the AI ​​classifier 540. The classification key 152 is also used to control the touch screen 27 to display a key sequence similar to that of a physical keyboard, or not to display the virtual keyboard 150 or the symbol input area 153, so as to receive operation gestures input by the user.

[0061] like Figure 4 and Figure 5 As shown, the present invention may also not use the press classifier 510 and the input identifier 570, and the touch data frame 100 is directly transmitted to the AI ​​classifier 540 for processing (that is, the output of step 2 will be output to step 6), that is, the classification of the touch data frame 100 is entirely performed by the AI ​​classifier 540.

[0062] The present invention implements correct touch control by classifying and identifying the touch data frame 100 using an AI algorithm, and uses an AI algorithm to analyze the touch data 110 captured from the touch data frame 100 of the dynamic input of the touch device, and identifies the user's touch action as a click of a finger or palm movement, or other touch actions that are not required for operation (for example, just placing the hand on the device without inputting). The present invention can adapt to various user usage conditions by training and testing the AI ​​learning of the AI ​​recognition module 200.

[0063] The advantage of the present invention is that it can pre-determine whether the stylus input, gesture action or virtual keyboard input on the touch screen can be supported by the operating system of the device on which the touch screen is installed. If so, the original operating system is used to identify gestures or keys. If not, the AI ​​recognition module is used to predict and identify complex and complicated gestures and stylus input. Therefore, when the user operates the touch screen, it can be as smooth and fast as using physical keys. The present invention uses an AI recognition module to simultaneously identify multi-key input, operation gestures and written symbols (including characters and mathematical equations), and also performs grammar correction and symbol correction on key input and written symbols, thereby improving the accuracy of gestures, key input and stylus input on the touch screen, as well as the response speed of touch input, which can increase the typing speed of the user on the touch screen; and it is not easy to make mistakes when typing and operating on the touch screen, thereby improving the user's operating experience.

[0064] The above detailed description is a specific description of a feasible embodiment of the present invention, but the embodiment is not intended to limit the patent scope of the present invention. Any equivalent implementation or modification that does not deviate from the technical spirit of the present invention should be included in the patent scope of the present invention.

Claims

1. A multi-input system for identifying gestures, written symbols and a virtual keyboard on a touch screen, characterized in that: include: A touch screen is used to receive a user's touch action and generate a corresponding touch signal; The touch screen is a touch panel of an electronic device; A touch IC is connected to the touch screen and is used to receive a touch signal from the touch screen. The touch IC converts the touch signal into a plurality of touch data, and the plurality of touch data form a two-dimensional touch data frame. A processor group is connected to the touch IC and an OS, receives the touch data frame from the touch IC to perform a touch data process, and the touch data process applies an algorithm that can be directly executed by the OS and an algorithm processed by AI; Wherein, in the touch data processing, when the touch data frame is greater than a specific number of inputs, indicating that the touch data frame is not data that can be directly processed by the OS, the touch data frame is transmitted to an AI recognition module connected to the processor group for AI recognition, and an AI recognition result output by the AI ​​recognition module is transmitted back to the processor group; when the AI ​​recognition result corresponds to a key input, the processor group performs language grammar correction on the key input, and transmits the result of the language grammar correction to the OS for processing; when the AI ​​recognition result corresponds to a written symbol including a string or an equation, the processor group transmits the written symbol to a symbol corrector for correction and transmits it to the OS for processing; when the AI ​​recognition result corresponds to an operation gesture, the processor group transmits the operation gesture to the OS for processing; In the touch data processing, the touch control IC is used to perform an initial processing, and the initial processing is that the touch control IC converts the touch signal of the touch screen into the touch data frame; A pre-processing module is connected to the touch control IC, wherein in the touch data processing, the pre-processing module is used to receive the touch data frame from the touch control IC, and perform noise filtering, normalization and segmentation on the touch data frame; The touch screen is used to display a virtual keyboard or a portion of a virtual keyboard, so that the user can visually view the virtual keyboard and input on the virtual keyboard.

2. The multi-input system for identifying gestures, written symbols and a virtual keyboard on a touch screen as claimed in claim 1, characterized in that: The virtual keyboard includes multiple function keys, a number key and a symbol key, which are respectively placed on the left and right sides of the virtual keyboard, and multiple character keys are placed in the center of the virtual keyboard; according to the position, the virtual keyboard is divided into an upper row key group and a lower row key group.

3. The multi-input system for identifying gestures, written symbols and a virtual keyboard on a touch screen as claimed in claim 1 or 2, characterized in that: Also includes an input identifier connected to the pre-processing module; In the touch data processing, when the touch screen displays the virtual keyboard, the input identifier determines whether the touch data frame corresponds to a key input or a non-key input, and the key input is a character, number or symbol; when the touch data frame corresponds to the key input, multiple continuous key inputs are transmitted to a corpus for corpus matching.

4. The multi-input system for identifying gestures, written symbols and a virtual keyboard on a touch screen as claimed in claim 3, wherein: It also includes a press classifier connected to the input identifier, which is used to determine whether the key input matched by the corpus of the touch data frame is supported by the OS. If so, it is transmitted to the OS for processing, and the touch IC is made to perform the initial processing again to process the next touch data frame.

5. The multi-input system for identifying gestures, written symbols and a virtual keyboard on a touch screen as claimed in claim 4, characterized in that: Also included is an AI classifier connected to the press classifier, wherein in the touch data processing, when the press classifier determines that the touch data frame is not supported by the OS, or the input identifier determines that the touch data frame is a non-key input, or the input identifier determines that the virtual keyboard is not displayed when receiving the touch data frame or cannot determine whether the virtual keyboard is displayed, the press classifier inputs the touch data frame into the AI ​​classifier for touch classification; In the touch classification, the AI ​​classifier sends the touch data frame from the press classifier or the input identifier to an AI recognition module located in the AI ​​classifier for recognition, and the AI ​​recognition module outputs the AI ​​recognition result after recognition, and the AI ​​recognition result is stored in the touch data frame to determine whether the touch data frame corresponds to a specific key input, operation gesture or written symbol; In the training phase of the AI ​​recognition module for writing symbol recognition, the input of the AI ​​recognition module is an equation or a character written by the user on the touch screen using a finger or a stylus, and the output of the AI ​​recognition module corresponds to a specific program code; In the AI ​​classifier, when the AI ​​recognition result corresponds to a specific operation gesture, the touch data frame is sent to the OS and processed by the OS, and the touch IC is made to perform the initial processing again to process the next touch data frame.

6. The multi-input system for identifying gestures, written symbols and a virtual keyboard on a touch screen as claimed in claim 5, characterized in that: Also included is a symbol corrector connected to the AI ​​classifier, in which, when the AI ​​recognition result corresponds to the written symbol, the touch data frame including the AI ​​recognition result is transmitted to the symbol corrector for symbol correction; wherein the AI ​​recognition result is regarded as a series of written symbols; the symbol corrector applies a pre-trained symbol model library to perform AI writing symbol prediction and error correction on the AI ​​recognition result; wherein the written symbol is a character or a mathematical equation; The symbol corrector transmits the corrected message to the OS for processing, and enables the touch IC to perform the initial processing again to process the next touch data frame.

7. The multi-input system for identifying gestures, written symbols and a virtual keyboard on a touch screen as claimed in claim 6, characterized in that: When the corrected written symbol is a character or a string, the symbol corrector applies the character editing unit to edit the character or string; when the corrected written symbol is a mathematical equation, and it is a string of specific program codes, the symbol corrector applies the mathematical program code editing language to compile the program code corresponding to the mathematical equation; when the written symbol corresponding to the AI ​​recognition result is not supported by the application, the program code corresponding to the written symbol is directly output.

8. The multi-input system for recognizing gestures, written symbols and a virtual keyboard on a touch screen as claimed in claim 5, characterized in that: A grammar corrector is also included to be connected to the AI ​​classifier. In the AI ​​classifier, when the AI ​​recognition result corresponds to the key input, the touch data frame including the AI ​​recognition result is transmitted to the grammar corrector for grammar correction; wherein the AI ​​recognition result is regarded as a series of input results of the virtual keyboard; the grammar corrector uses a pre-trained language grammar model library to perform AI language grammar prediction and error correction on the AI ​​recognition result; The syntax corrector transmits the corrected message to the OS for processing, and enables the touch IC to perform the initial processing again to process the next touch data frame.

9. The multi-input system for recognizing gestures, written symbols and a virtual keyboard on a touch screen as claimed in claim 1, wherein: The AI ​​recognition module stores the AI ​​recognition result outputted after recognition in the touch data frame to determine whether the touch data frame corresponds to a specific key input, operation gesture or written symbol; During the training phase of the AI ​​recognition module for writing symbol recognition, the input of the AI ​​recognition module is an equation or a character written by the user on the touch screen using a finger or a stylus, and the output of the AI ​​recognition module corresponds to a specific program code.

10. The multi-input system for recognizing gestures, written symbols and a virtual keyboard on a touch screen as claimed in claim 1, wherein: The virtual keyboard also includes a category key for controlling the touch screen to display a symbol input area; the category key is also used to control the touch screen to display a key sequence similar to that of the physical keyboard, or not to display the virtual keyboard or the symbol input area, so as to receive operation gestures input by the user.

11. The multi-input system for recognizing gestures, written symbols and a virtual keyboard on a touch screen as claimed in claim 1, wherein: Also included is a keyboard display controller connected to the pre-processing module, used to receive the touch data frame, and when the touch data frame corresponds to a keyboard display gesture, the touch screen is driven to display the virtual keyboard, and the touch data frame is subsequently processed by the pre-processing module; When the touch screen displays the virtual keyboard, the pre-processing module can simultaneously accept the user's key input on the virtual keyboard and the written symbols input on the touch screen.

12. A fast multi-input keyboard system for use on a touch screen, characterized in that: include: A touch screen, used to receive a user's touch action and generate a corresponding touch signal; wherein the touch screen is a touch panel of an electronic device; The touch screen displays a virtual keyboard; A touch IC is connected to the touch screen and is used to receive a touch signal from the touch screen. The touch IC converts the touch signal into a plurality of touch data. The plurality of touch data form a two-dimensional touch data frame. A processor group is connected to the touch IC and an OS, receives the touch data frame from the touch IC to perform a touch data process, and the touch data process applies an algorithm that can be directly executed by the OS; The virtual keyboard includes multiple function keys, a number key and a symbol key respectively placed on the left and right sides of the virtual keyboard, and multiple character keys placed in the center of the virtual keyboard; the virtual keyboard is divided into an upper row key group and a lower row key group according to the position.

13. The fast multi-input keyboard system for use on a touch screen as claimed in claim 12, characterized in that: The function keys include a Caps Lock key, an Enter key, a first Shift key, a Control key, a Space key, an Alt key, and a second Shift key; the upper row of keys is in a symmetrical M shape to accommodate the posture of pressing keys with both hands, and the lower row of keys is in a concave arc shape. The Control key, the Space key, and the Alt key belong to the lower row of keys and are positioned downward.

14. The fast multi-input keyboard system for use on a touch screen as claimed in claim 13, characterized in that: The upper key group has 12 keys, from left to right, respectively, the Caps Lock key, nine character keys, the symbol key and the enter key; each character key is used to input two or three different English letters; the lower key group includes 6 keys, from left to right, respectively, the first shift key, the Control key, the space key, the Alt key, the number key and the second shift key.

15. The fast multi-input keyboard system for use on a touch screen as claimed in claim 14, characterized in that: The nine character keys are, from left to right on the virtual keyboard, an ABC key corresponding to the letters A, B, and C, a DEF key corresponding to the letters D, E, and F, a GHI key corresponding to the letters G, H, and I, a JKL key corresponding to the letters J, K, and L, a MNO key corresponding to the letters M, N, and O, a PQR key corresponding to the letters P, Q, and R, a STU key corresponding to the letters S, T, and U, a VWX key corresponding to the letters V, W, and X, and a YZ key corresponding to the letters Y and Z; or The nine character keys on the virtual keyboard are, from left to right, a QAZ key corresponding to the letters Q, A, Z, a WSX key corresponding to the letters W, S, X, an EDC key corresponding to the letters E, D, C, an RFV key corresponding to the letters R, F, V, a TGB key corresponding to the letters T, G, B, a YHN key corresponding to the letters Y, H, N, a UJM key corresponding to the letters U, J, M, an IK key corresponding to the letters I, K, and an OLP key corresponding to the letters O, L, P.

16. The fast multi-input keyboard system for use on a touch screen as claimed in claim 13, characterized in that: The multiple character keys include seven upper character keys, a lower left character key and a lower right character key; each character key is used to input two or three different English letters; the upper key group includes 10 keys, which are, from left to right, the Caps Lock key, the seven upper character keys, the symbol key and the enter key; the lower key group includes 8 keys, which are, from left to right, the first shift key, the Control key, the lower left character key, the space key, the lower right character key, the Alt key, the number key and the second shift key; the lower left character key is an MNO key corresponding to the letters M, N, O, or a TGB key corresponding to the letters T, G, B; the lower right character key is a PQR key corresponding to the letters P, Q, R, or a YHN key corresponding to the letters Y, H, N.

17. The fast multi-input keyboard system for use on a touch screen as claimed in claim 16, characterized in that: The seven upper character keys in the upper row of keys are, from left to right, an ABC key corresponding to the letters A, B, C, a DEF key corresponding to the letters D, E, F, a GHI key corresponding to the letters G, H, I, a JKL key corresponding to the letters J, K, L, a STU key corresponding to the letters S, T, U, a VWX key corresponding to the letters V, W, X, and a YZ key corresponding to the letters Y, Z; or the seven upper character keys in the upper row of keys are, from left to right, a QAZ key corresponding to the letters Q, A, Z, a WSX key corresponding to the letters W, S, X, an EDC key corresponding to the letters E, D, C, an RFV key corresponding to the letters R, F, V, a UJM key corresponding to the letters U, J, M, an IK key corresponding to the letters I, K, and an OLP key corresponding to the letters O, L, P.

Images