A program that uses a device to move as if writing letters in space, acquires the movement trajectory, recognizes the letters, and then inputs them.
The character input method tracks device movement to convert 3D or 2D data into recognizable characters, addressing ease of use and privacy concerns, supporting diverse scripts and languages with enhanced recognition.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- 竹内常雄
- Filing Date
- 2024-06-30
- Publication Date
- 2026-06-12
AI Technical Summary
Existing electronic devices face challenges in character input, particularly for those unfamiliar with keyboards or voice recognition, and there is a need for a method that is secure and does not leak personal information to the surroundings.
A character input method that tracks the movement trajectory of a device, allowing users to input characters as if writing on paper, which is secure and does not require extensive training or visible input, using sensors to capture and convert 3D or 2D movement data into recognizable characters.
Enables easy character input for users without prior experience, maintains privacy by disguising input as device movement, and accommodates various languages and scripts, including cursive writing and Morse code, with improved recognition accuracy through machine learning.
Smart Images
Figure 0007873503000001
Abstract
Description
Technical Field
[0001] The character input method of the present invention is a method of inputting characters into an instrument according to the trajectory of the movement of the instrument. Accordingly, a method is provided that enables even those who are not accustomed to character input on electronic devices to easily input characters as if writing characters on paper.
Background Art
[0002] Today, electronic devices have been miniaturized and become wristwatch-type or mobile phone-type, making them very convenient to carry. However, the smaller and more portable an electronic device is, the more likely it is to lack a keyboard or have a small display, making it difficult to input characters even in the case of character input by pressing the display. Recently, due to the development of machine learning and artificial intelligence technologies, the recognition rate of voice recognition, which enables character input by simply speaking words towards such instruments, has also increased, and the number of people using it has been increasing.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] When inputting characters into an electronic device, there is a problem that one wants to input characters more easily without the need for habits or training such as pressing the keyboard displayed on the display with a finger. Although character input by voice recognition is a simple method, in an era when awareness regarding security and personal information protection is high, there is a problem that one wants to input characters in a way that does not easily leak information to the surroundings. Patent Document 1 is a method of inputting characters by handwriting characters on the display, but there is a need for a character input method that is more convenient and does not require time-consuming training and is difficult for information to leak to the surroundings.
Means for Solving the Problems
[0005] The character input method of the present invention is a method of inputting characters into a device by tracking the movement trajectory of the device. Therefore, even people who are not accustomed to inputting characters on electronic devices can easily input characters as if writing on paper. Furthermore, even if observed by others, it appears as if the user is simply shaking the device, making it difficult to determine what characters have been entered, thus solving the problem. [Brief explanation of the drawing]
[0006] [Figure 1] Processing flow of the present invention [Modes for carrying out the invention]
[0007] In the present invention, a device refers to a device that incorporates various sensors, microphones, cameras, and speakers, or a small computer incorporating these, or a computer with communication means capable of communicating with them via wired or wireless connections. Typical examples include smartphones and smartwatches. When multiple devices work together to perform character recognition or input, they must be able to communicate with each other via wireless or wired communication means.
[0008] When inputting text using devices such as smartphones, the main methods are typing on the on-screen keyboard, writing characters on the screen and using character recognition, or using voice input with character recognition. While typing on a keyboard is the easiest once you get used to it, it's still difficult if you're not somewhat familiar with input methods. When writing characters by hand, you have to place the smartphone on a table or hold it firmly in one hand, but you have to write within the limited area of the screen, and you can't write as large as you would on paper, which can be cumbersome. Voice input is fine if you do it in a private room, but it's difficult to use in public places such as trains because others might overhear you. The present invention, which involves shaking a smartphone or other device to form the shape of characters for input, requires no practice as long as you can write characters, can be done with one hand, and is difficult for others to understand even if they see you doing it. Furthermore, even people who cannot sit up or speak due to illness or injury can input text if they can move one hand.
[0009] The characters used for input are not limited to Japanese, hiragana, or katakana; they can be in any language, including English, German, or Arabic script. Any form of writing that can be written on paper is acceptable. Japanese cursive script, the characters used in shorthand, and English cursive script have the advantage of being able to be written quickly in a form close to a single stroke, and this advantage can be utilized in the input method of the present invention.
[0010] It's possible to input text using not just individual letters, but also simpler Morse code. Morse code consists of short and long signals, such as dots and dashes. By shaking the device slightly, you can input text using the short signals, and by shaking it widely, you can input text using the short signals.
[0011] Character input is possible by the number of times and timing of shaking the device. It works in a way similar to pager input, for example, by separating consonants and vowels. The first number represents the row of "a, ka, sa, ta, na, ha, ma, ya, ra, wan," and the next number represents the row of "ai, u, e, o." If you shake it 3 times first and then 5 times, it will be the vowel "o" in the "sa" row, so it will be "so." If you shake it 5 times first and then 2 times, it will be the vowel "i" in the "na" row, so it will be "ni." The order and combination of the number of shakes represent characters, and by shaking it according to these rules, the device can recognize characters and input text.
[0012] As shown in Figure 1, first, the character input process of the character input device is started (100), and the acquisition of the trajectory by the sensor begins. Next, the trajectory of the device itself is acquired by the sensor (200). Then, the trajectory data is converted (300), and character recognition (400) is performed based on that data. The user checks whether the recognized characters are correct (500), and if they are incorrect, the characters are corrected (600), and then the recognized characters are input as data to the device (700). If they are correct, the data is input as is (700).
[0013] To input text, initiate the text input process (100) to enable the acquisition of the device's trajectory. This can be done by starting document input using software such as a word processor, or by clicking a web search bar to enter text input mode.
[0014] Next, regarding the acquisition of the trajectory of the device's movement (200), in the method of the present invention, the device such as a smartphone is shaken to form the shape of the character to be input, and this movement is captured. Various sensors, microphones, speakers, cameras, etc. can be considered, but any device that can grasp the position, direction of movement, and size of the device itself is acceptable, and two or more devices can be combined in any combination. If the trajectory of the device or object being shaken is to be captured by another device rather than the device itself being shaken, any sensor or device can be used as long as it can grasp the position, direction of movement, and size of the device or object being shaken. Two or more devices can be combined in any combination.
[0015] There are various types of sensors and devices in equipment. These include GPS, magnetic sensors, gyroscopes, accelerometers, ambient light sensors, proximity sensors, U1 chips, Soli, LiDAR, WiFi and cellular signals, cameras, illuminance sensors, and barometric pressure sensors. These are just a few examples. Especially in recent smartphones, various sensors are installed.
[0016] The device is equipped with sensors that detect information about a point in space, as well as sensors that detect acceleration and tilt at that time. Any sensor or device is acceptable as long as it can collect data on the device's position, tilt, acceleration, and direction per unit time using one or more of these sensors or combinations, thereby obtaining the trajectory of the device's movement. The data is recorded as three-dimensional data using the X, Y, and Z axes. Any sensor or device is acceptable as long as it can acquire data from one or more sensors or devices to obtain data on the trajectory of the device's movement, and it can be a combination of sensors or devices rather than just one.
[0017] For example, if a sensor that accurately determines the position of the device itself is available, the position in three-dimensional space can be acquired at intervals or intermittently, and the connection of these points can be used to trace the trajectory of the device's movement.
[0018] The strength of the device's acceleration and the direction of the force in three-dimensional space are recorded for each unit of time. These are represented as three-dimensional vector arrows with force and direction. The acceleration is represented by the length of the vector line, and the direction in three-dimensional space is represented by the orientation of the vector line. Connecting these represents the trajectory of the device's movement.
[0019] By emitting sound from a speaker at each unit of time and capturing the reflected sound with a microphone, the position, direction of movement, and distance of the device itself in 3D space are obtained, and by connecting these, the trajectory of the device's movement is recorded.
[0020] Capture the response of an object that has the property of bouncing back when struck by sound waves or radio waves such as radar waves per unit time with a sensor, obtain the position, movement direction, and distance of the instrument itself in three-dimensional space, and connect them to form the trajectory of the instrument's movement.
[0021] Send radio waves such as wireless per unit time, and based on the time returned from one or more instruments, devices, etc. that received it, obtain the position, movement direction, and distance of the instrument itself in three-dimensional space, and connect them to form the trajectory of the instrument's movement.
[0022] Take images per unit time with the camera of the instrument itself that is waved to write characters in space, measure the displacement of the positions of some specific objects shown therein, obtain the movement direction and distance of the instrument itself in three-dimensional space, and connect them to form the trajectory of the instrument's movement.
[0023] Instead of the instrument itself that is waved to write characters in space, use the camera of another instrument to take pictures of the instrument or object being waved and the surrounding images, measure the displacement of the positions of some specific objects shown therein, obtain the movement direction and distance of the instrument itself that is waved to write characters in three-dimensional space, and connect them to form the trajectory of the instrument's movement.
[0024] In addition, various sensors are being developed daily, but any method is acceptable as long as it can grasp the position of the instrument itself. If it can be done, any method is fine.
[0025] Also, when writing characters with the instrument, if the instrument is made to know that it is in the middle of writing characters by pressing the hard key of the instrument or touching the display screen, it is possible to discard unnecessary information such as the movement of connecting characters to characters, and thus improve the recognition accuracy.
[0026] Note that the trajectory data is basically three-dimensional data consisting of three axes, but it is also possible to assume that the instrument is moved only on a plane such as a table as a premise for text input. In that case, it can be obtained as two-dimensional data from the beginning.
[0027] The trajectory data conversion (300) is the process of creating data from the trajectory for character recognition (400). The data created differs depending on whether character recognition (400) can only recognize 2D data or can also work with 3D data. The acquired trajectory is basically 3D data because the device is swung in the air. However, it may be converted to 2D data in some cases.
[0028] The trajectory acquired from the sensor is 3D data consisting of 3 axes. If character recognition (400) can recognize characters using 3D data, the 3D data is passed directly to character recognition (400). If 3D data cannot be recognized as characters, or if 2D character recognition is more accurate than 2D character recognition, then the 3D data needs to be converted to 2D.
[0029] When converting to 2D, there are two methods: first, store the trajectory data acquired using sensors, etc., in 3D and convert it all at once later; or, sequentially convert and arrange it on a 2D plane to record it as a trajectory.
[0030] When a person writes in three-dimensional space, they write as if they were writing on a plane, even though they are in three-dimensional space. Therefore, although the movement of the tool itself takes place in three-dimensional space, the characters are originally two-dimensional, so it is possible to correct the angles and distortions of the lines, which are the trajectory of the tool, sequentially and make them two-dimensional. By sequentially correcting the distortion and curvature of each individual line, it is possible to create two-dimensional data.
[0031] Since the writing is done by moving the tool in space rather than on a flat surface, the angle of the line being drawn will gradually shift from the previously drawn line to the currently drawn line. By sequentially correcting this angle and returning it to the original angle, a 2D data is obtained. The angle shift is corrected sequentially.
[0032] You can perform a sequential conversion, correcting 3D data into 2D one by one, or you can convert 3D trajectories character by character or in larger chunks from 3D to 2D data. When converting in chunks, you should still correct the angles and distortions of the lines in 3D all at once before converting to 2D.
[0033] Furthermore, by adding the constraint that the device is placed on a flat surface such as a table and moved by sliding, it is possible to use this as 2D data without correcting for such angular discrepancies.
[0034] Character recognition (400) utilizes technologies that have been used in OCR and other applications for some time, and its recognition accuracy has improved further with the recent use of machine learning and artificial intelligence. When using 2D data, it is no different from conventional character recognition, and is used as input just like data written on paper for character recognition. However, when writing in space by moving a device, there is a tendency for variations in the size of the characters to occur, with some characters being written large and others small. Therefore, by sequentially adjusting the size of the data and matching the size of the characters, misrecognition can be reduced.
[0035] Recent advancements in machine learning and artificial intelligence technologies have made it possible to output characters even when using 3D data as input. This conversion is performed using a program trained with machine learning and artificial intelligence techniques, taking 3D data of trajectories—the paths left when writing characters in space—as input. In other words, to convert to characters, it's necessary to prepare an appropriate amount of 3D data of trajectories and corresponding characters, and then perform machine learning using supervised learning or similar methods. This allows for a program that converts 3D data to characters. Using such a program, it's possible to directly convert 3D data into characters.
[0036] This is the confirmation of the recognized characters (500). If there is a mistake, the characters must be corrected (600) and then the recognized characters must be entered as data into the device (700). This is the same function used in typical smartphone text input where the user looks at the recognized characters on the display and corrects the misconversion. If the characters are being entered into the device itself, which is being shaken for text input, and it is not linked to other devices, the device's display cannot be seen while text input is in progress, so it is preferable to input all the characters at once, then confirm and correct all the recognized characters at once. This is because stopping the operation after each word input would make the input process feel cumbersome.
[0037] In such cases, you can also input text while checking the results of character recognition by voice using earphones or speakers, rather than using the display.
[0038] One method involves shaking the device with one hand to perform character recognition, confirming the entered characters, and then inputting the characters while looking at the display of another device, such as a smartphone or smartwatch. In this case, the devices need to exchange necessary information, such as 2D data, 3D data, or recognized characters, via some means of communication, such as wired or wireless. Even in this case, it is also possible to input characters while checking the character recognition results through earphones or speakers from the other device, rather than looking at the display of the other device.
[0039] Furthermore, the device can improve recognition accuracy by feeding back the captured trajectory data (which is in 2D or 3D format), the characters derived by character recognition, and the results of corrections made to the characters recognized by the user. By having the device learn the user's unique handwriting style through machine learning, the accuracy of character recognition can be further improved.
[0040] When converting characters using Morse code, a long shake of the instrument produces a long sound, and a short shake produces a short sound. This is then analyzed and converted into characters. The instrument's shakes are separated into long and short shakes, digitized, and then converted. The character input process begins (100), and the trajectory is acquired (200). The method for acquiring the trajectory is the same as before. After that, the trajectory data is digitized (300), and character recognition (400) is performed based on this 2D or 3D data. Here, character recognition is performed by first converting the instrument's trajectory data into long and short sounds, and then converting the combination of these into characters according to the rules of Morse code. The user checks (500) whether the recognized characters are correct. If they are incorrect, the characters are corrected (600), and then the recognized characters are input as data into the instrument (700). If they are correct, the data is input as is (700).
[0041] Character input is possible by the number of times and timing of shaking the device. It works in a way similar to pager input, for example, by separating consonants and vowels. The first number represents the row of "a, ka, sa, ta, na, ha, ma, ya, ra, wan," and the next number represents the row of "ai, u, e, o." If you shake it 3 times first and then 5 times, it will be the vowel "o" in the "sa" row, so it will be "so." If you shake it 5 times first and then 2 times, it will be the vowel "i" in the "na" row, so it will be "ni." The order and combination of the number of shakes represent characters, and by shaking it according to these rules, the device can recognize characters and input text.
[0042] The way the device is swung here is similar to the example of Morse code, but since there is no need to separate long and short tones, recognizing the divisions between swings is crucial. Earlier, I mentioned that you should swing it three times, then five times, but if the divisions are not properly recognized, it will be mistaken for a total of eight swings. The divisions are usually determined by timing. The user must consciously perform actions such as swinging at a specific rhythm, waiting a short time, and then starting the next swing, so that this can be correctly recognized during character recognition (400).
[0043] In addition, users can signal breaks to the device by pressing a hard key on the device or tapping the display. This method is more reliable because it doesn't rely on timing.
[0044] The process begins with character input (100) and trajectory acquisition (200). The method for acquiring the trajectory is the same as before. After that, the trajectory data is converted (300), and character recognition (400) is performed based on this 2D or 3D data. The character recognition performed here first recognizes how many times the device was shaken consecutively and the boundaries between shakes. Based on this, it is converted into characters according to the rules. The user checks (500) whether the recognized characters are correct, and if they are incorrect, the characters are corrected (600) before the recognized characters are input as data into the device (700). If they are correct, the data is input as is (700).
[0045] Furthermore, the method of acquiring a movement trajectory and then inputting characters after character recognition by moving a device as if writing characters in space according to the present invention is not limited to the above embodiment. Various modifications are possible based on the intent of the method of acquiring a movement trajectory and then inputting characters after character recognition by moving a device as if writing characters in space according to the present invention, and these modifications are not excluded from the scope of the method of acquiring a movement trajectory and then inputting characters after character recognition by moving a device as if writing characters in space according to the present invention. [Explanation of symbols]
[0046] 100... Start typing 200... Acquisition of the trajectory of the device itself using sensors. 300... Data conversion of the track 400…Character recognition 500... Is the recognized character correct? 600... Text correction 700... Text input
Claims
1. A program for causing a computer built into a single device held in the hand or worn on the body to function as a character recognition means that recognizes characters using character recognition technology based on the three-dimensional or two-dimensional data, or data converted from three-dimensional data, and transmits the character data recognized by the character recognition means as input data to other software running within the single device, wherein the character recognition means and the linked input means recognize multiple characters from a continuous trajectory without requiring visual inspection of the single device during character input, and after the completion of the input operation, execute a batch confirmation process that makes the stored multiple characters collectively verifiable on the single device.
2. The program according to Claim 1, wherein the sensors are a GPS, magnetic sensor, gyro sensor, accelerometer, ambient light sensor, proximity sensor, U1 chip, Soli, LiDAR, WiFi or mobile phone signal receiver, illuminance sensor, barometric pressure sensor, camera, or speaker and microphone, and the program realizes the function of acquiring the trajectory of the movement of the single device itself by intermittently or at unit time acquisition of information from the sensors, and can be used even if two or more of these sensors are combined, or in any combination.
3. The program according to Claim 2, wherein the sensor is a gyro sensor or an accelerometer, or a combination of both a gyro sensor and an accelerometer, and the program has the function of acquiring the trajectory of the movement of the single device by acquiring the direction of movement and acceleration of the single device itself, and can be used with two or more of these sensors, or any combination thereof.
4. The program according to Claim 2, wherein the sensors are a GPS, a magnetic sensor, an ambient light sensor, a proximity sensor, an illuminance sensor, and a barometric pressure sensor, and the program has the function of acquiring the trajectory of the movement of the single device by acquiring the position and movement information of the single device itself, and can be used even if two or more of these sensors are combined, or in any combination.
5. The program according to Claim 2, wherein the sensor is a speaker and a microphone, and the program has the function of acquiring the position, direction of movement, and distance of the single device itself in three-dimensional space by emitting sound from the speaker at each unit time and capturing the reflected sound with the microphone, thereby acquiring the trajectory of the movement of the single device itself, and can be used even if two or more of these sensors are combined, or in any combination.
6. The program according to claim 2, wherein the sensor is Soli or LiDAR, and the program captures the reflection of sound waves or radio waves bouncing off an object, and acquires the position, direction of movement, and distance of the single device itself in three-dimensional space, thereby realizing the function of acquiring the trajectory of the movement of the single device itself, and can be used even if two or more of these sensors are combined, or in any combination.
7. The program according to Claim 2, wherein the sensor is a receiver that receives radio waves of WiFi or mobile phone signals transmitted from the single device itself at unit time intervals, and the program realizes a function of acquiring the trajectory of the movement of the single device itself by acquiring the position, direction of movement, and distance of the single device itself in three-dimensional space based on the time from the transmission to the reception of the radio waves, and can be used even if two or more of these sensors are combined, or in any combination.
8. The program according to Claim 2, wherein the sensor is a camera, and the program captures an image at unit time intervals and measures the positional shift of a specific object captured therein to obtain the direction and distance of movement of the single device itself in three-dimensional space, thereby realizing the function of obtaining the trajectory of movement of the single device itself, and can be used even if two or more of these sensors are combined, or in any combination.
9. The program according to claim 1, wherein the program recognizes the length of the trajectory as a Morse code signal by treating a small shake of the single device in space as a short signal and a large shake as a long signal, and inputs the characters obtained from the recognized Morse code into the single device itself for use.
10. A system comprising a program according to any one of claims 1 to 9 and a processing unit that executes the program, characterized in that character recognition and transmission of character data to other software are completed within the single device.