A method in which the trajectory of the movement is acquired by moving the tool as if writing characters in space, and the characters are input after character recognition.

Abstract

To provide a character input method that is much simpler, requires no time-consuming training, and hardly causes information leaks to the surroundings in order to meet the need for more easier character input thar requires neither practice nor training such as depression of a keyboard displayed on a display with a finger when characters are input into an electronic apparatus and the need for character input by a method that hardly causes information leaks to the surroundings because character input by voice recognition, although a convenient approach, is incompatible with the current era with high awareness of security and personal information protection.SOLUTION: A character input method of the present invention is a method for inputting characters into an instrument by a trajectory of movement of the instrument. Thus, even a person who is not accustomed to character input of an electronic apparatus can easily input characters just like writing characters on paper.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The character input method of the present invention is a method for inputting characters into an instrument by tracing the trajectory of the instrument's movement, thereby providing a method that allows even those who are not accustomed to inputting characters into electronic devices to input characters as easily as writing characters on paper. [Background technology]

[0002] Nowadays, electronic devices have become smaller, and are now very convenient to carry around, such as wristwatches and mobile phones. However, the smaller and easier to carry electronic devices tend to lack keyboards and have smaller displays, making it difficult to input text by pressing the display. With the recent development of machine learning and artificial intelligence technology, the recognition rate of voice recognition, which allows text to be input simply by speaking into such devices, has improved, and more people are using them. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2014-232347 Summary of the Invention [Problem to be solved by the invention]

[0004] When inputting characters into an electronic device, there is a need to input characters more easily, without the need for training or familiarity with pressing a keyboard displayed on a display with one's fingers. Although inputting characters using voice recognition is a simple method, in an age where security and personal information protection are highly conscious, there is a need to input characters using a method that does not easily leak information to those around. Patent Document 1 describes a method for inputting characters by handwriting characters on a display, but there is a demand for a character input method that is easier, does not require time-consuming training, and is less likely to leak information to those around. [Means for solving the problem]

[0005] The character input method of the present invention involves inputting characters into the device by tracing the trajectory of the device's movement, so that even people who are not accustomed to inputting characters into electronic devices can input characters easily, just like writing on paper. Furthermore, even if people around observe the user, it looks like the user is simply waving the device, making it difficult for them to tell what characters have been input, thereby solving the problem. [Brief description of the drawings]

[0006] [Figure 1] Processing flow of the present invention DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] In the present invention, the term "instrument" refers to a device with various built-in sensors, microphones, cameras, and speakers, a small computer with built-in sensors, or a computer with communication means for wired or wireless communication with the devices. Typical examples are smartphones and smart watches. When multiple devices work together to perform character recognition or input, the devices must be capable of working together via wireless or wired communication means.

[0008] When inputting characters with a device such as a smartphone, the most common methods are to type on the keyboard in the display, write characters on the display and input them through character recognition, or input voice input through character recognition. Although inputting characters with a keyboard is the easiest once you get used to it, it is still difficult if you are not used to inputting characters to a certain extent. When writing characters, you place the smartphone on a desk and write, or hold it firmly in one hand, but you have to write within a limited area of ​​the display and cannot write large as you would on paper, which is cumbersome. Voice input is fine if you are doing it indoors where there are no other people, but it is difficult to implement in public places such as trains because others can hear you. The method of inputting characters by shaking a device such as a smartphone into the shape of a character of the present invention does not require any habit if you can write characters, can be implemented with one hand, and is an input method that makes it difficult for others to understand what you are writing even if they see you. In addition, even if you are unable to sit up or speak due to illness or injury, you can input characters if you can move one hand.

[0009] The characters to be input are not limited to Japanese, Hiragana, and Katakana, but may be any language, such as English, German, or Arabic. Any character shape that can be written on paper is acceptable. Japanese cursive script, characters used in shorthand, and English cursive script have the advantage that they can be written in a single stroke and can be written quickly, and this advantage can also be utilized in the input method of the present invention.

[0010] Instead of using the letters themselves, you can also use the simpler Morse code. Morse code has short and long signals, such as "ton" and "two," and you can input text by shaking the device slightly to get a short signal and shaking it widely to get a long signal.

[0011] Characters can be input by the number of times and timing of shaking the device. It is similar to pager input, for example, divided into consonants and vowels, the first number represents the row of "a ka sa ta na hama ya ra wan", the next number represents the row of "aiueo", if you shake it three times first and then five times, it will become "so" because it is the vowel "o" in the "sa" row, if you shake it five times first and then two times, it will become "ni" because it is the vowel "i" in the "na" row, etc. The order and combination of the number of times you shake it represents the characters, and by shaking it according to this rule, the characters can be recognized and characters can be input.

[0012] As shown in Figure 1, first the character input process of the tool that will input characters is started (100), and the sensor begins to acquire the trajectory. Next, the sensor acquires the trajectory of the tool itself (200). The trajectory is then digitized (300), and character recognition (400) is performed based on that data. The user checks (500) whether the recognized characters are correct, and if they are incorrect, they correct the characters (600), and then the recognized characters are input as data into the tool (700). If they are correct, the data is input as is (700).

[0013] To input text, start the text input process (100) to make it possible to obtain the trajectory of the tool. There are several ways to do this, such as starting document input using software such as a word processor, or clicking on a search box on the web to enter text input mode.

[0014] Next, in the method of the present invention, the movement of the tool is captured by shaking the tool such as a smartphone in the shape of the characters to be input. Various sensors, microphones, speakers, cameras, etc. are possible, but any device can be used as long as it can grasp the position, direction of movement, and size of the tool itself, and any combination of two or more devices can be used. In the case where the tool for inputting characters is not shaken itself, but the trajectory of the tool or object being shaken is grasped with another tool, any sensor or device can be used as long as it can grasp the position, direction of movement, and size of the tool or object being shaken. Any combination of two or more devices can be used.

[0015] There are various sensors and devices in the device: GPS, magnetic sensor, gyro sensor, acceleration sensor, ambient light sensor, proximity sensor, U1 chip, Soli, LiDAR, WiFi and mobile phone signals, camera, light sensor, air pressure sensor. In particular, recent smartphones are equipped with a variety of sensors.

[0016] The instrument is equipped with sensors that detect information about a point in space and sensors that detect the acceleration and tilt at that time. Any sensor or device can be used as long as it can collect data on the position of the instrument itself per unit time and the tilt, acceleration, direction, etc. of the instrument per unit time using one or a combination of these sensors or devices, and obtain the trajectory of the instrument's own movement. It is recorded as three-dimensional data on three axes: the X-axis, Y-axis, and Z-axis. Any sensor or device can be used as long as it can obtain data on the trajectory of the instrument's own movement by acquiring data from one or more sensors or devices, and a combination of sensors or devices is also acceptable.

[0017] For example, if a sensor that can accurately determine the position of the tool itself is available, the tool's position in three-dimensional space can be obtained every unit time or intermittently, and the connections between these points can be used to trace the tool's movement.

[0018] The strength of the acceleration of the instrument and the direction of force in three-dimensional space are recorded for each unit of time and are depicted as three-dimensional vector arrows with force and direction. The acceleration is expressed as the length of the vector line and the direction in three-dimensional space is expressed as the orientation of the vector line. By connecting these, the trajectory of the instrument's movement is created.

[0019] Sound is emitted from a speaker every unit time, and the reverberating sound is captured by a microphone to obtain the position, direction of movement, and distance of the device itself in three-dimensional space. These are then connected to form a trace of the device's movement.

[0020] Every unit of time, a sensor captures the echoes of radar waves and other sound waves, as well as radio waves, which have the property of bouncing off when they hit an object. The position, direction of movement, and distance of the device itself in three-dimensional space are obtained, and these are then connected to create a track of the device's movement.

[0021] Radio waves are emitted per unit time, and the position, direction of movement, and distance of the device itself in three-dimensional space are obtained based on the time it takes for one or more devices or equipment to receive the signal and connect these to create a track of the device's movement.

[0022] To write characters in space, images are taken at unit time intervals using a camera attached to the instrument itself, and by measuring the positional deviations of certain objects captured in the images, the direction and distance of movement of the instrument itself in three-dimensional space is obtained. By connecting these images, the trajectory of the instrument's movement is created.

[0023] Images of the tool being swung to write characters in space, the object being swung, and the surrounding area are captured with a camera on another tool, rather than the tool itself that is swung to write characters in space. By measuring the positional deviation of certain objects captured in the images, the direction and distance of movement in three-dimensional space of the tool being swung to write characters in space is obtained, and by connecting these images, the trajectory of the tool's movement is created.

[0024] In addition, various sensors are being developed every day, but they cannot grasp the position of the equipment itself. Any method is acceptable as long as it is possible to achieve this.

[0025] In addition, when writing characters with an instrument, if a hard key on the instrument is pressed or the display screen is touched to inform the instrument that a character is being written, unnecessary information such as connecting movements between characters will be discarded and not taken in, thereby improving recognition accuracy.

[0026] Although the trajectory data is basically three-dimensional data consisting of three axes, it is also possible to assume that the tool will be moved only on a flat surface such as a table for this character input. In this way, it is possible to obtain two-dimensional data from the beginning.

[0027] The digitization of the trajectory (300) is a process that creates data from the trajectory to be subjected to character recognition (400). The data that is created differs depending on whether the character recognition (400) can only recognize two-dimensional data or whether it can also recognize three-dimensional data. The acquired trajectory is basically three-dimensional data because the tool is swung in the air. However, it may be converted to two-dimensional data in some cases.

[0028] The trajectory obtained from the sensor is three-dimensional data consisting of three axes. If the character recognition (400) can recognize it as three-dimensional data, the three-dimensional data is passed directly to the character recognition (400). If three-dimensional data cannot be recognized as characters, or if two-dimensional and three-dimensional character recognition is more accurate in two dimensions, the three-dimensional data must be converted to two dimensions.

[0029] When converting to two dimensions, one method is to first store the trajectory data acquired using a sensor or other device in three dimensions and then convert it all at once later, or to convert it sequentially into two dimensions and record it as a trajectory.

[0030] When a person writes characters in three-dimensional space, they write as if they are writing on a flat surface. Therefore, the movement of the tool itself takes place in three-dimensional space, but since characters are originally two-dimensional, it is possible to make them two-dimensional by sequentially correcting the angle and distortion of the lines that form the tool's trajectory. By sequentially correcting the distortion and curvature of each line, it is possible to make the data two-dimensional.

[0031] Because characters are written by waving the tool in space, not on a flat surface, the angle of the line being drawn will gradually shift from the previous line. By successively correcting that angle and resetting it to the original angle, the data becomes two-dimensional. The angle deviation is corrected successively.

[0032] You can use a sequential conversion to convert 3D data into 2D data, or you can convert a 3D trajectory into 2D data for each character or a larger chunk of data. When converting in chunks, you also need to convert the angles and distortions of the 3D lines into 2D data.

[0033] Furthermore, if the constraint is added that the instrument be placed on a flat surface such as a table and moved by sliding it, the data can be used as is as 2D data without the need to correct for such angular deviations.

[0034] Character recognition (400) is a technology that has traditionally been used for OCR, etc., and in recent years, the accuracy of recognition has been improved through the use of machine learning and artificial intelligence. When using 2D data, there is no difference from conventional character recognition, so the data is used as input in the same way as data written on paper for character recognition. However, when writing by waving an instrument in space, there is a tendency for characters to be written large and small, which can vary depending on the character. Therefore, by successively adjusting the size of the data and adjusting the size of the characters, it is possible to reduce misrecognition.

[0035] With the recent development of machine learning and artificial intelligence technology, it is now possible to use 3D data as input and output characters. The 3D data of the trajectory of a character swung in space as if writing a character is used as input, and conversion is performed using a program trained using machine learning and artificial intelligence technology. In other words, to convert characters, it is necessary to prepare an appropriate amount of 3D data of the trajectory and the corresponding characters, and perform machine learning using supervised learning, etc. This allows the creation of a program that converts 3D data into characters. Using a program created in this way, it is possible to convert 3D data directly into characters.

[0036] The recognized characters are confirmed (500). If they are incorrect, the characters must be corrected (600) and then the recognized characters must be input as data into the device (700). This is a function used in normal character input on smartphones, where the user can view the recognized characters on the display and correct any misconversions. If characters are being input into the device itself that is being swung to input characters and it is not linked to another device, the display of the device cannot be seen while inputting characters, so it is preferable to input all at once, check the recognized characters all at once, and check and correct them all at once. This is because stopping the operation every time a word is input makes input feel cumbersome.

[0037] In such cases, you can input characters while listening to the results of character recognition aloud using earphones or speakers instead of on a display.

[0038] It is also possible to shake the device with one hand to perform character recognition, and check and input characters by looking at the display of another device, such as a smartphone or smartwatch. In that case, the devices need to exchange necessary information such as 2D data, 3D data, or recognized characters with each other using some kind of communication means, such as wired or wireless. Even in such cases, characters can be input while checking the results of character recognition through earphones or speakers from the other device, rather than looking at the display of the other device.

[0039] In addition, the accuracy of recognition can be improved by feeding back to the device the captured trajectory (2D or 3D data), the characters derived by character recognition, and the results of corrections made by the user to the recognized characters. The accuracy of character recognition can be further improved by having the device learn the user's writing habits through machine learning, etc.

[0040] When converting characters using Morse code, a long shake of the tool produces a long sound, and a short shake of the tool produces a short sound, which is then analyzed and converted into characters. The swing of the tool is divided into long and short swings, converted into data, and then converted. The character input process is started (100), and the trajectory is acquired (200). The method of acquiring the trajectory is the same as before. After that, the trajectory is converted into data (300), and character recognition (400) is performed based on the two-dimensional or three-dimensional data. The character recognition performed here first converts the tool's trajectory data into long and short sounds, and converts the combination into characters according to the rules of Morse code. The user checks (500) whether the recognized characters are correct, and if they are incorrect, corrects the characters (600), and then inputs the recognized characters into the tool as data (700). If they are correct, the data is input as is (700).

[0041] Characters can be input by the number of times and timing of shaking the device. It is similar to pager input, for example, divided into consonants and vowels, the first number represents the row of "a ka sa ta na hama ya ra wan", the next number represents the row of "aiueo", if you shake it three times first and then five times, it will become "so" because it is the vowel "o" in the "sa" row, if you shake it five times first and then two times, it will become "ni" because it is the vowel "i" in the "na" row, etc. The order and combination of the number of times you shake it represents the characters, and by shaking it according to this rule, the characters can be recognized and characters can be input.

[0042] The swinging of the device here is similar to the Morse code example, but since there is no need to separate long and short sounds, it is important to recognize the divisions of the swing. As mentioned earlier, you swing three times, then five times, but if the divisions are not properly recognized, it will be mistaken for eight swings in total. Divisions are usually made by timing. The user must be aware of swinging at a specific rhythm, wait a short time, and then start the next swing, and this must be correctly recognized during character recognition (400).

[0043] Alternatively, the user can notify the device of the break by pressing a hard key on the device or tapping the display. This provides more reliable data since it does not rely on timing.

[0044] The character input process starts (100) and the trajectory is acquired (200). The method of acquiring the trajectory is the same as before. After that, the trajectory is digitized (300) and character recognition (400) is performed based on the 2D or 3D data. The character recognition performed here first recognizes how many consecutive shakes have been made and where the breaks are. Based on this, it is converted into characters according to rules. The user checks (500) whether the recognized characters are correct, and if they are incorrect, they correct the characters (600) before inputting the recognized characters into the device as data (700). If they are correct, the data is input as is (700).

[0045] It should be noted that the method of the present invention of obtaining a trajectory of movement by moving an instrument as if writing characters in a space, recognizing the characters, and then inputting the characters is not limited to the above embodiment, and various modifications are possible based on the spirit of the method of the present invention of obtaining a trajectory of movement by moving an instrument as if writing characters in a space, and then inputting the characters after character recognition, and these are not excluded from the scope of the method of the present invention of obtaining a trajectory of movement by moving an instrument as if writing characters in a space, and then inputting the characters after character recognition. [Explanation of symbols]

[0046] 100…Start entering text 200...Acquisition of the trajectory of the instrument itself using a sensor 300...Trajectory data 400…Character recognition 500...Is the recognized character correct? 600…Text correction 700…Character input

Claims

1. A method comprising the steps of: moving a single instrument held in the hand or worn on the body in the shape of a character by waving it through space; obtaining a trajectory of the movement of the single instrument itself using a sensor equipped on the single instrument that is capable of obtaining the position or movement of the instrument itself; obtaining three-dimensional or two-dimensional data of the trajectory of the movement of the single instrument, or data converted from three-dimensional data to two-dimensional data; recognizing characters based on this three-dimensional or two-dimensional data using character recognition technology; inputting the recognized characters into the single instrument itself; and enabling the characters to be used by the single instrument itself.

2. The method described in claim 1, wherein the sensor is a GPS, a magnetic sensor, a gyro sensor, an acceleration sensor, an ambient light sensor, a proximity sensor, a U1 chip, Soli, LiDAR, a receiver for receiving WiFi or mobile phone signals, an illuminance sensor, an air pressure sensor, a camera, or a speaker and a microphone, and information from the sensor is obtained intermittently or at unit time intervals to obtain a trajectory of the movement of the single device itself, and two or more of these sensors may be combined or used in any combination.

3. The method described in claim 2, wherein the sensor is a gyro sensor or an acceleration sensor, or a combination of both a gyro sensor and an acceleration sensor, and obtains the direction and acceleration of the movement of the single instrument itself, thereby obtaining the trajectory of the movement of the single instrument itself, and two or more of these sensors may be combined or used in any combination.

4. The method described in claim 2, wherein the sensors are a GPS, a magnetic sensor, an ambient light sensor, a proximity sensor, a luminance sensor, and a barometric pressure sensor, and the position and movement information of the single device itself is obtained to obtain a trajectory of the movement of the single device itself, and two or more of these sensors may be combined and used in any combination.

5. The method described in claim 2, wherein the sensor is a speaker and a microphone, and by emitting sound from the speaker per unit time and capturing the reverberating sound with the microphone, the position, direction of movement, and distance of the single instrument itself in three-dimensional space are obtained, thereby obtaining the trajectory of the movement of the single instrument itself, and two or more of these sensors may be combined or used in any combination.

6. The method described in claim 2, wherein the sensor is Soli or LiDAR, and captures the echo of sound waves or radio waves bouncing off an object, and obtains the position, direction of movement, and distance of the single instrument itself in three-dimensional space, thereby obtaining the trajectory of the movement of the single instrument itself, and two or more of these sensors may be combined or used in any combination.

7. The sensor is a receiver that receives a wireless radio wave of a WiFi or mobile phone signal transmitted from a single device itself per unit time, and the radio wave is transmitted and then received. The method of claim 2, wherein the position, direction of movement, and distance of the single instrument itself in three-dimensional space are obtained based on the time it takes for the sensor to receive a signal, thereby obtaining a trajectory of the movement of the single instrument itself, and wherein two or more of these sensors may be combined in any combination.

8. The method described in claim 2, wherein the sensor is a camera that takes images every unit time and measures the positional shift of a specific object captured in the images to obtain the direction and distance of movement of the single instrument itself in three-dimensional space, thereby obtaining the trajectory of the movement of the single instrument itself, and wherein two or more of these sensors may be combined and used in any combination.

9. A method as described in claim 1, in which a small wave of the device in space represents a short signal and a large wave represents a long signal, thereby recognizing the length of the trajectory as a Morse code, and the characters obtained from the recognized Morse code are input into the single device itself and used.

Images