Learning system, inference system, learning method, computer program, and trained model

Abstract

To provide a leaning system which identifies a handwritten character without considering a handwriting range, at a lower cost.SOLUTION: In a learning inference apparatus of a character recognition system, a character recognition program 40 of a processor 20 includes: a dataset registration unit 403 which generates, for every multiple characters, a dataset 56 including correct answer data 55 indicating an identifier of a character and image data 54 of a force degree that represents a degree of force of a finger acting on a digital pen 3 in each time during writing with the digital pen 3, causes a dataset storage unit 404 to store the dataset; and a machine learning unit 405 which executes machine learning by using the correct data 55 and the image data 54 included in the dataset 56 as training data and input data, respectively, to generate a trained model 61.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to AI technology for recognizing so-called handwritten characters. [Background technology]

[0002] 2. Description of the Related Art Conventionally, a technology has been widely used in which characters handwritten by a user are recognized by a computer and converted into text data.

[0003] For example, according to the technology described in Non-Patent Document 1, when writing is done with a special pen equipped with a camera on special paper on which multiple special patterns are printed, the camera reads the patterns. Then, a computer calculates the handwriting based on the results of reading the patterns, and identifies the characters represented by the handwriting.

[0004] According to the technology described in Non-Patent Document 2, when writing is done with a finger or a slider on a smart pad, a trajectory is detected based on the pressure of the finger or the slider. Then, the character represented by the detected handwriting is identified by a computer. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] "Smart Writing Set," Moleskine, retrieved February 12, 2021, https: / / www.moleskine.co.jp / smartwritingset / p0202 [Non-Patent Document 2] "How to use Bamboo Slate and Bamboo Folio", Wacom, retrieved February 12, 2021, https: / / www.wacom.com / ja-jp / getting-started / bamboo-slate-and-bamboo-folio Summary of the Invention [Problem to be solved by the invention]

[0006] According to the conventional techniques described in Non-Patent Document 1 or Non-Patent Document 2, text data of characters can be obtained without scanning paper on which characters are written. However, special paper or a special product such as a smart pad is required.

[0007] However, these dedicated products have a limited range in which characters can be written. Therefore, they cannot recognize characters written on a wide paper such as B4 or A3 size. It would be possible to prepare a dedicated product with a wider range, but this would be inconvenient to carry around and expensive, making it unrealistic.

[0008] In view of the above problems, an object of the present invention is to make it possible to identify handwritten characters at a lower cost than conventional methods and without taking into account the range of characters that can be written. [Means for solving the problem]

[0009] The learning system according to one embodiment of the present invention is configured to, for each of a plurality of characters, generate a character identifier that identifies the character, and a force intensity value that indicates the degree of force that a finger exerts on the writing implement at each time while the character is being written with the writing implement. and the acceleration and angular velocity occurring in the writing implement; and acquiring a data set including the character identifiers included in the data set as training data, and , the acceleration, and the angular velocity Using as input data For CNN (Convolutional Neural Network) and a learning means for generating a trained model by performing machine learning.

[0010] Preferably, The acquisition means acquires, as the data set, a first action force degree, a second action force degree, and a third action force degree which are degrees of force that the finger exerts on the writing implement in a first direction, a second direction, and a third direction, respectively, as the action force degree, a first acceleration, a second acceleration, and a third acceleration which are accelerations on the X-axis, the Y-axis, and the Z-axis, respectively, as the acceleration, and a first angular velocity, a second angular velocity, and a third angular velocity which are angular velocities on the X-axis, the Y-axis, and the Z-axis, respectively, as the angular velocity, and the learning means acquires the first action force degree, the second action force degree, and the The machine learning is performed using as input data image data of a displacement image in which an action force image having two consecutive action force images arranged in a first line image, a second line image, and a third line image each representing a third action force, two consecutive acceleration images arranged in a fourth line image, a fifth line image, and a sixth line image each representing the first acceleration, the second acceleration, and the third acceleration, respectively, for a given time, and two consecutive angular velocity images arranged in a seventh line image, an eighth line image, and a ninth line image each representing the first angular velocity, the second angular velocity, and the third angular velocity, respectively, for a given time.

[0011] A learning system according to another embodiment of the present invention has an acquisition means for acquiring a dataset including, for each of a plurality of characters, a character identifier for identifying the character, and a first action force degree, a second action force degree, and a third action force degree which are the degrees of force exerted by a finger on the writing instrument in a first direction, a second direction, and a third direction, respectively, at each time while the character is being written with the writing instrument, and a learning means for generating a learned model by performing machine learning on a Convolutional Neural Network (CNN) using the character identifiers included in the dataset as training data and image data of a displacement image in which an action force degree image in which a first line image, a second line image, and a third line image are arranged to represent the first action force degree, the second action force degree, and the third action force degree, respectively, for each time included in the dataset as input data. Effect of the Invention

[0012] According to the present invention, it is possible to make it possible to identify handwritten characters at a lower cost than before and without taking into consideration the range in which characters can be written. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a character recognition system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning inference device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a learning inference device. [Figure 4] FIG. 2 is a diagram showing an example of the appearance of a digital pen. [Diagram 5] 1 is a cross-sectional view showing an example of a digital pen cut at a grip. [Figure 6] FIG. 2 is a diagram illustrating an example of a hardware configuration of a digital pen. [Figure 7] 11A to 11C are diagrams illustrating examples of changes in acceleration, angular velocity, and force values. [Figure 8] 13 is a flowchart illustrating an example of the flow of a data set generation process. [Figure 9] FIG. 2 is a diagram showing an example of a displacement image represented by image data. [Figure 10] FIG. 1 is a diagram illustrating an example of a neural network. [Figure 11] 11 is a flowchart illustrating an example of an overall processing flow by a character recognition program. [Figure 12] FIG. 13 is a diagram showing a modified example of a neural network. [Figure 13] FIG. 13 is a diagram showing a modified example of a neural network. [Figure 14] FIG. 13 is a diagram showing a modified example of image data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] [Overall structure] FIG. 1 is a diagram showing an example of the overall configuration of a character recognition system 1. FIG. 2 is a diagram showing an example of the hardware configuration of a learning inference device 2. FIG. 3 is a diagram showing an example of the functional configuration of the learning inference device 2. FIG. 4 is a diagram showing an example of the appearance of a digital pen 3. FIG. 5 is a cross-sectional view showing an example of the digital pen 3 cut at a grip 37. FIG. 6 is a diagram showing an example of the hardware configuration of the digital pen 3.

[0015] 1 is a system that recognizes characters written by a user using AI (Artificial Intelligence) technology, and is composed of a learning inference device 2 and a digital pen 3. The learning inference device 2 and the digital pen 3 are connected to each other by wire or wirelessly.

[0016] The learning and inference device 2 generates a learned model by machine learning and infers characters written by a user based on the learned model. In the following, an example will be described in which a personal computer is used as the learning and inference device 2.

[0017] As shown in FIG. 2, the learning inference device 2 is composed of a processor 20, a RAM (Random Access Memory) 21, a ROM (Read Only Memory) 22, an auxiliary storage device 23, a network interface 24, a serial interface 25, a wireless communication device 26, a display 27, a keyboard 28, and a pointing device 29.

[0018] In addition to an operating system, computer programs such as a character recognition program 40 are installed in the ROM 22 or the auxiliary storage device 23. The character recognition program 40 realizes functions such as a trimming processing unit 401, a displacement image generating unit 402, a data set registration unit 403, a data set storage unit 404, a machine learning unit 405, a trained model storage unit 406, a character inference unit 407, and an inference result output unit 408 shown in FIG.

[0019] The RAM 21 is the main memory of the learning and inference device 2. Computer programs such as the character recognition program 40 are loaded into the RAM 21 as appropriate.

[0020] The processor 20 executes a computer program loaded into the RAM 21. As the processor 20, a GPU (Graphics Processing Unit) or a CPU (Central Processing Unit) is used.

[0021] The network interface 24 communicates with other devices using protocols such as TCP / IP (Transmission Control Protocol / Internet Protocol). As the network interface 24, a NIC (Network Interface Card) or a communication device for Wi-Fi is used.

[0022] The serial interface 25 communicates with the peripheral devices by a serial communication method. A board conforming to a standard such as USB (Universal Serial Bus) is used as the serial interface 25. The wireless communication device 26 communicates with the peripheral devices by short-range wireless communication. A board conforming to a standard such as Bluetooth is used as the wireless communication device 26. In this embodiment, the serial interface 25 and the wireless communication device 26 are used particularly for communicating with the digital pen 3.

[0023] The display 27 displays a screen for inputting commands or data, a screen showing the results of calculations performed by the processor 20, and the like.

[0024] The keyboard 28 and the pointing device 29 are input devices for an operator to input commands, data, and the like.

[0025] As shown in FIG. 4, FIG. 5, or FIG. 6, the digital pen 3 is made up of a case 31, a cartridge 32, a microcontroller 33, a serial interface 34, three force sensors 35, and three operational amplifiers 36, etc.

[0026] The case 31 is a cylindrical housing, and houses the cartridge 32, the microcontroller 33, the serial interface 34, the force sensor 35, and the operational amplifier 36. The grip 37 of the case 31 is provided with three recesses 38. Hereinafter, the recesses 38 may be referred to as a "first recess 38A," a "second recess 38B," and a "third recess 38C," respectively. The user holds the digital pen 3 so that his or her thumb, index finger, and middle finger touch the first recess 38A, the second recess 38B, and the third recess 38C, respectively.

[0027] The cartridge 32 is a so-called replacement cartridge. The main body of the cartridge 32 is filled with ink, and the ink flows out onto the paper when the ball at the tip of the cartridge 32 rotates on the paper.

[0028] The microcontroller 33 is a device for collecting various information when the user is writing characters and providing it to the learning and inference device 2, and is composed of a processor 301, a memory 302, an acceleration sensor 303, a gyro sensor 304, an AD converter 305, and a wireless communication device 306, etc.

[0029] The processor 301 is a calculation device such as a CPU, and executes a data transfer program stored in the memory 302 .

[0030] The acceleration sensor 303 detects acceleration on each of the three axes (X-axis, Y-axis, and Z-axis). The maximum value of the acceleration detected by the acceleration sensor 303 is the acceleration a max The gyro sensor 304 detects the angular velocity of each of the three axes. The maximum value of the angular velocity detected by the gyro sensor 304 is the angular velocity ω maxIt should be noted that one IMU (Inertial Measurement Unit) may be used as the acceleration sensor 303 and the gyro sensor 304.

[0031] The AD converter 305 converts the analog signal input from the force sensor 35 via the operational amplifier 36 into a digital signal.

[0032] The wireless communication device 306 communicates with the learning and inference device 2 by short-range wireless communication based on the Bluetooth standard.

[0033] As the microcontroller 33, a commercially available microcontroller such as Arduino Nano-33-BLE sense may be used.

[0034] The serial interface 34 communicates with the learning and inference device 2 via a serial communication method such as USB.

[0035] The force sensor 35 is a sensor that detects the magnitude of a force, and three of them are provided in the grip 37 so as to surround the cartridge 32. These force sensors 35 can detect the force acting on the grip 37 by breaking it down into three directional components. A commercially available force sensor such as the HSFPAR303A from Alps Alpine Co., Ltd. is used as the force sensor 35. The force sensor 35 detects a maximum force value F max It is possible to detect a force up to 100 . Hereinafter, the force sensor 35 provided on the opposite side of the first recess 38A across the cartridge 32 will be referred to as the "first force sensor 35A." Similarly, the force sensors 35 provided on the opposite sides of the second recess 38B and the third recess 38C will be referred to as the "second force sensor 35B" and the "third force sensor 35C."

[0036] In this embodiment, the three force sensors 35 detect the magnitude of forces from three directions acting on the grip 37 when the user is holding the digital pen 3 to write characters. In particular, when the user is writing characters by placing the tip of the digital pen 3 against paper, a force is applied to the tip. Then, a reaction to this force is generated in the opposite direction and acts on the user's fingers, and the force of the fingers acts on the grip 37.

[0037] An operational amplifier 36 is provided for each force sensor 35, and amplifies an analog signal output from the force sensor 35. For example, an AD623ARMZ (8-Lead Mini Small Outline Package) manufactured by Analog Devices is used as the operational amplifier 36. In addition, resistors and capacitors are used to connect the force sensors 35 and the operational amplifier 36.

[0038] Hereinafter, the operational amplifiers 36 corresponding to the first force sensor 35A, the second force sensor 35B, and the third force sensor 35C, respectively, may be distinguished and described as the "first operational amplifier 36A," the "second operational amplifier 36B," and the "third operational amplifier 36C."

[0039] Hereinafter, the processing of each part of the learning and inference device 2 shown in FIG. 3 and each part of the digital pen 3 shown in FIGS. 4 to 6 will be broadly divided into a data set preparation phase, a machine learning phase, and an inference phase and will be described.

[0040] [Preparing the dataset] Figure 7 shows the acceleration a x , a y , a z , angular velocity ω x , ω y , ω z , and changes in the force values ​​F1, F2, and F3. Fig. 8 is a flowchart illustrating an example of the flow of a data set generation process. Fig. 9 is a diagram illustrating an example of a displacement image represented by image data 54.

[0041] The operator of the character recognition system 1 has the user write each of a number of predetermined characters multiple times (e.g., 300 times) on a piece of paper placed on a desk with the digital pen 3. When generating a trained model for one specific user, the operator has the user write all of the characters. When generating a general-purpose trained model, the operator has multiple users write the characters.

[0042] In the following, we will explain an example in which 26 lowercase letters from "a" to "z" are machine-learned as multiple characters, with 10 users writing the same character 30 times in succession for each character. In this case, data of 26 characters x 30 times x 10 people is collected by the process described below.

[0043] While the user is writing with the digital pen 3, the acceleration sensor 303 (see FIG. 6) detects accelerations a x , a y , a z The gyro sensor 304 continues to detect the angular velocities ω x , ω y , ω z The first force sensor 35A, the second force sensor 35B, and the third force sensor 35C continue to detect the magnitudes of forces F1, F2, and F3 generated in the first recess 38A, the second recess 38B, and the third recess 38C, respectively, as shown in FIG. 7C, and continue to output analog signals indicating the detected forces to the first operational amplifier 36A, the second operational amplifier 36B, and the third operational amplifier 36C.

[0044] The first operational amplifier 36A, the second operational amplifier 36B, and the third operational amplifier 36C amplify the analog signals from the first force sensor 35A, the second force sensor 35B, and the third force sensor 35C, respectively, and output the amplified signals to the AD converter 305.

[0045] The AD converter 305 converts the analog signals input from the first operational amplifier 36A, the second operational amplifier 36B, and the third operational amplifier 36C into digital signals, thereby generating force sequence data 51, which is electronic data indicating the force values ​​F1, F2, and F3 at each time.

[0046] The serial interface 34 receives the force sequence data 51 and the acceleration a at each time detected by the acceleration sensor 303. x , a y , a z and the angular velocity ω at each time detected by the gyro sensor 304. x , ω y , ω z The data is transmitted to the learning and inference device 2 as a set together with angular velocity sequence data 53 indicating the above. Alternatively, the wireless communication device 306 may transmit these data.

[0047] The above processing in the digital pen 3 is realized by the processor 301 controlling each unit based on a data transfer program.

[0048] Furthermore, the user or developer inputs the characters handwritten by the user to the learning and inference device 2 by operating the keyboard 28 or the pointing device 29. In other words, the user or developer provides the learning and inference device 2 with the characters that are the correct answer.

[0049] In the learning and inference device 2, a data set for machine learning is generated by the trimming processing unit 401, the displaced image generating unit 402, and the data set registering unit 403 (see FIG. 3) in the procedure shown in FIG.

[0050] When the trimming processing unit 401 receives a set of force sequence data 51, acceleration sequence data 52, and angular velocity sequence data 53 and obtains the correct character (#801, #802 in FIG. 8), it extracts data for the time period from the start to the end of writing for each writing of the character from each of the force sequence data 51, acceleration sequence data 52, and angular velocity sequence data 53 as follows (#803).

[0051] It is considered that the digital pen 3 is placed on the paper and a character is started to be written when the size of the range of the force value F1 (the difference between the maximum and minimum values) continues to be within a certain value for a certain period of time or more, and then the size of this range exceeds this certain value. In addition, it generally takes about 1.0 to 1.5 seconds to write a character once.

[0052] Therefore, the data set registration unit 403 detects this timing, and extracts data in section L, whose start time and end time are a predetermined time (e.g., 0.13 seconds) before this timing and a predetermined time (e.g., 1.43 seconds) after this timing, from the force sequence data 51, the acceleration sequence data 52, and the angular velocity sequence data 53. The following description will be given taking an example in which the length (time) of section L is 1.56 seconds. Data indicating force values ​​F1, F2, and F3 at each time in section L, acceleration a x , a y , a z Data showing the angular velocity ω x , ω y , ω z The data indicating these are referred to as "force data 510," "acceleration data 520," and "angular velocity data 530," respectively.

[0053] According to the process of step #803, 30 sets of force data 510, acceleration data 520, and angular velocity data 530 are extracted from one set of force sequence data 51, acceleration sequence data 52, and angular velocity sequence data 53.

[0054] The trimming processing unit 401 may extract the force data 510, the acceleration data 520, and the angular velocity data 530 by other methods. For example, a time for writing characters may be determined in advance, and the user may write the characters. That is, the user writes characters once at that time interval. For example, the user writes characters once every two seconds.

[0055] Then, the trimming processing unit 401 may extract the data in each time period as force data 510, acceleration data 520, and angular velocity data 530 from the force sequence data 51, acceleration sequence data 52, and angular velocity sequence data 53, respectively. The start of each time period may be notified to the user by a beep sound.

[0056] The displacement image generating unit 402 generates image data 54 representing a displacement image as shown in Fig. 9 based on a set of force data 510, acceleration data 520, and angular velocity data 530 (#804). The displacement image is made up of 18 lines. Each line is made up of 240 pixels.

[0057] The first, second, and third lines respectively represent the displacement of the force values ​​F1, F2, and F3 by shading. Specifically, the force values ​​F1, F2, and F3 shown in the force data 510 are sampled at L / 240, i.e., 1.56 seconds / 240=6.5 milliseconds, and starting from the leftmost pixel, the force values ​​F1, F2, and F3 for every 6.5 milliseconds from the start time are represented by shading (gradation) according to their magnitude.

[0058] The seventh, eighth, and ninth lines are the force values ​​F1 and F2, respectively, sampled at the same rate as the acceleration a x , a y , a z The displacement is represented by the shade of gray. In other words, starting from the leftmost pixel, the acceleration a x , a y , a z are represented by a density according to their size.

[0059] The 13th, 14th, and 15th lines are the angular velocity ω x , ω y , ω z In other words, starting from the leftmost pixel, the displacement of the angular velocity ω x , ω y , ω z are represented by a density according to their size.

[0060] The 4th, 5th and 6th lines are copies of the 1st, 2nd and 3rd lines respectively, the 10th, 11th and 12th lines are copies of the 7th, 8th and 9th lines respectively, the 16th, 17th and 18th lines are copies of the 13th, 14th and 15th lines respectively.

[0061] In other words, the displacement image is a grayscale image consisting of 240×18 pixels. When generating a grayscale image with 16 gradations as the displacement image, the gradations of the force values ​​F1, F2, and F3 are expressed as int{(16 / P max )×P1}, int{(16 / P max )×F2}, int{(16 / P max )×F3}, where int(X) is the integer part of X. Similarly, the acceleration a x , a y , a z Each gradation is int{(16 / a max )×a x}, int {(16 / a max )×a y}, int {(16 / a max )×a z}, and the angular velocity ω x , ω y , ω z Each gradation is int{(16 / ω max )×ω x}, int{(16 / ω max )×ω y}, int{(16 / ω max )×ω z},.

[0062] The process of step #804 is executed for each of the 30 sets of force data 510, acceleration data 520, and angular velocity data 530. Therefore, according to the process of step #804, 30 pieces of image data 54 are generated.

[0063] Then, the dataset registration unit 403 generates 30 datasets 56 (#805) by merging the correct answer data 55 indicating the correct character acquired in step #802 with each of the 30 pieces of image data 54, and stores the 30 datasets 56 in the dataset storage unit 404 (#806). Strictly speaking, the correct answer data 55 indicates an identifier, for example, an ASCII code, corresponding to the correct character.

[0064] The process of steps #803 to #806 is executed every time one set of force sequence data 51, acceleration sequence data 52, and angular velocity sequence data 53 is received and a correct character is obtained. Therefore, by the above-mentioned process of the data set registration unit 403, 26 [characters] × 30 [times] × 10 [people] = 7800 data sets 56 are generated and stored in the data set storage unit 404.

[0065] [Machine Learning] FIG. 10 is a diagram showing an example of a neural network 60. As shown in FIG.

[0066] The machine learning unit 405 (see FIG. 3) generates a trained model by performing machine learning based on the dataset 56 stored in the dataset storage unit 404. Below, the machine learning method will be described using an example in which a CNN (Convolutional Neural Network) is used.

[0067] A neural network 60 is prepared in advance in the machine learning unit 405. The neural network 60 is a CNN, and as shown in Fig. 10, is composed of an input layer 601, a first convolutional layer 602, a first pooling layer 603, a second convolutional layer 604, a second pooling layer 605, a fully connected layer 606, and an output layer 607. The machine learning unit 405 performs the following calculations in each layer.

[0068] The input layer 601 is composed of 240×18 nodes, and receives the displacement image shown in the image data 54 included in the data set 56 as input.

[0069] The first convolutional layer 602 convolves the displacement image input to the input layer 601 with a filter (kernel) and substitutes each obtained value into an activation function to generate a first feature map 57A. As the activation function, for example, a sigmoid function, a tanh function, or a ReLU function is used. The first feature map 57A is output to the first pooling layer 603. Below, an example will be described in which the processing of the first convolutional layer 602 is performed under the conditions (1-1) to (1-5). (1-1) Convolve with 70 different filters. (1-2) The filter size is 2x2. (1-3) The lateral stride is 1. (1-4) The vertical stride is 1. (1-5) Perform zero padding.

[0070] That is, as can be seen from (1-1) and (1-5), in this embodiment, when one image data 54 is input to the input layer 601, 70 feature maps, each consisting of 240×18 nodes, are obtained as the first feature map 57A.

[0071] The first pooling layer 603 generates a second feature map 57B by reducing the first feature map 57A output from the first convolution layer 602 using a filter. Below, an example of compression using a filter with a size of 3×2 will be described. Therefore, in this embodiment, a feature map with a size of 80×9 is generated as the second feature map 57B from the first feature map 57A. Furthermore, the reduction may be performed by any of the methods of maximum value pooling, average value pooling, and Lp pooling, but it is preferable to perform the reduction by maximum value pooling.

[0072] As described above, 70 first feature maps 57A are generated from one image data 54 by the first convolutional layer 602, and therefore 70 second feature maps 57B are generated from one image data 54. These 70 second feature maps 57B are output to the second convolutional layer 604 as a set.

[0073] The second convolutional layer 604 generates a third feature map 57C by convolving a set (i.e., 70) of second feature maps 57B with a filter and substituting each obtained value into an activation function. As the activation function, for example, a sigmoid function, a tanh function, or a ReLU function is used. The third feature map 57C is output to the second convolutional layer 604. Below, an example will be described in which the processing of the second convolutional layer 604 is performed under the conditions (2-1) to (2-5). (2-1) Convolve with 140 different filters. (2-2) The filter size is 70x3x3. (2-3) The lateral stride is 1. (2-4) The vertical stride is 1. (2-5) Perform zero padding.

[0074] That is, as can be seen from (2-1) and (2-5), in this embodiment, 140 feature maps each consisting of 80×9 nodes are obtained as a set (70 pieces) of second feature maps 57B to third feature maps 57C.

[0075] The second pooling layer 605 generates a fourth feature map 57D by reducing the third feature map 57C output from the second convolution layer 604 using a filter. Hereinafter, a case where compression is performed using a filter of 3×3 size will be described as an example. Therefore, in this embodiment, 140 feature maps of 27×3 size are generated as the fourth feature map 57D from one set of the third feature map 57C. Although the reduction may be performed by any method, it is preferable to perform the reduction by maximum value pooling. The 140 fourth feature maps 57D are output to the fully connected layer 606 as one set.

[0076] The fully connected layer 606 is a neural network that calculates weighted connections in the same way as a multi-layer perceptron, and calculates the value of each unit (node) by an activation function. In this embodiment, a trained model for inferring 26 lowercase letters of the alphabet as characters is generated, so the feature variables (likelihood) of each of the 26 lowercase letters are calculated. ReLU is used as the activation function. A sigmoid function or a tanh function may be used instead of ReLU.

[0077] The output layer 607 has a node corresponding to each of the 26 lowercase letters of the alphabet. All feature variables from the 26 units in the bottom layer of the fully connected layer 606 are input to each node. Each node then calculates the likelihood of the character (lowercase letter) corresponding to itself using a likelihood function such as a softmax function.

[0078] The machine learning unit 405 uses some of the 300 data sets 56 for each of the 26 characters stored in the data set storage unit 404 as training data and the remainder as validation data to train the neural network 60. At this time, for each of the 26 characters, for example, 80%, i.e., 240 data sets 56 are used as training data and the remaining 20%, i.e., 60 data sets 56 are used as validation data.

[0079] Furthermore, the learning data (data set 56) includes image data 54 and correct answer data 55, and the machine learning unit 405 uses the image data 54 as input data (explanatory variable data) and the correct answer data 55 as teacher data (objective variable data, correct answer data). That is, the machine learning unit 405 inputs the image data 54 to the neural network 60 and obtains the likelihood of each character by performing the calculation shown in FIG. 10. Then, the parameters (e.g., weighting coefficients or bias values) of each activation function of the neural network 60 are adjusted so that the likelihood corresponding to the character shown in the correct answer data 55 among these likelihoods is as close to "1" as possible. This process is performed for all learning data.

[0080] After adjusting the parameters of each activation function of the neural network 60 using 240 x 26 pieces of training data, the machine learning unit 405 verifies the accuracy of the adjusted neural network 60 using 60 x 26 pieces of validation data (data set 56).

[0081] That is, the verification data includes image data 54 and correct answer data 55, and when image data 54 is input to input layer 601 of neural network 60, the likelihood of each of 26 characters (lowercase alphabetical characters) is output from each node of output layer 607. Machine learning unit 405 verifies whether or not the character corresponding to the node with the highest likelihood matches the character shown in correct answer data 55. If they match, it is correct, and if they do not match, it is incorrect.

[0082] This kind of verification is carried out for 60 x 26 pieces of verification data. If the accuracy rate, i.e., the number of correct answers to the number of verification data, is a predetermined ratio (e.g., 95%) or more, it is judged as passing, and if it is less than the predetermined value, it is judged as failing.

[0083] If it is determined that the model has passed, the machine learning unit 405 stores the adjusted neural network 60 in the trained model storage unit 406 as a trained model 61. On the other hand, if it is determined that the model has failed, a new data set 56 is further prepared and machine learning is continued, or the filter is changed and machine learning is performed again, until a trained model with a certain level of accuracy is obtained.

[0084] 〔inference〕 The learning and inference device 2 infers the characters written by the user with the digital pen 3 based on the learned model 61 as follows.

[0085] When a user writes a character on a piece of paper with digital pen 3, digital pen 3 records the force values ​​F1, F2, F3, and acceleration a in the same manner as in the data set preparation phase. x , a y , a z , and angular velocity ω x , ω y , ω z , and generates force sequence data 51, acceleration sequence data 52, and angular velocity sequence data 53, which are then transmitted to the learning and inference device 2 (see FIG. 6).

[0086] In the learning and inference device 2, when the character inference unit 407 receives the force sequence data 51, the acceleration sequence data 52, and the angular velocity sequence data 53 from the digital pen 3, it extracts the force data 510, the acceleration data 520, and the angular velocity data 530 from each of the received data, similar to the phase of preparing the data set, and generates image data 54. Then, the image data 54 is applied to the trained model 61 as input data for inference. Then, the likelihood of each lowercase letter of the alphabet is obtained by performing the same process as that of each layer of the neural network 60 shown in FIG.

[0087] Then, the character inference unit 407 infers that the lowercase letter with the highest likelihood is the character corresponding to the image data 54 .

[0088] The inference result output unit 408 outputs the inference result, i.e., the inferred characters, by displaying them on the display 27 (see FIG. 2). Alternatively, the inference result output unit 408 may output the inferred characters by transmitting text data indicating the inferred characters to another device.

[0089] [Overall process flow] FIG. 11 is a flowchart illustrating an example of the overall processing flow by the character recognition program 40.

[0090] Next, the overall process flow of the learning inference device 2 will be described with reference to a flowchart.

[0091] The learning and inference device 2 executes the process of constructing an AI and the process of recognizing characters based on the character recognition program 40 in the procedure shown in FIG.

[0092] In the phase of preparing the data set, the learning and inference device 2 performs a process of generating and storing the data set 56 (#11 in FIG. 11). The procedure of this process is as described in FIG.

[0093] When a certain number of data sets 56 are collected, the learning and inference device 2 generates and stores a trained model 61 by performing machine learning using the CNN illustrated in FIG. 10 (#12, #13).

[0094] In the inference phase, when the learning inference device 2 receives force sequence data 51, acceleration sequence data 52, and angular velocity sequence data 53 from the digital pen 3 (#14), it infers a character based on the learned model 61 (#15) and outputs the inference result (#16).

[0095] According to the present embodiment, data for identifying handwritten characters can be obtained without a dedicated paper or pad, so that a system capable of identifying handwritten characters without considering the range in which the characters can be written can be realized at a lower cost than in the past. Moreover, by using force values, acceleration, and angular velocity as such data, the accuracy of identification can be improved.

[0096] [Modifications] 12 and 13 are diagrams showing modified examples of the neural network 60. Fig. 14 is a diagram showing modified examples of the image data 54.

[0097] In this embodiment, as described above in (1-4), the first convolutional layer 602 in FIG. 10 performs convolution with the vertical stride set to 1, but it may also be set to 2. As a result, in the first convolutional layer 602, the force value pixel and the acceleration pixel are not convoluted, and features can be extracted for all pairs of two pixels among the pixels of force value F1, the pixels of force value F2, and the pixels of force value F3 (pairs of pixels of force value F1 and force value F2, pairs of pixels of force value F2 and force value F3, pairs of pixels of force value F3 and pixel of force value F1). Acceleration a x , a y , a z and angular velocity ω x , ω y , ω z The same applies to.

[0098] In this embodiment, as described above in (2-4), the second convolutional layer 604 performs convolution with the vertical stride set to 1, but it may also be set to 2 or 3. In this embodiment, as described above in (1-3), the first convolutional layer 602 performs convolution with the horizontal stride set to 1, but the convolution may be performed with the stride set to 2 or 3. The same is true for the second convolutional layer 604.

[0099] In this embodiment, a single displacement image (see FIG. 9) representing the force value, acceleration, and angular velocity is generated, and convolution and pooling are performed on this displacement image. However, convolution and pooling may be performed on each part of the force value, acceleration, and angular velocity independently so that the force value, acceleration, and angular velocity are not mixed in the convolution layer and pooling layer.

[0100] For example, as shown in Figures 12 and 13, an input layer 611, a first convolutional layer 612, a first pooling layer 613, a second convolutional layer 614, and a second pooling layer 615 are provided with independent layers for force values, acceleration, and angular velocity.

[0101] The displacement image of the image data 54 is divided into a first displacement image representing the displacement of the force value, a second displacement image representing the displacement of the acceleration, and a third displacement image representing the displacement of the angular velocity. Then, the image data 541 of the first displacement image, the image data 542 of the second displacement image, and the image data 543 of the third displacement image are input to the corresponding layers in the input layer 611.

[0102] Then, in the first convolutional layer 612, first feature maps 57F, 57G, and 57H are generated by independently performing convolution on each of the image data 541, 542, and 543. The convolution method is the same as the convolution method in the first convolutional layer 602 in FIG.

[0103] In the first pooling layer 613, the first feature maps 57F, 57G, and 57H are pooled independently to generate second feature maps 57J, 57K, and 57L. The pooling method is the same as the pooling method in the first pooling layer 603.

[0104] In the second convolutional layer 614, the second feature maps 57J, 57K, and 57L are convolved independently to generate third feature maps 57P, 57Q, and 57R. The convolution method is the same as the convolution method in the second convolutional layer 604.

[0105] In the second pooling layer 615, the third feature maps 57P, 57Q, and 57R are pooled independently to generate fourth feature maps 57T, 57U, and 57V. The pooling method is the same as that in the second pooling layer 605.

[0106] In the fully connected layer 616 and the output layer 617, the same processing as in the fully connected layer 606 and the output layer 607 is performed, respectively. However, the fourth feature maps 57T, 57U, and 57V are input to the fully connected layer 616 instead of the fourth feature map 57D.

[0107] In this embodiment, in each of the machine learning and inference phases, the trained model 61 is generated or a character is inferred based on the force values ​​detected by each of the three force sensors 35 as input (explanatory variable) data, but the trained model 61 may be generated or a character may be inferred based on the force values ​​detected by each of four or more force sensors 35.

[0108] For example, when four force sensors 35 are used, one force sensor 35 is provided in each of the three recesses 38 (see Figs. 4 and 5), and a fourth force sensor 35 is provided on the side surface of the rear of the case 31. In addition, a fourth operational amplifier 36 is provided. The force value F4 detected by the fourth force sensor 35 is transmitted to the learning and inference device 2 as force sequence data 51 together with the force values ​​F1 to F3, and is used as part of the image data 54. As a result, in this modified example, image data of a displacement image as shown in Fig. 14(A) is obtained as the image data 54.

[0109] Then, using such a displaced image, machine learning is performed on the neural network 60 shown in Fig. 10 to generate a trained model 61. After that, inference is performed by this trained model 61.

[0110] Alternatively, the trained model 61 may be generated by performing machine learning on the neural network 60 shown in Figures 12 and 13. Furthermore, in this case, if the first convolutional layer 612 performs convolution on the second and third displaced images with the vertical stride set to 2, the first displaced image may be a displaced image in which each line image is arranged three times in the order of the force values ​​F1 to F4 as shown in Figure 14(B).

[0111] Then, the first convolution layer 612 sets the vertical stride to 3 and performs convolution on the first displacement image using a 2×3 filter. The first pooling layer 613, the second convolution layer 614, and the second pooling layer 615 change the filter size as appropriate to perform convolution or pooling.

[0112] Although an example has been described in which an activation function is provided in each of the first convolutional layer 602 and the second convolutional layer 604, there is a case in which an activation function is provided in each of the first pooling layer 603 and the second pooling layer 605. In this case, the machine learning unit 405 adjusts the weight coefficient or bias value of the activation function of each of the first pooling layer 603 and the second pooling layer 605.

[0113] 3 is provided in the learning and inference device 2, but may be provided in a plurality of devices. For example, the trimming processing unit 401, the displaced image generating unit 402, the data set registration unit 403, and the data set storage unit 404 are provided in a first computer, the machine learning unit 405 is provided in a second computer, and the learned model storage unit 406, the character inference unit 407, and the inference result output unit 408 are provided in a third computer. The first computer, the second computer, and the third computer can communicate with each other via a communication line such as the Internet, a LAN (Local Area Network), or a public line.

[0114] The first computer receives force sequence data 51, acceleration sequence data 52, and angular velocity sequence data 53 from the digital pen 3, and upon obtaining the correct character from the user, generates a data set 56 and transmits it to the second computer.

[0115] When the second computer receives the dataset 56 from the first computer, it stores it in the dataset storage unit 404. Then, when a certain number of datasets 56 have been stored, the second computer performs machine learning to generate a trained model 61 and transmits it to the third computer.

[0116] When the third computer receives the trained model 61 from the second computer, it stores it in the trained model storage unit 406. Then, when the third computer receives the force sequence data 51, the acceleration sequence data 52, and the angular velocity sequence data 53 from the digital pen 3, it infers a character based on the trained model 61 and outputs the inference result. A plurality of third computers may be provided.

[0117] A plurality of digital pens 3 may be used. For example, each user may be given one digital pen 3, and data for machine learning or verification (force sequence data 51, acceleration sequence data 52, and angular velocity sequence data 53) may be collected from each digital pen 3.

[0118] In this embodiment, the digital pen 3 transmits data for machine learning or verification directly to the learning and inference device 2, but the data may also be transmitted via the user's smartphone by wirelessly connecting the digital pen 3 to the smartphone.

[0119] In this embodiment, the case of identifying lowercase letters of the alphabet has been described as an example, but the character recognition system 1 can also be used to identify other characters or symbols. For example, in addition to uppercase letters of the alphabet, it can be used to identify numbers, symbols, katakana, hiragana, kanji, etc.

[0120] In this embodiment, an example has been described in which machine learning and inference are performed based on the magnitude of force (force value) as the degree of force exerted by a finger on the digital pen 3, but machine learning and inference may also be performed based on pressure, i.e., the magnitude of force per unit area.

[0121] In the present embodiment, the digital pen 3 is held so that the thumb, index finger, and middle finger are in contact with the first recess 38A, the second recess 38B, and the third recess 38C, respectively. However, the digital pen 3 may be held so that the thumb is in contact with the second recess 38B or the third recess 38C. Therefore, matrices for converting the force value, acceleration, and angular velocity into values ​​when the digital pen 3 is held so that the thumb is in contact with the first recess 38A may be prepared for each of the cases where the thumb is in contact with the second recess 38B and the third recess 38C. Then, machine learning and inference may be performed based on the converted values.

[0122] Alternatively, the positions or shapes of first recess 38A, second recess 38B, and third recess 38C may be designed so that a predetermined finger can easily come into contact with them.

[0123] When generating a general-purpose trained model, as described above, multiple users are made to write characters. However, there are right-handed users and left-handed users. Therefore, a trained model for right-handed users and a trained model for left-handed users may be generated independently and used separately during inference. Alternatively, a dataset 56 may be generated so as to include information on handedness, and machine learning may be performed using this information as input data (explanatory variable data).

[0124] In this embodiment, a pen equipped with an ink cartridge (refill) is used as the digital pen 3, but other types of writing implements such as a pen for a whiteboard or a slider for a tablet computer may also be used.

[0125] In this embodiment, machine learning is performed based on the force value, acceleration, and angular velocity, but machine learning may be performed based only on the force value among these elements. Alternatively, machine learning may be performed based only on the force value and acceleration among these elements, or machine learning may be performed based only on the force value and angular velocity. Machine learning may be performed by combining other elements with the force value instead of acceleration or angular velocity.

[0126] In addition, the configuration of the entire or each part of the character recognition system 1, the learning and inference device 2, and the digital pen 3, the processing content, the processing order, the data configuration, the image configuration, etc. may be modified as appropriate in accordance with the spirit of the present invention. [Explanation of symbols]

[0127] 2 Learning and inference device (learning system, inference system) 3 Digital pen (writing instrument) 403 Data Set Registration Unit (Means of Acquisition) 405 Machine Learning Unit (Learning Method) 408 Inference result output unit (output means) 56 Datasets 57A, 57F, 57G, 57H First feature map 57B, 57J, 57K, 57L Second feature map 57C, 57P, 57Q, 57R Third feature map 57D, 57T, 57U, 57V Fourth feature map 60 Neural Network (CNN) 601, 611 Input layer 602, 612 First convolution layer 603, 613 First Pooling Layer 604, 614 Second convolution layer 605, 615 Second Pooling Layer 606, 616 fully connected layer 607, 617 Output layer 61 Trained Models

Claims

1. An acquisition means for acquiring, for each of a plurality of characters, a data set including a character identifier that identifies the character, an action force degree that represents a degree of force that a finger exerts on the writing implement, and an acceleration and an angular velocity that occur in the writing implement at each time while the character is being written with the writing implement; A learning means for generating a trained model by performing machine learning on a Convolutional Neural Network (CNN) using the character identifier included in the data set as training data and the action force, the acceleration, and the angular velocity included in the data set as input data; A learning system comprising:

2. the acquiring means acquires, as the data set, a first action force degree, a second action force degree, and a third action force degree which are degrees of force exerted by the finger on the writing implement in a first direction, a second direction, and a third direction, respectively, as the action force degrees, a first acceleration, a second acceleration, and a third acceleration which are accelerations on an X-axis, a Y-axis, and a Z-axis, respectively, as the accelerations, and a first angular velocity, a second angular velocity, and a third angular velocity which are angular velocities on the X-axis, a Y-axis, and a Z-axis, respectively, as the angular velocities; the learning means performs the machine learning using, as the input data, image data of a displacement image including two consecutive action force images in which a first line image, a second line image, and a third line image representing the first action force, the second action force, and the third action force, respectively, are arranged, two consecutive acceleration images in which a fourth line image, a fifth line image, and a sixth line image representing the first acceleration, the second acceleration, and the third acceleration, respectively, are arranged, and two consecutive angular velocity images in which a seventh line image, an eighth line image, and a ninth line image representing the first angular velocity, the second angular velocity, and the third angular velocity, respectively, are arranged, The learning system according to claim 1 .

3. The CNN has, in order from the top, an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a fully connected layer, and an output layer; The input layer receives the image data, in the first convolution layer, a first convolution process is performed on the image data with a plurality of first filters such that the first line image and the second line image are convolved partially by part, the second line image and the third line image are convolved partially by part, the third line image and the first line image are convolved partially by part, the fourth line image and the fifth line image are convolved partially by part, the fifth line image and the sixth line image are convolved partially by part, the sixth line image and the fourth line image are convolved partially by part, the seventh line image and the eighth line image are convolved partially by part, the eighth line image and the ninth line image are convolved partially by part, and the ninth line image and the seventh line image are convolved partially by part, thereby generating a plurality of first feature maps; In the first pooling layer, a first pooling process is performed on the first feature maps to generate a plurality of second feature maps; In the second convolutional layer, a second convolution process is performed on the second feature maps using a plurality of second filters to generate a plurality of third feature maps; In the second pooling layer, a second pooling process is performed on the third feature maps to generate fourth feature maps; In the fully connected layer, a feature variable for each of the plurality of characters is calculated based on the plurality of fourth feature maps; In the output layer, a likelihood of each of the plurality of characters is calculated based on the feature variables of each of the plurality of characters; The learning means performs the machine learning by adjusting weight coefficients or bias values ​​of an activation function used in the first convolution layer or the first pooling layer, an activation function used in the second convolution layer or the second pooling layer, and an activation function used in the fully connected layer. The learning system according to claim 2 .

4. an acquiring means for acquiring, for each of a plurality of characters, a data set including a character identifier that identifies the character, and a first action force degree, a second action force degree, and a third action force degree that are degrees of force that a finger exerts on the writing implement in a first direction, a second direction, and a third direction, respectively, at each time while the character is being written with the writing implement; a learning means for performing machine learning on a Convolutional Neural Network (CNN) using the character identifiers included in the data set as training data and image data of a displacement image in which an action force image, in which a first line image, a second line image, and a third line image each representing the first action force degree, the second action force degree, and the third action force degree, respectively, for each time point included in the data set, is consecutively displayed twice as input data, to generate a learned model; A learning system comprising:

5. The CNN has, in order from the top, an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a fully connected layer, and an output layer; The input layer receives the image data, In the first convolution layer, a first convolution process is performed on the image data with a plurality of first filters such that the first line image and the second line image are convolved with each other in parts, the second line image and the third line image are convolved with each other in parts, and the third line image and the first line image are convolved with each other in parts, thereby generating a plurality of first feature maps; In the first pooling layer, a first pooling process is performed on the first feature maps to generate a plurality of second feature maps; In the second convolutional layer, a second convolution process is performed on the second feature maps using a plurality of second filters to generate a plurality of third feature maps; In the second pooling layer, a second pooling process is performed on the third feature maps to generate fourth feature maps; In the fully connected layer, a feature variable for each of the plurality of characters is calculated based on the plurality of fourth feature maps; In the output layer, a likelihood of each of the plurality of characters is calculated based on the feature variables of each of the plurality of characters; The learning means performs the machine learning by adjusting weight coefficients or bias values ​​of an activation function used in the first convolution layer or the first pooling layer, an activation function used in the second convolution layer or the second pooling layer, and an activation function used in the fully connected layer. The learning system according to claim 4.

6. an inference means for inferring a character by applying an action force representing a degree of force exerted by a finger on the writing instrument and an acceleration and an angular velocity generated in the writing instrument at each time while the character is being written with the writing instrument to a trained model generated by the learning system according to any one of claims 1 to 3; 1. An inference system comprising:

7. an inference means for inferring a character by applying a first action force degree, a second action force degree, and a third action force degree, which are degrees of force exerted by a finger on the writing tool in a first direction, a second direction, and a third direction, respectively, at each time while the character is being written with the writing tool, to a trained model generated by the learning system according to claim 4 or claim 5; 1. An inference system comprising:

8. a computer is caused to execute a process of acquiring, for each of a plurality of characters, a data set including a character identifier that identifies the character, and an action force degree that represents a degree of force that a finger exerts on the writing instrument, and an acceleration and an angular velocity that occur in the writing instrument, at each time while the character is being written with the writing instrument; causing the computer to execute a process of generating a trained model by performing machine learning on a Convolutional Neural Network (CNN) using the character identifier included in the data set as training data and the action force, the acceleration, and the angular velocity included in the data set as input data; A learning method comprising:

9. causing a computer to execute a process of acquiring, for each of a plurality of characters, a data set including a character identifier that identifies the character, and a first action force degree, a second action force degree, and a third action force degree that are degrees of force that a finger exerts on the writing instrument in a first direction, a second direction, and a third direction, respectively, at each time while the character is being written with the writing instrument; The computer is caused to execute a process of generating a trained model by performing machine learning on a Convolutional Neural Network (CNN) using the character identifier included in the data set as training data and using image data of a displacement image in which an action force image in which a first line image, a second line image, and a third line image each representing the first action force degree, the second action force degree, and the third action force degree, respectively, for each time included in the data set, is consecutive twice as input data. A learning method comprising:

10. A computer program for use in a machine learning computer, comprising: The computer includes: A process of acquiring a data set including, for each of a plurality of characters, a character identifier that identifies the character, and an action force level that represents a degree of force that a finger exerts on the writing instrument, as well as an acceleration and an angular velocity that occur on the writing instrument, at each time while the character is being written with the writing instrument; A process of generating a trained model by performing machine learning on a Convolutional Neural Network (CNN) using the character identifier included in the data set as training data and the action force, the acceleration, and the angular velocity included in the data set as input data; A computer program characterized by:

11. A computer program for use in a machine learning computer, comprising: The computer includes: obtaining, for each of a plurality of characters, a data set including a character identifier that identifies the character, and a first action force degree, a second action force degree, and a third action force degree that are degrees of force that a finger exerts on the writing instrument in a first direction, a second direction, and a third direction, respectively, at each time while the character is being written with the writing instrument; a process of generating a trained model by performing machine learning on a Convolutional Neural Network (CNN) using the character identifier included in the data set as training data and image data of a displacement image in which an action force image in which a first line image, a second line image, and a third line image each representing the first action force degree, the second action force degree, and the third action force degree, respectively, for each time included in the data set, are consecutively displayed twice as input data; A computer program characterized by:

12. A trained model generated by the training system according to claim 5, The neural network includes an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a fully connected layer, and an output layer, in that order from the top. When image data of a displacement image in which a first line image, a second line image, and a third line image are arranged to represent a first action force degree, a second action force degree, and a third action force degree, which are degrees of force that a finger exerts on the writing instrument in a first direction, a second direction, and a third direction, respectively, at each time while a character is being written with the writing instrument, are input to the input layer, a first convolution process is performed on the image data with a plurality of first filters such that the first line image and the second line image are convolved partially, the second line image and the third line image are convolved partially, and the third line image and the first line image are convolved partially, a plurality of first feature maps are generated in the first convolution layer; A first pooling process is performed on the first feature maps to generate a plurality of second feature maps in the first pooling layer; a plurality of third feature maps are generated in the second convolutional layer by performing a second convolution process with a plurality of second filters on the plurality of second feature maps; A second pooling process is performed on the third feature maps to generate a fourth feature map in the second pooling layer; Calculating feature variables for each of the plurality of characters in the fully connected layer based on the plurality of fourth feature maps; a likelihood of each of the plurality of characters is calculated in the output layer based on the feature variables of each of the plurality of characters; A trained model that causes a computer to function in a way that

Images