Character recognition device, program, and method

Abstract

Provided is a character recognition device, program, and method capable of improving the recognition accuracy of characters with respect to stroke data in a vector format. A character recognition device (16) acquires stroke data (48) representing a set of indication positions and pen pressure values sequentially detected by a user's writing operation, generates a feature amount set related to a stroke group (52) written by the writing operation using the acquired stroke data (48), and recognizes a single character or a character string represented by the stroke group (52) using the generated feature amount set. In the feature amount set, a tip feature amount related to the tip shape of each stroke is included.

Description

Technical Field

[0001] This invention relates to a character recognition device, program, and method. Background Technology

[0002] Previously, character recognition techniques were known to identify characters by processing data representing individual characters or strings of characters. When the data being processed is in raster format, i.e., an image, convolutional neural networks are often used to recognize characters (see, for example, Patent Documents 1 and 2).

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-027598

[0006] Patent Document 2: Japanese Patent Application Publication No. 2020-119559 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] However, convolutional neural networks may not be effective when processing data in vector format, such as stroke data (or digital ink). In other words, there are technological strategies aimed at improving the accuracy of text recognition.

[0009] The present invention was made in view of the following problems, and its object is to provide a character recognition device, program and method that can improve the recognition accuracy of characters relative to stroke data in vector format.

[0010] Methods for solving problems

[0011] The character recognition device of the first invention comprises: a data acquisition unit that acquires stroke data representing a set of indication positions and pen pressure values ​​that are sequentially detected by a user's writing operation; a feature generation unit that uses the stroke data acquired by the data acquisition unit to generate a feature set related to a group of strokes written by the writing operation; and a character recognition unit that uses the feature set generated by the feature generation unit to recognize a single character or character string represented by the group of strokes, wherein the feature set includes an end feature related to the end shape of each stroke.

[0012] The second character recognition program in this invention causes a computer to perform: an acquisition step, acquiring stroke data representing a set of indication positions and pen pressure values ​​that are sequentially detected through a user's writing operation; a generation step, using the acquired stroke data to generate a set of feature quantities associated with the group of strokes written through the writing operation; and a recognition step, using the generated set of feature quantities to recognize a single character or character string represented by the group of strokes, wherein the set of feature quantities includes end feature quantities associated with the end shape of each stroke.

[0013] In the character recognition method of the third invention, the computer performs: an acquisition step, acquiring stroke data representing a set of indication positions and pen pressure values ​​that are sequentially detected through the user's writing operation; a generation step, using the acquired stroke data to generate a set of feature quantities associated with the group of strokes written through the writing operation; and a recognition step, using the generated set of feature quantities to recognize a single character or character string represented by the group of strokes, wherein the set of feature quantities includes end feature quantities associated with the end shape of each stroke.

[0014] Invention Effects

[0015] According to the present invention, the recognition accuracy of characters relative to stroke data in vector format can be improved. Attached Figure Description

[0016] Figure 1 This is an overall structural diagram of an input system incorporating a character recognition device according to one embodiment of the present invention.

[0017] Figure 2 It is shown Figure 1 The flowchart shows the operation of the text recognition device.

[0018] Figure 3 It is shown Figure 1 A diagram illustrating an example of the data structure of ink data.

[0019] Figure 4 It is shown by Figure 3 An example of a graph showing the content of stroke data visualization.

[0020] Figure 5 This is a diagram illustrating an example of how the feature set for each bounding box is generated.

[0021] Figure 6 This is an example of how pen pressure changes when writing strokes with different end shapes.

[0022] Figure 7 This is a diagram illustrating an example of the network structure possessed by the identifier.

[0023] Figure 8This is an example of the text string recognition results.

[0024] Figure 9 This is a diagram illustrating one example of the network structure possessed by the recognizer in another example.

[0025] Figure 10 This is a diagram illustrating an example of the data structure of textual information. Detailed Implementation

[0026] The following is a reference to the appendix. Figure 1 The embodiments of the present invention will be described below. To facilitate understanding, the same reference numerals will be used as much as possible to label the same components and steps in the various drawings, and sometimes repeated descriptions will be omitted.

[0027] [Overall structure of input system 10]

[0028] Figure 1 This is an overall structural diagram of an input system 10 incorporating a character recognition device 16 according to one embodiment of the present invention. The input system 10 is configured to provide a "digital ink service" that processes a user's handwritten text as text data. Specifically, the input system 10 comprises one or more user devices 12, one or more electronic pens 14, and a character recognition device 16.

[0029] User device 12 is a computer owned by a user utilizing digital ink service and has the function of detecting the indicated position of electronic pen 14. User device 12 may be, for example, a tablet, smartphone, personal computer, etc.

[0030] The electronic pen 14 is a pen-shaped indicating device configured to communicate unidirectionally or bidirectionally with the user device 12. The electronic pen 14 is, for example, an active electrostatic coupling (AES) or electromagnetic induction (EMR) stylus. By holding the electronic pen 14 and moving its tip against the touch surface of the user device 12, the user can write pictures or text onto the user device 12.

[0031] The character recognition device 16 is a computer that performs overall control related to character recognition, and can be either cloud-based or locally deployed. Here, the character recognition device 16 is illustrated as a single computer, but it can also be replaced by a cluster of computers forming a distributed system. Specifically, the character recognition device 16 is configured to include a communication unit 20, a control unit 22, and a storage unit 24.

[0032] The communication unit 20 is an interface for receiving and transmitting signals to external devices. Thus, the character recognition device 16 obtains ink data 42 from the user device 12 and supplies its own generated text data 44 to the user device 12.

[0033] The control unit 22 is composed of a processor including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The control unit 22 functions as the data acquisition unit 26, the recognition processing unit 28, and the output processing unit 30, which will be described later, by reading and executing the programs and data stored in the storage unit 24.

[0034] The storage unit 24 stores the programs and data required by the control unit 22 to control each component. The storage unit 24 is composed of a non-transitory storage medium that can be read by a computer. Here, the storage medium that can be read by a computer is a removable medium such as an optical disk, ROM, CD-ROM, flash memory, etc., a hard disk drive (HDD), a solid state drive (SSD), etc., built into the computer system.

[0035] It should be noted that the storage unit 24 contains a database related to digital ink (hereinafter referred to as digital ink DB40), and stores ink data 42, text data 44 and parameter set 46.

[0036] [Action of the character recognition device 16]

[0037] The input system 10 in this embodiment is configured as described above. Next, regarding the operation of the character recognition device 16, which constitutes part of the input system 10, while referring to... Figure 2 Flowchart and Figures 3-8 While explaining.

[0038] <Flowchart Explanation>

[0039] exist Figure 2 In step SP10, the data acquisition unit 26 acquires data (hereinafter referred to as "stroke data 48") representing a set of indicated positions and pen pressure values ​​that are sequentially detected through the user's writing operation. Specifically, the data acquisition unit 26 extracts the stroke data 48 from the ink data 42 supplied by the user device 12 to obtain the stroke data 48.

[0040] Figure 3 It is shown Figure 1This diagram illustrates an example of the data structure of ink data 42. In this diagram, an example of InkML (Ink Markup Language) described in XML (eXtensible Markup Language) format is given for illustration. It should be noted that the data format of ink data 42, i.e., the so-called "ink description language," is not limited to InkML; for example, it could also be WILL (Wacom Ink Layer Language) or ISF (Ink Serialized Format).

[0041] In this example, ink data 42 comprises stroke data 48 describing the writing state of at least one stroke. In this example, the writing states of a total of 20 strokes are described from top to bottom in the order of stroke writing. Additionally, one stroke is defined by... <trace>A description of a plurality of point data arranged in sequence within the tag. For the convenience of illustration, only the point data representing the start point and end point of a stroke are recorded, and the description of point data representing a plurality of passing points is omitted.

[0042] Each piece of point data is composed of a combination of the indicated position (X coordinate value, Y coordinate value) of the electronic pen 14 and a pen pressure value (for example, an 8-bit level value), and is separated by a delimiter such as a comma. The point data may include, in addition to the aforementioned indicated position and pen pressure value, the writing order of strokes, the tilt angle of the electronic pen 14, and the like.

[0043] Figure 4 is a diagram illustrating an example of the visualized content of Figure 3 stroke data 48. On the display panel of the user device 12, a two-dimensional coordinate system for determining a touch position (hereinafter referred to as a sensor coordinate system: X-Y) is defined. A stroke group 52 composed of 20 strokes is arranged in the entire area 50 of the sensor coordinate system. The stroke group 52 represents a character string composed of Chinese characters and Hiragana.

[0044] The Chinese character at the beginning on the left side of the drawing includes an end 54 representing the trace of a "stop stroke (tome)", an end 56 representing the trace of a "flick stroke (harai)", and an end 58 representing the trace of a "jump stroke (hane)". The "stop stroke" is an operation of suddenly stopping the moving pen tip and then lifting the pen tip off the touch surface. The "flick stroke" is an operation of sliding the moving pen tip while lifting it off the touch surface. The "jump stroke" is an operation of suddenly stopping the moving pen tip, then changing the direction of the pen tip while performing the aforementioned "flick stroke".

[0045] In Figure 2 step SP12, the recognition processing unit 28 specifies a bounding box 60 as a recognition target from the entire area 50 defined by the stroke data 48 acquired in step SP10. The shape of the bounding box 60 is uniquely determined by four variables: the X coordinate value / Y coordinate value of a reference position (for example, the vertex closest to the origin O), the length of the side in the X direction, and the length of the side in the Y direction.

[0046] In step SP14, the recognition processing unit 28 (more specifically, the feature quantity generation unit 32) generates a feature quantity set within the bounding box 60 specified in step SP12. Here, the feature quantity set includes [1] feature quantities related to the trajectory of each stroke (hereinafter referred to as trajectory feature quantities) and [2] feature quantities related to the end shape of each stroke (hereinafter referred to as end feature quantities).

[0047] Figure 5 This diagram illustrates an example of a method for generating the feature set for each bounding box. The feature generation unit 32 defines a two-dimensional coordinate system (hereinafter referred to as the normalized coordinate system; X'-Y') corresponding to the touch surface of the user device 12. The origin O' of this normalized coordinate system corresponds to the vertex of the bounding box 60 located closest to the origin O of the sensor coordinate system. The X' axis of the normalized coordinate system is parallel to the X-axis of the sensor coordinate system, and the Y' axis of the normalized coordinate system is parallel to the Y-axis of the sensor coordinate system. Furthermore, the scales of the X' and Y' axes are normalized so that the coordinates of the four vertices constituting the bounding box 60 are (0, 0), (1, 0), (0, 1), and (1, 1).

[0048] Feature quantity generation unit 32 refers to stroke data 48 ( Figure 3 The feature generation unit 32 obtains the coordinates (X, Y) of the starting point Ps and the ending point Pe of the stroke in the sensor coordinate system. Then, the feature generation unit 32 derives the coordinates (X', Y') of the starting point Ps and the ending point Pe of the stroke in the normalized coordinate system by performing a linear transformation of the coordinate system. It should be noted that when the starting point Ps or the ending point Pe of the stroke is outside the bounding box 60, the feature generation unit 32 can treat the passing points on the boundary line as imaginary starting points Ps or ending points Pe to generate trajectory features.

[0049] Figure 6 This is a graph illustrating an example of pen pressure variation when writing strokes with different end shapes. The horizontal axis of the graph represents time (t), and the vertical axis represents pen pressure. Time t = Ts corresponds to the execution time of the pen-pressing operation, where the electronic pen 14 is brought against the touch surface. On the other hand, time t = Te corresponds to the execution time of the pen-lifting operation, where the electronic pen 14 is lifted off the touch surface.

[0050] When the end shape is a "pause," the pen pressure tends to remain roughly constant while decreasing sharply just before t=Te. When the end shape is a "lift," the pen pressure tends to remain roughly constant while decreasing slowly from near t=Te. When the end shape is a "flick," the pen pressure tends to remain roughly constant while temporarily increasing near t=Te and then decreasing at a slope intermediate between the "pause" and "lift."

[0051] The feature generation unit 32 uses various methods, including pattern matching, to estimate the end shape for each stroke. If the estimated result does not conform to any of the stroke's characteristics (pause / lift / flick), an attribute of "unknown" or "does not conform" can be assigned. It should be noted that the end feature is not limited to recognition values ​​such as pause / lift / flick; it can also be various values ​​representing shape characteristics (e.g., the slope of the decrease in pen pressure, the starting position of the decrease, the time series of pen pressure values, etc.).

[0052] exist Figure 2 In step SP16, the recognition processing unit 28 (specifically, the character recognition unit 34) uses the feature set generated in step SP14 to perform character recognition processing. Before this recognition, the character recognition unit 34 selects a suitable character type from multiple recognizers 70 that have undergone machine learning. Examples of character types include kanji, hiragana, katakana, Arabic numerals, Latin letters, and symbols.

[0053] Figure 7 This diagram illustrates an example of the network structure of the recognizer 70. The recognizer 70 is, for example, constructed from a hierarchical neural network consisting of an input layer 72, an intermediate layer 74, and an output layer 76. The operational rules of the recognizer 70 are derived from a set of learned parameters, namely parameter set 46. Figure 1 The value of ) is determined. Parameter set 46 may include, for example, coefficients describing the activation function of the operational unit, weighting coefficients equivalent to the strength of synaptic binding, the number of operational units constituting each layer, the number of intermediate layers 74, etc. Parameter set 46, with its values ​​determined by the end of machine learning, is stored in storage unit 24. Figure 1 ( ), read out as needed.

[0054] Input layer 72 is the layer that inputs trajectory features and end features, and in this example, it consists of 5n operational units. Here, the natural number n is equivalent to the number of strokes. The feature set is an input vector consisting of 5n components arranged in the order of stroke ID: [1] the X' coordinate of the starting point Ps, [2] the Y' coordinate of the starting point Ps, [3] the X' coordinate of the ending point Pe, [4] the Y' coordinate of the ending point Pe, and [5] the identification value of the end shape.

[0055] The intermediate layer 74 consists of one or more layers and has the function of reducing the dimension of the feature set. That is, the number of operation units constituting the intermediate layer 74 is preferably less than 5n.

[0056] Output layer 76 is the layer that outputs the tag group of the characters. In this example, it consists of operational units representing the number of hiragana characters (e.g., 46 characters excluding voiced / semi-voiced dots). When the activation function of the operational unit is the Softmax function, this tag group is an output vector consisting of 46 components representing the probability of each character.

[0057] Alternatively, if the character type is "Kanji", a recognizer 70 suitable for recognizing Kanji can be selected. Or, if the character type cannot be determined, a recognizer 70 suitable for recognizing multiple characters (e.g., Hiragana and Katakana) can be selected.

[0058] Furthermore, the character recognition unit 34 inputs the feature set corresponding to the bounding box 60 from the input layer 72, and outputs the label group corresponding to the bounding box 60 to the output layer 76 through the operation of the intermediate layer 74. For example, when the label value represents probability, the character with the highest probability is recognized as the "character candidate".

[0059] exist Figure 2 In step SP18, the identification processing unit 28 confirms whether all bounding boxes 60 were specified in step SP12. If unspecified bounding boxes 60 remain (step SP18: No), the process returns to step SP12.

[0060] In subsequent recognition processes, the recognition processing unit 28 generates feature sets (SP14) and recognizes characters (SP16) sequentially while varying the combination of the four variables that determine the bounding box 60. Furthermore, if all bounding boxes 60 are specified for the entire region 50 (step SP18: Yes), it proceeds to the next step SP20.

[0061] In step SP20, the recognition processing unit 28 gathers the recognition results from step SP16 and generates text data 44 represented by stroke group 52. Specifically, the recognition processing unit 28 extracts four characters by selecting the combination of the bounding box 60 with the highest probability and the label.

[0062] like Figure 8 As shown, bounding boxes B1, B2, B3, and B4 are defined to enclose the four characters respectively. Furthermore, by connecting the four characters sequentially from left to right, text data 44 representing the Japanese word for "writing" is obtained. Ink data 42 and text data 44 are represented by digital ink DB.

[0063] exist Figure 2 In step SP22, the output processing unit 30 outputs the text data 44 generated in step SP20 according to the request from the user device 12. Thus, a series of actions of the character recognition device 16 ( Figure 2 (The flowchart ends here.)

[0064] <Other identification examples>

[0065] Figure 9 This diagram illustrates one example of the network structure of the recognizer 80 in another example. The recognizer 80 is composed of a hierarchical neural network, which is determined by the number of computational units and... Figure 7 It consists of an input layer 72, different input layers 82, an intermediate layer 74, and an output layer 76.

[0066] Input layer 82 is a layer that only inputs trajectory features, and in this example, it consists of 4n (n is the number of strokes) computational units. This feature set is an input vector consisting of 4n components arranged in the order of stroke ID: [1] X' coordinate of starting point Ps, [2] Y' coordinate of starting point Ps, [3] X' coordinate of ending point Pe, and [4] Y' coordinate of ending point Pe.

[0067] Figure 10 This is a diagram illustrating an example of the data structure of text information 84. Text information 84 is tabular data showing the correspondence between [1] character recognition information, [2] stroke order, [3] type of end shape and [4] importance of end shape.

[0068] The following is about based on Figure 9 and Figure 10 The method for recognizing characters with a specific structure, according to Figure 2 The flowchart will be used to illustrate this. It should be noted that steps other than SP16 and SP20 are as described above, so further explanation is omitted.

[0069] exist Figure 2 In step SP16, the recognition processing unit 28 (more specifically, the character recognition unit 34) uses the feature set generated in step SP14 to perform character recognition processing. Prior to this recognition, the character recognition unit 34 uses multiple recognizers 80 that have undergone machine learning... Figure 9 Select one or more recognizers 80.

[0070] Furthermore, the character recognition unit 34 inputs the feature set corresponding to the bounding box 60 (here, only the trajectory feature set) from the input layer 92, and outputs the label group corresponding to the bounding box 60 to the output layer 76 through the operation of the intermediate layer 74. For example, when the label value represents a probability, characters with a high probability (e.g., the highest position, up to the third highest position) are recognized as "character candidates".

[0071] exist Figure 2 In step SP20, the recognition processing unit 28 gathers the recognition results from step SP16 and generates text data 44 represented by stroke group 52. Specifically, the recognition processing unit 28 extracts four characters by selecting the label with the highest consistency in the end shape from among a plurality of selected character candidates. Specifically, the consistency between the end shape estimated in step SP14 and the end shape possessed by the character information 84 is evaluated for each stroke. This consistency is evaluated, for example, by summing the scores weighted according to the importance of the character information 84.

[0072] In this way, the recognition processing unit 28 can also use the recognizer 80, which takes a set of features excluding terminal features as input, to select multiple character candidates and use the terminal features to determine the characters.

[0073] [Effect of the character recognition device 16]

[0074] As described above, the character recognition device 16 includes: a data acquisition unit 26 that acquires stroke data 48 representing a set of indicated positions and pen pressure values ​​sequentially detected through a user's writing operation; a feature generation unit 32 that uses the stroke data 48 acquired by the data acquisition unit 26 to generate a feature set related to the stroke group 52 written through the writing operation; and a character recognition unit 34 that uses the feature set generated by the feature generation unit 32 to recognize a single character or character string represented by the stroke group 52. The feature set includes end features related to the end shape of each stroke.

[0075] Furthermore, in this character recognition method and program, the character recognition device 16, acting as a computer, performs the following steps: an acquisition step (SP10), acquiring stroke data 48 representing a set of indication positions and pen pressure values ​​sequentially detected through a user's writing operation; a generation step (SP14), using the acquired stroke data 48 to generate a feature set associated with the stroke group 52 written through the writing operation; and a recognition step (SP16), using the generated feature set to recognize a single character or character string represented by the stroke group 52. The feature set includes end features associated with the end shape of each stroke.

[0076] Thus, since the feature set includes end features related to the end shape of each stroke, it is possible to accurately capture the unique shape features of handwritten characters, thereby improving the recognition accuracy of characters relative to the vector format stroke data 48.

[0077] Alternatively, the character recognition unit 34 may be configured to include a recognizer 70 that takes a feature set including terminal features as input and outputs the probability of each character. Or, the character recognition unit 34 may also be configured to include a recognizer 80 that takes a feature set excluding terminal features as input and outputs the probability of each character, and when multiple character candidates are obtained by the recognizer 80, the terminal features are used to determine the character.

[0078] It should be noted that the end feature value can be a value that identifies a pause, a lift, or a flick of the pen. Additionally, the type of script can include at least one of the following: kanji, hiragana, katakana, and Arabic numerals.

[0079] [Variation Example]

[0080] It should be noted that the present invention is not limited to the above-described embodiments, and can be freely modified within the scope of the spirit of the invention. Alternatively, the various structures can be combined arbitrarily without causing technical inconsistencies.

[0081] In the above embodiments, hierarchical neural networks are used to construct recognizers 70 and 80 (see reference). Figure 7 and Figure 9 However, machine learning methods are not limited to this. For example, a variety of methods can be used, including support vector machines, decision trees (random forests as an example), and boosting methods (gradient boosting as an example).

[0082] In the above embodiment, the feature quantity generation unit 32 ( Figure 1 The example described uses only the start and end points of the strokes to generate trajectory feature quantities, but the method of generating trajectory feature quantities is not limited to this. For example, the feature quantity generation unit 32 may also use at least one transit point (e.g., the midpoint between the start and end points) in addition to the start and end points to generate trajectory feature quantities.

[0083] [Explanation of reference numerals in the attached figures]

[0084] 10…Input system, 12…User device, 14…Electronic pen, 16…Character recognition device, 26…Data acquisition unit, 28…Recognition processing unit, 30…Output processing unit, 32…Feature quantity generation unit, 34…Character recognition unit, 48…Stroke data, 52…Stroke group, 70, 80…Recognizer.< / trace>

Claims

1. A character recognition device, comprising: The data acquisition unit acquires stroke data representing a set of indicator positions and pen pressure values ​​that are sequentially detected through the user's writing operations; The feature generation unit uses the stroke data acquired by the data acquisition unit to generate a set of features related to the stroke group written through the writing operation; and The character recognition unit uses the feature set generated by the feature generation unit to recognize individual characters or character strings represented by the stroke group. The feature set includes end features related to the end shape of each stroke. The character recognition unit includes a recognizer that takes a set of features excluding the terminal features as input and outputs the probability of each character. When the character recognition unit obtains multiple character candidates from the recognizer, it uses the terminal feature quantity to determine the character.

2. The character recognition device according to claim 1, The end feature value is the value used to identify whether a stroke is paused, lifted, or flicked.

3. The character recognition device according to claim 1, The character recognition unit recognizes at least one character among Chinese characters, hiragana, katakana, and Arabic numerals.

4. A computer-readable storage medium comprising a character recognition program that causes a computer to perform: The acquisition step involves acquiring stroke data representing a set of indicator positions and pen pressure values ​​that are sequentially detected through the user's writing operations. The generation step involves using the acquired stroke data to generate a set of features associated with the stroke group written through the writing operation; and The identification step involves using the generated feature set to identify individual characters or character strings represented by the stroke groups. The feature set includes end features related to the end shape of each stroke. In the recognition step, the feature set excluding the terminal feature quantity is used as input, and the probability of each character is output. The recognition step uses the terminal feature quantity to determine the character when multiple character candidates are obtained.

5. A character recognition method, executed by a computer: The acquisition step involves acquiring stroke data representing a set of indicator positions and pen pressure values ​​that are sequentially detected through the user's writing operations. The generation step involves using the acquired stroke data to generate a set of features associated with the stroke group written through the writing operation; and The identification step involves using the generated feature set to identify individual characters or character strings represented by the stroke groups. The feature set includes end features related to the end shape of each stroke. In the recognition step, the feature set excluding the terminal feature quantity is used as input, and the probability of each character is output. The recognition step uses the terminal feature quantity to determine the character when multiple character candidates are obtained.

Images