Information system based on automatic line feed in handwritten document

Abstract

To provide an information system based on automatic line feed in a handwritten document.SOLUTION: When a user actually writes, a system calculates a standard height and a standard width of a handwritten original. The system detects and deletes an abnormal stroke to prevent a situation where the abnormal stroke affects a division and an identification result obtained thereafter. The system then roughly divides the handwritten original into a plurality of lines using a pre-division unit. The system sets an isolated point and an indirect stroke as one line, then re-divides a line that is too large, and performs line alignment and spatial merging on a result obtained when the division is completed. When a word interval in the middle of the line is too large, the system divides the line into a plurality of lines again and improves accuracy of the division. Finally, the system determines, on the basis of whether the whole is a word, whether to add a second division scheme. The present invention takes into account various situations that occur when an ordinary user writes by hand, such as writing style issues of the character size, the isolated point, and the indirect stroke, and a program of the present invention appropriately handles these writing style issues, thereby improving efficiency of line feed.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to line-wrapping processing of electronic handwriting, and more particularly to an information system based on automatic line-wrapping of handwritten documents. [Background technology]

[0002] In the past, when a user wrote characters on an electronic writing panel, the characters written by the user were converted into computer text by a program on the computer side, and the characters were then arranged in order up to the boundary where the program could write, and then a line break was automatically performed.

[0003] Handwritten character segmentation technology is a technology that mainly separates letters and numbers in handwritten text. It allows computers to identify and interpret handwritten text, and supports the application of machine learning and natural language processing. Its main principle is to use various data processing techniques to analyze the characteristics of handwritten characters, such as their shape, size, spelling, continuity, etc., and to identify each character based on these characteristics.

[0004] Regarding handwritten character segmentation, the prior art mainly includes deep learning methods, traditional computer vision methods, and unsupervised learning methods. Based on the deep learning method, a customized convolutional neural network (CNN) with end-to-end capabilities is used to realize the character segmentation task. Summary of the Invention [Problem to be solved by the invention]

[0005] However, traditional computer vision methods usually use a full image processing pipeline to extract multiple features of handwritten text, and then use image segmentation to classify and identify these features. Unsupervised learning methods are methods that achieve the character segmentation task with only a single image, without requiring any tag information or a set of pre-annotated images to learn from.

[0006] The present invention mainly solves the problem that when processing handwritten input on an electronic writing panel, the handwritten input characters are reasonably broken into lines, mainly due to people's handwritten input habits and external factors when writing a large amount of text, the actual stroke order is not the same as the stroke order of the characters in the handwritten manuscript, or the strokes themselves are irregular, making it difficult to break the lines.

[0007] Therefore, the present inventors believed that the above-mentioned drawbacks could be improved, and as a result of extensive research, they came to propose the present invention, which effectively improves the above-mentioned problems through rational design.

[0008] The present invention has been made in view of the above circumstances, and has as its object to provide an information system based on automatic line feed processing for handwritten documents. [Means for solving the problem]

[0009] In order to solve the above problems, the present invention employs the following means. According to one aspect of the present invention, an information system based on automatic line breaks in handwritten documents performs line breaks on a handwritten manuscript generated by a writer writing on an electronic writing panel, the system records each stroke (i.e., each coordinate point of the stroke) and the time when each coordinate point was generated, and hereinafter, the direction of the line when writing is referred to as the row direction (x direction), and the direction of skipping lines is referred to as the column direction (y direction), and the system: A processor and a memory, the processor being used to perform operations required by the system, and the memory being used to store data or operation programs of the computer program of the system, related operation results and related data; a line height unit used to calculate a standard height and a standard width of the handwritten document; an abnormality deletion unit connected to the line height unit and receiving the standard height and standard width calculated by the line height unit, the abnormality deletion unit deletes abnormal strokes in advance from the handwritten document, and the abnormal stroke is defined as a stroke whose height is higher than the standard height and a predetermined multiple used for abnormal deletion, or the abnormal stroke is defined as a stroke whose width is higher than the predetermined multiple used for abnormal deletion of the standard width; a pre-division unit connected to the anomaly deletion unit and receiving the processed handwritten document from the anomaly deletion unit, the pre-division unit first divides the handwritten document into a number of lines, the pre-division method is according to the line direction of writing, and detects the leftmost end and the rightmost end of the first touched line, that is, from left to right, the first touched point of the line is the leftmost end, and the last touched point of the line is the rightmost end, and then the pre-division is performed with the set multiple of the standard height pre-division as the line height, to obtain one line, and the remaining handwritten document is obtained in the same manner by the pre-division unit; a special line unit connected to the pre-division unit, which first sets each isolated point and each inverted stroke of the handwritten manuscript after the pre-division to a single line, and these pre-set lines are merged into other lines in the following steps; an oversized line dividing unit connected to the special line unit, the oversized line dividing unit is used to re-divide the oversized line divided by the previous dividing unit, and the definition of the oversized line is that if the height of the line divided by the previous dividing unit is higher than a set multiple used for the oversized line of the standard height, the oversized line dividing unit determines that the line is oversized; a space merging unit connected to the oversized line dividing unit, for receiving the handwritten document processed by the oversized line dividing unit, for determining whether a new line is established after dividing the oversized line, and for abandoning the division and accepting the original oversized line as one line if the new line is not established, and for merging the isolated point and the line formed by the oversized writing into another line that is close to the isolated point and the oversized writing; a multi-line unit connected to the spatial merging unit and receiving the handwritten manuscript processed by the spatial merging unit, the multi-line unit calculates the distance between strokes in the lines that have already been divided, the distance being the difference between the rightmost stroke of the previous stroke and the leftmost stroke of the next stroke, and if the distance is greater than a set width used for the multiple lines, the multi-line unit uses the two strokes as a boundary and incorporates the previous and next strokes into two different lines, respectively; A line feed is performed on the handwritten manuscript input based on the above method. Effect of the Invention

[0010] As described above, the present invention has the following advantages. The present invention takes into account various situations that occur when ordinary users write by hand, such as problems with writing style, such as character size, isolated points, and inverted strokes, and the program of the present invention properly handles these problems and improves the efficiency of line breaks. Also, the present invention takes into account problems that occur when first breaking a line, such as splitting multiple lines into one line, and corrects these problems to improve the accuracy of line breaks. The present invention realizes line splitting of handwritten multiple lines of characters and obtains favorable test results, which is not achievable with the prior art.

[0011] Other objects, configurations and effects of the present invention will become apparent from the following detailed description of the preferred embodiments of the present invention. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an information system based on automatic line breaks in handwritten documents according to one embodiment of the present invention; [Diagram 2] 1 is a process flowchart showing an information system based on automatic line breaks in handwritten documents according to an embodiment of the present invention. [Figure 3A] FIG. 2 is a diagram showing the distance between each point in a stroke and the Y axis according to one embodiment of the present invention. [Figure 3B] FIG. 4 is a distribution diagram of the relationship between all stroke points and Dy in a handwritten document according to an embodiment of the present invention. [Figure 4] 1 is a schematic diagram showing xy directions of a handwritten document according to an embodiment of the present invention; [Diagram 5] 1 is a schematic diagram showing an information system based on automatic line breaks in handwritten documents according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments of the present invention will be described in detail. However, the present invention is not limited to these, and various modifications are possible within the scope of the description. The technical scope of the present invention also includes embodiments obtained by appropriately combining the technical means disclosed in the different embodiments.

[0014] First, an information system 1 based on automatic line feed for handwritten documents according to the present invention will be described in detail with reference to the accompanying drawings, Figs. 1 to 5 as appropriate.

[0015] In this specification, the handwritten manuscript is a digital handwritten manuscript, that is, a handwritten manuscript generated by a writer writing on the electronic writing panel 8 (see step 110). At this time, the electronic writing panel 8 records each stroke (i.e., the coordinate points of each stroke) and the time when each coordinate point was generated, so that the time sequence when each stroke and its coordinate points were input can be known. Hereinafter, in the specification, the row direction of the handwritten manuscript is arranged on the x-axis of the coordinate axis, and the column direction is arranged on the y-axis of the coordinate axis (see FIG. 4).

[0016] The information system 1 of the present invention that solves the above problems mainly comprises the following components, each of which will be described below.

[0017] <Host 100> The host 100 includes a processor 2 and a memory 6. The processor 2 is used to execute the operations required by the system 1 (see FIG. 1). The memory 6 is used to store the data of the computer program or operation program of the system 1, related operation results and related data. All the operation results and related data are stored in the memory 6.

[0018] <line height unit 10> The strokes of all the handwritten manuscripts are arranged based on the height of the strokes, and then the height of the strokes of a certain proportional value is taken as the standard height. Similarly, the strokes of all the handwritten manuscripts are arranged based on the width of the strokes, and then the width of the strokes of a certain proportional value is taken as the standard width. For example, a handwritten manuscript has a total of 100 strokes, and then the strokes are arranged from shortest to tallest based on the height of the strokes, and the 89th tallest stroke is taken as the standard height. Similarly, the 100 strokes of the handwritten manuscript are arranged from shortest to longest based on the width of the strokes, and the 89th widest stroke is taken as the standard width (see step 115).

[0019] <Anomalous Deletion Unit 20> The abnormality deletion unit 20 is connected to the line height unit 10 and receives the standard height and standard width calculated by the line height unit 10. The abnormality deletion unit 20 deletes abnormal strokes in advance from the handwritten document (see step 120). The definition of abnormal strokes is that a stroke whose height is higher than a certain multiple of the standard height or whose width is higher than a certain multiple of the standard width is regarded as an abnormal stroke. For example, if the height of a stroke is higher than five times the standard height, the stroke is regarded as an abnormal stroke and deleted in advance to prevent the abnormal stroke from affecting subsequent system processing.

[0020] <Front division unit 30> The pre-division unit 30 is connected to the anomaly deletion unit 20 and receives the handwritten document after it has been processed by the anomaly deletion unit 20. First, the pre-division unit 30 roughly pre-divides the handwritten document into a number of lines, and the method is to divide the handwritten document along the y-axis direction. For the first touched line, the leftmost end and the rightmost end are detected, that is, from left to right, the first touched point of the line is the leftmost end, and the last touched point of the line is the rightmost end. Then, the first line is obtained by pre-dividing the line by a certain set multiple N1 of the standard height. For the remaining handwritten document, the same method is used to obtain each successive line (see step 125).

[0021] The pre-division conditions will now be described. Condition 1: Subtract the maximum x-axis value of all conventional strokes that are larger than the standard height from the maximum x-axis value of all strokes in the current line. Condition 2: Subtract the maximum y-axis value of all strokes in the current line that are greater than twice the standard height from the minimum y-axis value of the previous stroke. Condition 3: The minimum value of the y-axis of the conventional stroke is greater than the maximum value of the y-axis of the shift offset. The definition of the shift offset (offset) is to set the minimum value of the x-axis of the conventional stroke as xmin and the maximum value as xmax. The maximum value of the y-axis of all strokes that fall within the (xmin, xmax) range of the current line must be less than the offset formed by the line of the minimum value of the y-axis of the conventional stroke.

[0022] If either condition 1 and condition 3 or condition 2 and condition 3 are satisfied, the current stroke is determined to belong to the next line.

[0023] Of two consecutive lines, the difference between the rightmost side of the previous line and the leftmost side of the next line is calculated as the horizontal axis difference dx, and the difference between the bottommost side of the first character of the previous line and the topmost side of the first character of the next line is calculated as the height difference dy. The horizontal axis difference dx must be greater than the set multiple (e.g., 5 times) of the standard width described above. The height difference dy must be greater than 0. In this case, the division of the two lines is established. The clearer the shift amount of the x-axis between the two lines, the more accurate the division. If dx and dy do not satisfy the above conditions, the two consecutive lines are recognized as a single line.

[0024] <Special Unit 40> The pre-division unit 30 is connected to receive the handwritten document divided by the pre-division unit 30. First, the isolated points and the inverted writing are set to be in a line by themselves, and these lines are merged with other lines close to each other in the following steps (see step 130).

[0025] The definition of the isolated point is as follows: the y-axis distance Dy is obtained by subtracting the y-axis value of the midpoint of the stroke in the y-axis direction from the y-axis value of the point in the stroke, and the y-axis distance is calculated for all points of all strokes in the handwritten manuscript in this manner, and these y-axis distances Dy are distributed to obtain a distribution close to the normal distribution (see Figures 3A and 3B). If all points of a stroke are outside 95.449974% (i.e., outside two standard differences) of this distribution, the stroke is regarded as an isolated point, and the stroke is set to a line by itself.

[0026] The input handwritten manuscript also records the time sequence in which each stroke was written. Therefore, the inverted stroke is defined as a difference in the y-axis direction of the line to which the stroke belongs and the currently written stroke belongs when writing that is greater than N5 times the standard height. This difference is set as the distance difference between the lowest point of the stroke on the y-axis and the highest point of the line to which the currently written stroke belongs.

[0027] <Line division unit 50 too large> The oversized line splitting unit 50 is connected to the extra line unit 40 and the pre-segmenting unit 30, and receives the handwritten document after being processed by the extra line unit 40. The oversized line splitting unit 50 is used to re-segment the oversized lines split by the pre-segmenting unit 30 (see step 135).

[0028] The definition of the oversized line is that if the height (y-axis direction) of the line after division by the pre-division unit 30 is higher than the set multiple N2 of the standard height, the line is determined to be oversized. In this case, a clustering algorithm (e.g., k-means algorithm) is applied to perform a clustering calculation according to the coordinate axis (y-axis) of the height of each stroke, and each stroke is classified based on its height, and then each stroke (character) is broken into a new line based on the calculation result. Thus, the oversized line is divided again into multiple lines. After the oversized line is calculated, the divided lines divided from the oversized line and the original divided lines are rearranged, and the rearranged order of the lines matches the order of the original handwritten manuscript (see step 140).

[0029] <Spatial Merging Unit 60> The space merging unit 60 is connected to the oversized line dividing unit 50 and receives the handwritten document processed by the oversized line dividing unit 50. The space merging unit 60 calculates the horizontal axis difference dx and the height difference dy for the new line after dividing the oversized line, and determines whether such a line break is established. If not, the division is abandoned and the original oversized line is received as one line (see step 145).

[0030] Also, the lines formed by the isolated points and the downstrokes are merged into other lines based on the above calculation.

[0031] <Multiline unit 70> It is connected to the spatial merging unit 60 and receives the handwritten document after processing by the spatial merging unit 60. The multi-line unit 70 calculates the distance between strokes in the divided lines, which is the difference between the rightmost stroke of the previous stroke and the leftmost stroke of the next stroke. If the distance is greater than a preset width N7, the two strokes are boundaries, and the previous and next strokes are merged into two different lines (see step 150).

[0032] <All Words Judgment Unit 80> It is connected to the line height unit 10, receives the standard height and standard width calculated by the line height unit 10, and determines whether the handwritten document is multiple lines (see step 155), and if it is multiple lines, receives the above-mentioned line break result (see step 160). If it is not multiple lines, it determines whether it is a single character (see step 165). It calculates the height and width of the handwritten document, and if the height is smaller than N3 times (e.g., 3 times) the above-mentioned standard height, or if the width is smaller than N4 times (e.g., 3 times) the above-mentioned standard width, it determines that the handwritten document is a single character. It uses the aspect ratio as the basis for the determination, and if the aspect ratio is smaller than 2, it determines that the handwritten document is a single character (see step 170). If it is not a word, it determines the whole text as one line (see step 175).

[0033] <Test unit> It is used to test all other manuscripts.

[0034] When the user actually writes, the system calculates the standard height and standard width of the handwritten document. Abnormal strokes are detected and removed to prevent the abnormal strokes from affecting the segmentation and subsequent identification results. Then, a pre-segmentation unit is used to roughly segment the handwritten document into multiple lines. The isolated points and the inverted strokes are set to one line, and then lines that are too large are re-segmented, and line alignment and spatial merging are performed again on the segmentation result. If the spacing between words in the middle of a line is too large, it is re-segmented into multiple lines to improve the accuracy of line segmentation. Finally, it is determined whether a second segmentation method needs to be added based on whether the whole text is a word.

[0035] The present invention is not limited to the above-described embodiment, and various modifications are possible. For example, the present invention includes a configuration that is substantially the same as the configuration described in the embodiment. A substantially identical configuration is, for example, a configuration that has the same function, method, and result, or a configuration that has the same purpose and effect. The present invention also includes a configuration in which non-essential parts of the configuration described in the embodiment are replaced. The present invention also includes a configuration that has the same action and effect as the configuration described in the embodiment, or a configuration that can achieve the same purpose. The present invention also includes a configuration in which publicly known technology is added to the configuration described in the embodiment. [Explanation of symbols]

[0036] 1 System 2 processors 6. Memory 8. Electronic writing panel 10 line height units 20 Abnormal deletion units 30 Front split unit 40 Special Line Unit 50 Line division units too large 60 Spatial Merging Unit 70 Multi-line units 80 All-word decision unit 100 hosts

Claims

1. An information system based on automatic line-breaking of handwritten documents, which performs line breaks on a handwritten manuscript generated by a writer writing on an electronic writing panel, said system records each stroke (i.e., each coordinate point of the stroke) and the time when each coordinate point was generated, and hereinafter, the direction of the line when writing is referred to as the row direction (x direction) and the direction of skipping lines is referred to as the column direction (y direction), said system: A processor and a memory, the processor being used to perform operations required by the system, and the memory being used to store data or operation programs of the computer program of the system, related operation results and related data; a line height unit used to calculate a standard height and a standard width of the handwritten document; an abnormality deletion unit connected to the line height unit and receiving the standard height and standard width calculated by the line height unit, the abnormality deletion unit deletes abnormal strokes in advance from the handwritten document, and the abnormal stroke is defined as a stroke whose height is higher than the standard height and a predetermined multiple used for abnormal deletion, or the abnormal stroke is defined as a stroke whose width is higher than the predetermined multiple used for abnormal deletion of the standard width; a pre-division unit connected to the anomaly deletion unit and receiving the processed handwritten document from the anomaly deletion unit, the pre-division unit first divides the handwritten document into a number of lines, the pre-division method is according to the line direction of writing, and detects the leftmost end and the rightmost end of the first touched line, that is, from left to right, the first touched point of the line is the leftmost end, and the last touched point of the line is the rightmost end, and then the pre-division is performed with the set multiple of the standard height pre-division as the line height, to obtain one line, and the remaining handwritten document is obtained in the same manner by the pre-division unit; a special line unit connected to the pre-division unit, which first sets each isolated point and each inverted stroke of the handwritten manuscript after the pre-division to a single line, and these pre-set lines are merged into other lines in the following steps; an oversized line dividing unit connected to the special line unit, the oversized line dividing unit is used to re-divide the oversized line divided by the previous dividing unit, and the definition of the oversized line is that if the height of the line divided by the previous dividing unit is higher than a set multiple used for the oversized line of the standard height, the oversized line dividing unit determines that the line is oversized; a space merging unit connected to the oversized line dividing unit, for receiving the handwritten document processed by the oversized line dividing unit, for determining whether a new line is established after dividing the oversized line, and for abandoning the division and accepting the original oversized line as one line if the new line is not established, and for merging the isolated point and the line formed by the oversized writing into another line that is close to the isolated point and the oversized writing; a multi-line unit connected to the spatial merging unit and receiving the handwritten manuscript processed by the spatial merging unit, the multi-line unit calculates the distance between strokes in the lines that have already been divided, the distance being the difference between the rightmost stroke of the previous stroke and the leftmost stroke of the next stroke, and if the distance is greater than a set width used for the multiple lines, the multi-line unit uses the two strokes as a boundary and incorporates the previous and next strokes into two different lines, respectively; An information system based on automatic line breaks for handwritten documents, characterized in that line breaks are performed for handwritten manuscripts inputted based on the above method.

2. 2. The information system based on automatic line breaks for handwritten documents according to claim 1, further comprising a whole word determination unit connected to the line height unit, receiving the standard height and standard width calculated by the line height unit and determining whether the handwritten document is a single character, wherein the system calculates the height and width of the handwritten document, and recognizes the handwritten document as a single character if the height is smaller than a multiple of the standard height used for whole word determination, or if the width is smaller than a multiple of the standard width used for whole word determination, or if the aspect ratio is smaller than 2.

3. 2. An information system based on automatic line breaks for handwritten documents as described in claim 1, characterized in that the line height unit arranges the strokes of a handwritten document based on the height of each stroke, and then sets the height of the strokes of the set proportional value as the standard height, and similarly arranges the strokes of all handwritten documents based on the width of the strokes, and then sets the width of the strokes of the set proportional value as the standard width.

4. 2. The information system based on automatic line breaks for handwritten documents according to claim 1, further comprising: a horizontal axis difference dx between the rightmost side of the previous line and the leftmost side of the next line of two consecutive lines; a height difference dy between the bottommost side of the first character of the previous line and the topmost side of the first character of the next line of two consecutive lines; the horizontal axis difference dx must be greater than a preset multiple of the standard width used for the previous division; and the height difference dy must be greater than 0; the division of the two consecutive lines is established, and the clearer the shift amount of the x-axis between the two consecutive lines is, the more accurate the division is; and if dx and dy do not satisfy the above conditions, the two consecutive lines are recognized as a single line.

5. The definition of the isolated point is to obtain a y-axis distance Dy by subtracting the y-axis value of the midpoint of the stroke in the y-axis direction from the y-axis value of a point in the stroke, and in this manner, the y-axis distances are calculated for all points of all strokes in the handwritten manuscript, and these y-axis distances Dy are distributed. If all points of a stroke are outside two standard differences of this distribution, the stroke is regarded as an isolated point, and the stroke is set to a line by itself.

6. The information system based on automatic line breaks for handwritten documents as described in claim 1, characterized in that the inverted stroke is defined as a difference in the y-axis direction of the line to which the stroke and the stroke currently being written belong when writing, which is greater than a set multiple used for determining inverted stroke of the standard height, and this difference is set as the distance difference between the lowest point of the stroke on the y-axis and the highest point on the y-axis of the line to which the stroke currently being written belongs.

7. The information system based on automatic line breaks for handwritten documents as claimed in claim 1, characterized in that the oversized line dividing unit applies a clustering algorithm to obtain the coordinate axis of the height of each stroke, perform a clustering calculation, classify each stroke based on its height, and then break each stroke based on the calculation result, thereby re-dividing the oversized line into multiple lines.

8. 2. The information system based on automatic line breaks for handwritten documents according to claim 1, wherein the oversized line dividing unit calculates the oversized line, then rearranges the divided lines divided from the oversized line and the original divided lines, and makes the rearranged lines conform to the arrangement order of the original handwritten manuscript.

9. The information system based on automatic line breaks for handwritten documents as described in claim 4, characterized in that the space merging unit calculates the above-mentioned horizontal axis difference dx and the height difference dy of the new line after dividing the above-mentioned oversized line, determines whether such a line break is valid, and if not, abandons the division and accepts the original oversized line as one line.

Images