Recognition processing device, recognition processing program, recognition processing method, and recognition processing system

Abstract

Obtain features of online text input more efficiently and use machine learning. [Solution] The present invention relates to a recognition processing device, and the recognition processing device of the present invention is characterized by comprising: normalization means for acquiring input stroke data indicating an input pattern of strokes made by an electronic pen in chronological order for each input character and normalizing the acquired input stroke data to an input pattern with a fixed number of samples to acquire normalized stroke data, input vector data acquisition means for converting the normalized stroke data normalized by the normalization means into input vector data expressed by feature quantities with the fixed number of samples, and character recognition processing means for performing character recognition processing on the input vector data acquired by the input vector data acquisition means using a learning model trained by machine learning using the input vector data acquired by the input vector data acquisition means.

Description

[Technical field]

[0001] The present invention relates to a recognition processing device, a recognition processing program, a recognition processing method, and a recognition processing system, and can be applied to, for example, online handwritten character recognition processing. [Background technology]

[0002] Conventionally, a method has been proposed for online handwritten character recognition processing in which features are acquired from the strokes (handwriting) made when inputting characters, machine learning is performed based on the acquired features to acquire a learning model, and the acquired learning model is used to recognize the input characters.

[0003] A technique described in Patent Document 1 is one example of a method for performing character recognition processing by performing machine learning on the feature amounts of strokes as described above.

[0004] In the technology described in Patent Document 1, the position of strokes for each character over time (hereinafter referred to as the "input pattern") is obtained as a sample, feature points (positions that make up the strokes) are matched between the sampled input pattern and a standard pattern (an input pattern when input in standard handwriting), and the difference between the matched feature points is obtained as a feature value to be used in character recognition processing.

[0005] In the technology described in Patent Document 1, machine learning is performed using the feature values ​​acquired for learning as training data to acquire a learning model. In the technology described in Patent Document 1, in character recognition processing, the differences between all standard patterns and feature points of a sample input pattern are calculated and acquired as feature amounts, and the feature amounts from all the acquired standard patterns are input to the learning model to perform character recognition processing. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2018-112521 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the technology described in Patent Document 1, since a suitable standard pattern must be prepared for each character in order to obtain a learning model, the cost of creating the learning model is high.In addition, in the technology described in Patent Document 1, since the feature amounts between the sample input pattern and all standard patterns must be obtained and then recognized during character recognition processing, the processing load is very high.

[0008] In the technology described in Patent Document 1, the difference from a standard pattern is used as the feature value for the character input feature in order to secure a larger amount of information, as described above. However, as described above, the technology described in Patent Document 1 requires a great deal of resources (e.g., labor costs, hardware resources, etc.) for creating a learning model and for recognition processing.

[0009] Therefore, there is a demand for a recognition processing device, a recognition processing program, a recognition processing method, and a recognition processing system that can obtain features of online character input more efficiently and perform machine learning. [Means for solving the problem]

[0010] The first recognition processing device of the present invention is a device that displays an input pattern of strokes made by an electronic pen in chronological order for each input character. The input pattern includes position information of feature points in a time series order. Obtain input stroke data and normalize the input stroke data to an input pattern with a fixed number of samples. Normalization process is performed to The present invention has a normalization means for acquiring normalized stroke data, an input vector data acquisition means for converting the normalized stroke data normalized by the normalization means into input vector data expressed by features of the fixed number of samples, and a character recognition processing means for performing character recognition processing on the input vector data acquired by the input vector data acquisition means using a learning model that has been machine-learned using the input vector data acquired by the input vector data acquisition means. The normalization process performed by the normalization means includes a first normalization process step of thinning out feature points from the input stroke data so that intervals between all feature points adjacent in chronological order are equal to or greater than a predetermined threshold value, thereby obtaining intermediate stroke data, and a second normalization process step of interpolating feature points from the intermediate stroke data so that intervals between all feature points adjacent in chronological order are equal to or less than the threshold value. It is characterized by:

[0011] The second recognition processing program of the present invention is a computer is caused to function as: a normalization means for acquiring input stroke data indicating an input pattern of strokes made by an electronic pen in chronological order for each input character, the input pattern including position information of feature points in chronological order, and performing a normalization process for normalizing the acquired input stroke data to an input pattern with a fixed number of samples to acquire normalized stroke data; an input vector data acquisition means for converting the normalized stroke data normalized by the normalization means into input vector data expressed by feature amounts of the fixed number of samples; and a character recognition processing means for performing a character recognition process on the input vector data acquired by the input vector data acquisition means, using a learning model trained by machine learning using the input vector data acquired by the input vector data acquisition means, wherein the normalization process performed by the normalization means includes a first normalization processing step for acquiring intermediate stroke data by thinning out feature points from the input stroke data such that an interval between all feature points adjacent in chronological order is equal to or greater than a predetermined threshold; and a second normalization processing step for acquiring data by interpolating feature points from the intermediate stroke data such that an interval between all feature points adjacent in chronological order is equal to or less than the threshold. .

[0012] A third aspect of the present invention is a recognition processing method performed by a recognition processing device, the recognition processing device comprising: a normalization means; An input vector data acquisition means; The character recognition apparatus has a character recognition processing means and a character recognition result output means, wherein the normalization means acquires input stroke data indicating an input pattern of strokes made by an electronic pen in chronological order for each input character, the input stroke data including position information of feature points in chronological order of the input pattern, and performs a normalization process to normalize the acquired input stroke data into an input pattern with a fixed number of samples to acquire normalized stroke data, the input vector data acquisition means converts the normalized stroke data normalized by the normalization means into input vector data expressed by feature amounts of the fixed number of samples, and the character recognition processing means performs character recognition processing on the input vector data acquired by the input vector data acquisition means using a learning model trained by machine learning using the input vector data acquired by the input vector data acquisition means, and the normalization process performed by the normalization means includes a first normalization processing step of thinning out feature points from the input stroke data so that an interval between all adjacent feature points in chronological order is equal to or greater than a predetermined threshold value to acquire intermediate stroke data, and a second normalization processing step of interpolating feature points from the intermediate stroke data so that an interval between all adjacent feature points in chronological order is equal to or less than the threshold value.

[0013] The fourth invention of the present invention is a recognition system having a pen tablet capable of receiving input using an electronic pen and a front electronic pen, and a recognition processing device that recognizes characters written on the pen tablet by a user with the electronic pen, characterized in that the recognition processing device of the first invention of the present invention is applied as the recognition processing device. Effect of the Invention

[0014] According to the present invention, feature quantities of online character input can be obtained more efficiently and machine learning can be performed. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram showing an overall configuration according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of input vector data according to the first embodiment; [Diagram 3] FIG. 2 is a diagram showing a specific example (part 1) of input vector data according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing a specific example (part 2) of input vector data according to the first embodiment. [Diagram 5] FIG. 2 is a diagram showing an example of the configuration of a character input screen according to the first embodiment. [Figure 6] 4A to 4C are diagrams illustrating an example of the configuration of input stroke data acquired by a stroke data processing unit according to the first embodiment. [Figure 7] 4 is a flowchart showing a process in which a stroke data processing unit according to the first embodiment generates input vector data for online character recognition processing from input stroke data. [Figure 8] 5A to 5C are diagrams illustrating a process in which the stroke data processing unit according to the first embodiment normalizes input vector data for online character recognition processing to obtain first normalized stroke data. [Figure 9] 6A to 6C are diagrams illustrating processing performed by the stroke data processing unit according to the first embodiment to thin out feature points of first normalized stroke data. [Figure 10] FIG. 13 is a diagram illustrating an example in which the stroke data processing unit according to the first embodiment acquires second normalized stroke data by performing feature point interpolation processing for filling intervals between feature points of first normalized stroke data to a predetermined interval or less. [Figure 11]FIG. 11 is a diagram illustrating an example of third normalized stroke data obtained by the stroke data processing unit according to the first embodiment extracting N+1 feature points from the second normalized stroke data. [Figure 12] 13A to 13C are diagrams illustrating an image based on third normalized stroke data acquired by a stroke data processing unit according to the first embodiment. [Figure 13] 13 illustrates an example of input vector data acquired by the stroke data processing unit according to the first embodiment based on the third normalized stroke data. [Figure 14] 11 is a flowchart showing a process in which a stroke data processing unit according to the first embodiment generates input image data for offline character recognition processing from input stroke data. [Figure 15] 6A to 6C are diagrams illustrating normalization processing in the process in which the stroke data processing unit according to the first embodiment generates input image data for offline AI processing. [Figure 16] FIG. 13 is a diagram showing input image data acquired by the stroke data processing unit according to the first embodiment by drawing each feature point of the fifth normalized stroke data in a normalized area of ​​64 pixels×64 pixels. [Figure 17] 5 is a flowchart showing a learning process when the character recognition processing unit according to the first embodiment operates in a learning mode. [Figure 18] 5 is a flowchart showing a recognition process when the character recognition processing unit according to the first embodiment operates in a recognition process mode. [Figure 19] 13 is a timing chart (part 1) showing the state of the pen tip over time when a user inputs characters into the pen tablet using an electronic pen in the second embodiment. [Figure 20] 13 is a timing chart (part 2) showing the pen tip state over time when a user inputs characters to a pen tablet using an electronic pen in the second embodiment. [Figure 21] The graph shows the results of counting the pen tip state at each characteristic point shown in the timing chart of FIG. [Figure 22] FIG. 20 is a diagram showing each sample (characteristic point) shown in the timing chart of FIG. 19. [Figure 23] 20 is a diagram showing an example of input stroke data when a second normalization method is applied to each feature point shown in the timing chart of FIG. 19. FIG. [Figure 24] FIG. 20 is a diagram showing an example of input stroke data when the third normalization method is applied to each feature point shown in the timing chart of FIG. 19. [Diagram 25] FIG. 20 is a diagram showing an example of input stroke data when a fourth normalization method is applied to each feature point shown in the timing chart of FIG. 19. [Figure 26] FIG. 20 is a diagram showing an example of input stroke data when the fifth normalization method is applied to each feature point shown in the timing chart of FIG. 19. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] (A) First embodiment Hereinafter, a first embodiment of a recognition processing device, a recognition processing program, a recognition processing method, and a recognition processing system according to the present invention will be described in detail with reference to the drawings. In this embodiment, an example in which an information processing terminal is configured as the recognition processing device of the present invention will be described.

[0017] (A-1) Configuration of the First Embodiment Fig. 1 is a block diagram showing the overall configuration of a recognition processing system 1 according to a first embodiment. Note that reference characters in parentheses in Fig. 1 are reference characters that are used only in a second embodiment described later.

[0018] The recognition processing system 1 includes an information processing terminal 10, a pen tablet 20, and an electronic pen 30. The pen tablet 20 is a device capable of receiving input using the electronic pen 30.

[0019] The pen tablet 20 is a device that detects the pen tip 31 of the electronic pen 30 on the surface of the display panel 21 (a device that functions as a so-called "pointing device"). In addition, the display panel 21 can also output an image based on a video signal supplied from the information processing terminal 10. In the recognition processing system 1, by displaying the trajectory of the pen tip 31 on the display panel 21, it is possible to accept character input operations from the user.

[0020] The information processing terminal 10 includes a control unit 11, a video IF 12, and a USB port 13.

[0021] The information processing terminal 10 can be configured by installing a program (including the recognition processing program according to the embodiment) in various computers (for example, a PC, etc.).

[0022] The control unit 11 includes a content processing unit 111 , a display driver 112 , a pen tablet driver 113 , and a character recognition processing unit 114 .

[0023] The content processing unit 111 is an application program that accesses the pen tablet 20 via a display driver 112 and a pen tablet driver 113, and processes various contents using the pen tablet 20 and the electronic pen 30 (for example, contents including various operation screens involving character input using the pen tablet 20 and the electronic pen 30).

[0024] The content processing unit 111 displays an operation screen (GUI) accompanying character input via a display driver 112, and acquires contents input using the electronic pen 30 on the liquid crystal tablet (for example, information on the coordinates of the portion touched by the electronic pen 30, etc.) via a pen tablet driver 113. The content processing unit 111 acquires stroke data during character input (data including the coordinates of the electronic pen 30 for each time series; hereinafter, referred to as "input stroke data") via the pen tablet driver 113. In this embodiment, the content processing unit 111 supplies the input stroke data to a character recognition processing unit 114.

[0025] The character recognition processing unit 114 performs character recognition processing based on input stroke data, and has a stroke data processing unit 1141, an online AI processing unit 1142, an offline AI processing unit 1143, and a character recognition result output unit 1144. The character recognition processing unit 114 has a function corresponding to the recognition processing program according to this embodiment.

[0026] The stroke data processing unit 1141 performs processing to generate, from the input stroke data, vector data (input vector data) applicable to processing by the online AI processing unit 1142, and image data (hereinafter referred to as "input image data") applicable to processing by the offline AI processing unit 1143.

[0027] In this embodiment, the input stroke data and input image data are provided from the content processing unit 111 to the character recognition processing unit 114, but the source of providing the input stroke data and input image data to the character recognition processing unit 114 is not limited to this. For example, input stroke data and input image data created externally may be provided to the character recognition processing unit 114 for processing.

[0028] When input vector data is supplied, the online AI processing unit 1142 performs character recognition processing or learning processing based on the input vector data. When the character recognition processing unit 114 operates in a learning mode, the online AI processing unit 1142 performs machine learning processing using a set of input vector data and a correct answer label (teacher label) to update a learning model. When input vector data is supplied, the online AI processing unit 1142 performs character recognition processing or learning processing based on the input vector data. When the character recognition processing unit 114 operates in a learning mode, the online AI processing unit 1142 performs machine learning processing using a set of input vector data and a correct answer label. When the character recognition processing unit 114 operates in a recognition processing mode, the online AI processing unit 1142 performs character recognition processing using a learned learning model based on the input vector data, and outputs a character recognition result (hereinafter also referred to as an "online character recognition result"). In this embodiment, the online AI processing unit 1142 adds reliability data to the online character recognition result.

[0029] When input image data is supplied, the offline AI processing unit 1143 performs character recognition processing or learning processing based on the input image data. When the character recognition processing unit 114 operates in a learning mode, the offline AI processing unit 1143 performs machine learning processing using a set of input image data and a correct answer label to update a learning model. When input image data is supplied, the offline AI processing unit 1143 performs character recognition processing or learning processing based on the input image data. When the character recognition processing unit 114 operates in a learning mode, the offline AI processing unit 1143 performs machine learning processing using a set of input image data and a correct answer label. When the character recognition processing unit 114 operates in a recognition processing mode, the offline AI processing unit 1143 performs character recognition processing using a learned learning model based on the input image data, and outputs a character recognition result (hereinafter also referred to as an "offline character recognition result"). In this embodiment, the offline AI processing unit 1143 adds reliability data to the offline character recognition result.

[0030] In this embodiment, the online AI processing unit 1142 and the offline AI processing unit 1143 can be configured using various machine learning engines (AI platforms). Therefore, in this embodiment, the details of the machine learning method (the method of creating a learning model) in the online AI processing unit 1142 and the offline AI processing unit 1143 and the recognition process (determination process) using the created learning model will be omitted.

[0031] When character recognition processing unit 114 operates in the recognition processing mode, character recognition result output unit 1144 outputs a final character recognition result (hereinafter also referred to as an "output character recognition result") based on the online character recognition result and the offline character recognition result. Character recognition result output unit 1144 may evaluate the online character recognition result and the offline character recognition result, and output one of the character recognition results based on the evaluation result. For example, character recognition result output unit 1144 may adopt the online character recognition result or the offline character recognition result, which has a higher added evaluation value, as the output character recognition result.

[0032] In this embodiment, when content processing unit 111 operates character recognition processing unit 114 in the learning mode, it supplies input stroke data as well as a correct label (an identifier of a correct character corresponding to the input stroke data) corresponding to the input stroke data to character recognition processing unit 114. Also, in this embodiment, when content processing unit 111 operates character recognition processing unit 114 in the recognition processing mode, it supplies input stroke data to character recognition processing unit 114 and obtains an output character recognition result.

[0033] Next, a configuration example of the input stroke data acquired by the stroke data processing unit 1141 will be described.

[0034] The input stroke data includes information regarding the position of the pen tip 31 of the electronic pen 30 for each time series recognized by the pen tablet 20 (display panel 21).

[0035] Hereinafter, the direction parallel to the surface of the display panel 21 that comes into contact with the pen tip 31 of the electronic pen 30 (hereinafter referred to as the "display contact surface") is referred to as the "lateral direction", and the direction perpendicular to the display contact surface is referred to as the "height direction". In addition, below, the state (status) regarding the height direction between the pen tip 31 and the display contact surface is referred to as the "pen tip state". In the first embodiment, the term "pen tip state" is described as a status indicating either a state in which the pen tip 31 comes into contact with the display contact surface (hereinafter referred to as the "contact state") or a state in which the pen tip 31 does not come into contact with the display contact surface (hereinafter referred to as the "non-contact state").

[0036] In this embodiment example, the input stroke data is described as including information on the lateral position of the pen tip 31 recognized by the pen tablet 20 (display panel 21) (hereinafter referred to as the "sample" or "sample position") and information on the pen tip state (the vertical position of the pen tip 31) recognized by the pen tablet 20 (display panel 21).

[0037] In the following, as shown in FIG. 1, the horizontal direction (left-right direction) of the pen tablet 20 (display panel 21) as the "screen / touch panel / pointing device" is taken as the X-axis, and the vertical direction (up-down direction) of the pen tablet 20 (display panel 21) as the "screen / touch panel / pointing device" is taken as the Y-axis, to represent the coordinate system of the sample position in the input stroke data. In this embodiment, the coordinate system of the sample position has the upper left point of the pen tablet 20 (display panel 21) as the origin (x=0, y=0), the downward direction as the "direction in which the Y coordinate increases (+Y direction)", and the rightward direction as the "direction in which the X coordinate increases (+X direction)". Therefore, in the following, the upward direction is the "direction in which the Y coordinate decreases (-Y direction)", and the leftward direction as the "direction in which the X coordinate decreases (-X direction)". In the following, the positions along the trajectory of the sample positions (samples) for each time series (including positions on a normalized area to be described later) are collectively referred to as "feature points".

[0038] Next, an overview of the character recognition process by the online AI processing unit 1142 will be described.

[0039] First, an example of the configuration of input vector data processed by the online AI processing unit 1142 will be described.

[0040] In this embodiment, the input vector data is data configured of M-dimensional vectors corresponding to N feature points for each character in each time series (that is, data represented by a determinant of M columns by N rows).

[0041] In this embodiment, the input vector data includes, for each feature point, the coordinates of the current feature point, the amount of movement (vector) to the next feature point in the time series, and information indicating the pen tip state.

[0042] In this embodiment, the input vector data is assumed to be vector data represented by seven-dimensional parameters (M=7) shown in FIG.

[0043] In this embodiment, the input vector data is described as including the X coordinate (hereinafter, VX) of the feature point, the Y coordinate (hereinafter, VY) of the feature point, the rightward movement amount (hereinafter, VR) from the feature point to the feature point of the next time series, the upward movement amount (hereinafter, VU) from the feature point to the feature point of the next time series, the leftward movement amount (hereinafter, VL) from the feature point to the feature point of the next time series, the downward movement amount (hereinafter, VD) from the feature point to the feature point of the next time series, and a value (hereinafter, VT) indicating the pen tip state in the section between the feature point and the next time series. Note that VT is set to either "1" indicating a state in which the pen tip 31 of the electronic pen 30 is in contact with the display contact surface (hereinafter, referred to as a "contact state") or "0" indicating a state in which the pen tip 31 is not in contact (hereinafter, referred to as a "non-contact state"). Therefore, if the feature quantity of a time series t (t is an integer between 1 and N) is V(t), then V(t) can be expressed as in the following equation (1). If the input vector data for one character is Z, then Z can be expressed as a determinant, as in the following equation (2). (t)={VX(t),VY(t),VR(t),VU(t),VL(t),VD(t),VT(t)} …(1)

number

[0044] As described above, among the feature amounts V(t), VR(t), VU(t), VL(t), and VD(t) represent the amounts of movement to the feature points at the next time series t + 1. In this embodiment, in the coordinate system representing the input vector data, the upper left is set as the origin (X = 0, Y = 0), the value of X increases in the right direction, and the value of Y increases in the downward direction. Then, the direction in which Y increases (+Y direction) on the Y-axis is the "downward direction" and the direction in which Y decreases (-Y direction) is the "upward direction". Also, the direction in which X increases (+X direction) on the X-axis is the "right direction" and the direction in which X decreases (-X direction) is the "left direction". In this case, since VR(t) represents the amount of movement in the +X direction, VR(t) = VX(t + 1) - VX(t) when VX(t + 1) > VX(t), and VR(t) = 0 when VX(t + 1) ≤ VX(t). Also, since VL(t) represents the amount of movement in the -X direction, VL(t) = VX(t) - VX(t + 1) when VX(t + 1) < VX(t), and VL(t) = 0 when VX(t + 1) ≥ VX(t). Further, since VU(t) represents the amount of movement in the -Y direction, VU(t) = VY(t) - VY(t + 1) when VY(t + 1) < VY(t), and VU(t) = 0 when VY(t + 1) ≥ VY(t). Furthermore, since VD(t) represents the amount of movement in the +Y direction, VD(t) = VY(t + 1) - VY(t) when VY(t + 1) > VY(t), and VD(t) = 0 when VY(t + 1) ≤ VY(t).

[0045] In this embodiment, the amount of movement (amount of motion) in the input vector data is expressed in four dimensions (VR(t), VU(t), VL(t), VD(t)), but it may be expressed in two dimensions (horizontal and vertical) if negative values ​​are acceptable. By expressing the amount of movement in the input vector data in four dimensions (VR(t), VU(t), VL(t), VD(t)), even changes on the same axis can be expressed as features of different items, which can affect the accuracy of machine learning (i.e., adjustment of the judgment process by AI). For example, in this embodiment, the amount of movement of the input vector data can be expressed in either two dimensions (x(t), Y(t)) or four dimensions (VR(t), VU(t), VL(t), VD(t)), and the processing using the one with better recognition accuracy can be performed by the operator's operation, etc.

[0046] Next, a specific example of the feature amount V(t) at time t constituting the input vector data will be described with reference to FIGS.

[0047] Fig. 3 is a diagram showing the feature amount V(1) when the feature points at time t = 1 and t = 2 are P1 and P2, respectively. In Fig. 3, the coordinates of P1 are (X, Y) = (1, 2), and the coordinates of P2 are (X, Y) = (2, 1).

[0048] Then, the feature amount V(1) can be expressed as the following equation (3). V(1)={VX(1),VY(1),VR(1),VU(1),VL(1),VD(1),VT(1)} ={1,2,1,1,0,0,1}…(3)

[0049] Figure 4 shows feature quantities V(1), V(2), and V(3) when feature points at time t=1, t=2, t=3, and t=4 are P1, P2, P3, and P4, respectively. In Figure 4, the coordinates of P1 are (X,Y)=(1,3), the coordinates of P2 are (X,Y)=(3,1), the coordinates of P3 are (X,Y)=(1,1), and the coordinates of P4 are (X,Y)=(3,3). In Figure 4, the section between P2 and P3 is considered to be a section in a non-contact state.

[0050] Then, the feature amounts V(1), V(2), and V(3) can be expressed by the following equations (4) to (6), respectively. V(1)={VX(1),VY(1),VR(1),VU(1),VL(1),VD(1),VT(1)} ={1,3,2,1,0,0,1}…(4) V(2)={VX(2),VY(2),VR(2),VU(2),VL(2),VD(2),VT(2)} ={3,1,0,0,2,0,0}…(5) V(3)={VX(3),VY(3),VR(3),VU(3),VL(3),VD(3),VT(3)} ={1,1,2,0,0,2,1}…(6)

[0051] (A-2) Operation of the First Embodiment Next, the operation of the recognition processing system 1 of this embodiment having the above configuration (each procedure of the recognition processing method according to the embodiment) will be described.

[0052] First, an example of processing for accepting character input from a user using the pen tablet 20 (display panel 21) and the electronic pen 30 in the recognition processing system 1 will be described.

[0053] As described above, in the recognition processing system 1 of this embodiment, regardless of whether the system is operating in the learning mode or the recognition processing mode, the content processing unit 111 displays an operation screen (hereinafter referred to as the "character input screen") on the pen tablet 20 (display panel 21) for accepting character input from the user (character input using the electronic pen 30).

[0054] As the character input screen, operation screens of various configurations can be applied, but for example, an operation screen as shown in FIG. 5 may be applied.

[0055] In the character input screen shown in FIG. 5, rectangular areas capable of accepting character input (hereinafter, referred to as "character input fields") are arranged. In FIG. 5, one character input field F101 is arranged. In FIG. 5, characters that request the user to input characters (characters that function as correct labels in learning processing and determination processing) are added to the character input fields. For example, the character "10" (kanji) is added to each character input field F101. This allows the recognition processing system 1 to accept input of the added character (input using the electronic pen 30) from the user within the frame of each character input field. FIG. 5 shows a state in which the character "10" is handwritten with the electronic pen 30 within the frame of the character input field F101. Note that, in FIG. 5, for the sake of simplicity, an example in which one character input field is arranged on one screen is shown, but the layout of such a character input field is not limited, and it goes without saying that a plurality of character input fields may be arranged.

[0056] In the character input screen of this embodiment, for the sake of simplicity, an example will be given in which one area (character input field) is set for one character input, but the recognition processing system 1 may also be configured to accept multiple character inputs in one area and extract the characters.

[0057] In this embodiment, the content processing unit 111 is capable of presenting a character input screen such as that shown in FIG. 5 to a user, accepting writing input from the user using the electronic pen 30, and acquiring input stroke data corresponding to the input character based on the input.

[0058] In this embodiment, the content processing unit 111 is described as acquiring, as sample positions, the coordinates when the pen tip state changes from a non-contact state to a contact state, the coordinates for each predetermined period (e.g., about 0.1 seconds) while the pen tip state is in the contact state, and the coordinates when the pen tip state changes from a contact state to a non-contact state.

[0059] FIG. 6 is a diagram showing an example of the configuration of input stroke data acquired by the stroke data processing unit 1141 when the character "10" is handwritten with the electronic pen 30 within the frame of the character input field F101 as shown in FIG.

[0060] Fig. 6(a) is a diagram depicting sample points when the character "10" is handwritten with the electronic pen 30. In the coordinate system shown in Fig. 6(a), the X range is 0 to 1000, and the Y range is 0 to 10000. That is, the image of the input stroke data shown in Fig. 6 is an image of 1000 pixels x 1000 pixels.

[0061] Fig. 6(b) is a diagram showing the values ​​for each time series number of the input stroke data shown in Fig. 6(b). The time series number is a value that indicates the order of the time series, and the smaller the value, the earlier the time series (time). As shown in Fig. 6(b), in the input stroke data, information on the X and Y coordinate values ​​and the pen tip state value is recorded for each time series number.

[0062] At this time, the stroke data processing unit 1141 manages the data of each sample (feature point) of the input stroke data by dividing it into strokes (strokes). For example, by using data with a pen tip state of "0" as a boundary in the input stroke data, a list of sample positions for each stroke (stroke) can be obtained. For example, in FIG. 6(a), the 14th sample position from the top has a pen tip state of "0", so the sample positions from 1 to 13 in the time series are the first stroke, and the sample positions from 14 to 23 in the time series are the second stroke.

[0063] Next, a process in which the stroke data processing unit 1141 generates input vector data for online character recognition processing from input stroke data will be described with reference to the flowchart of FIG.

[0064] First, it is assumed that input stroke data for one character is supplied from the content processing unit 111 to the stroke data processing unit 1141 and is held therein (S101).

[0065] Next, the stroke data processing unit 1141 acquires data (hereinafter referred to as "first normalized stroke data") normalized so that the input stroke data fits exactly within an area of ​​a predetermined resolution (hereinafter referred to as "normalized area") (S102).

[0066] FIG. 8 is a diagram showing a process in which the stroke data processing unit 1141 normalizes input vector data for online character recognition processing to obtain first normalized stroke data.

[0067] FIG. 8(a) is an image obtained by cutting out only the area where the sample positions (feature points) are drawn from the input stroke data shown in FIG.

[0068] In the input stroke data shown in Fig. 6, the maximum X value is 635 and the minimum X value is 427. Also, in the input stroke data shown in Fig. 6, the maximum Y value is 658 and the minimum Y value is 388. Therefore, the image in Fig. 8(a) (the cropped image) is an image of 208 pixels x 270 pixels (the number of pixels in the X direction is 208 and the number of pixels in the Y direction is 270).

[0069] FIG. 8(b) shows an image obtained by converting the image in FIG. 8(a) into a normalized region of 100 pixels×100 pixels (a region with an aspect ratio of 1:1).

[0070] FIG. 8(c) is a diagram showing data (first normalized stroke data) corresponding to each feature point (each pixel) of the image in the normalized region of FIG. 8(b).

[0071] As shown in FIG. 8(b) and FIG. 8(c), the stroke data processing unit 1141 normalizes the input stroke data to a normalized area of ​​100 pixels×100 pixels by providing two-pixel margins at the top, bottom, left, and right edges. That is, the stroke data processing unit 1141 essentially performs a process of normalizing the input stroke data to an area of ​​96 pixels×96 pixels. In the example of FIG. 8, the stroke data processing unit 1141 obtains the coordinates of each feature point after a resolution conversion process for converting an image of the input stroke data (an image of 208 pixels×270 pixels) into an image of 96 pixels×96 pixels (an image with an aspect ratio of 1:1), thereby obtaining the first normalized stroke data shown in FIG. 8(c). At this time, various image processing methods can be applied to the specific method of the resolution conversion process performed by the stroke data processing unit 1141, so a description of the specific process will be omitted.

[0072] Next, the stroke data processing unit 1141 performs processing to thin out feature points from the first normalized stroke data so that the distance between each feature point and its adjacent feature point becomes equal to or greater than a predetermined distance (S103).

[0073] For example, the following formula (7) may be calculated based on a feature point of a time series t and a feature point of the next time series t+1, and if the formula holds, it may be determined that the distance between the two feature points is equal to or smaller than a predetermined value.

[0074] In equation (7), the x-coordinate of a feature point in time series t is x(t) and the y-coordinate is y(t), and the x-coordinate of a feature point in the next time series t+1 after time series t is x(t+1) and y(t+1). In addition, in formula (7), SIZE is the horizontal and / or vertical resolution of the entire image (here, 100). Here, a description will be given of applying SIZE / 100=100 / 10=10 as the threshold value (right side of formula (7)) for determining whether or not to thin out feature points, but this threshold value may be set to an arbitrarily designed value (for example, a suitable value may be searched for and set through experiments, etc.).

[0075] Here, the stroke data processing unit 1141 applies the following formula (7) to the feature points of each time series to perform calculations, and if the formula holds, thins out the feature points of the next time series. After thinning out the feature points of time series t+1, the stroke data processing unit 1141 may apply the following formula (7) to the feature points of the next time series as time series t+1 to perform calculations, and if the formula holds, repeat the thinning process.

[0076] Also, the stroke data processing unit 1141 performs the thinning process for each stroke. That is, the stroke data processing unit 1141 may repeat the thinning process for each stroke until the following formula (7) is not satisfied for all feature points of the time series (because the distance between all feature points is equal to or greater than a predetermined value). For example, the stroke data processing unit 1141 may extract the data of the first stroke (data of time series numbers 1 to 13) from the first normalized stroke data shown in FIG. 8(c) and perform the above-mentioned thinning process, and then extract the data of the second stroke (data of time series numbers 14 to 22) and perform the above-mentioned thinning process. {x(t+1)-x(t)} 2 +{y(t+1)-y(t)} 2>SIZE / 10 …(7)

[0077] FIG. 9 is a diagram showing a state in which feature points have been thinned out from the first normalized stroke data shown in FIGS. 8(b) and 8(c) by the above-mentioned processing.

[0078] In the case of character input using the electronic pen 30, when the non-contact state is changed to the contact state, the pen tip 31 of the electronic pen 30 slides on the display panel 21, generating dense feature points, but these dense feature points do not constitute the shape of a character, and therefore become noise in the data for character recognition. Therefore, the stroke data processing unit 1141 performs the above-mentioned thinning process on the input stroke data to remove noise and improve the learning accuracy and recognition accuracy.

[0079] Fig. 9(a) is an image showing the first normalized stroke data after the thinning process, and Fig. 9(b) shows the first normalized stroke data after the thinning process.

[0080] In the first normalized stroke data shown in Fig. 9, the number of feature points has been thinned from 22 before the thinning process to 16. As shown in Fig. 9, when thinning out feature points from the first normalized stroke data, the stroke data processing unit 1141 renumbers the time series numbers so that there are no gaps.

[0081] 9, the feature points of time series numbers 1 to 9 are feature points of the first stroke, and the feature points of time series numbers 10 to 16 are feature points of the second stroke. The stroke data processing unit 1141 manages a list of feature points of each stroke before and after thinning.

[0082] Next, the stroke data processing unit 1141 generates data (hereinafter referred to as "second normalized stroke data") by performing a process of interpolating feature points (hereinafter referred to as "feature point interpolation process") based on the thinned-out first normalized stroke data so that the distance between each feature point is equal to or less than a predetermined value, including non-contact sections (sections between each stroke) (S104).

[0083] For example, the following formula (8) may be calculated based on a feature point of a time series t and a feature point of the next time series t+1, and if the formula holds, a new feature point (a feature point of a time series between two feature points) may be interpolated (added) to a position between the two feature points (for example, an intermediate position). {x(t+1)-x(t)} 2 +{y(t+1)-y(t)} 2 >SIZE / 10 …(8)

[0084] Here, we will explain the case where SIZE / 100=100 / 10=10 is applied as the threshold value (right side of equation (8)) for determining whether or not to thin out sample positions. However, this threshold value may be set to an arbitrarily designed value (for example, by searching for and setting a suitable value through experiments, etc.).

[0085] Here, the stroke data processing unit 1141 performs calculations by applying formula (8) to the sample positions of each time series, and if the formula holds, interpolates a new feature point between the sample positions and the next time series.

[0086] In this case, the x-coordinate of the newly added feature point may be set to "{x(t+1)+x(t)} / 2" (i.e., the average value of x(t+1) and x(t)), and the y-coordinate may be set to "{y(t+1)+y(t)} / 2" (i.e., the average value of y(t+1) and y(t)).

[0087] The stroke data processing unit 1141 recursively performs feature point interpolation processing for each stroke until equation (8) is no longer satisfied for all sample positions in the time series. For example, when the stroke data processing unit 1141 interpolates a new feature point between a feature point in time series t and a feature point in time series t+1, the stroke data processing unit 1141 may perform calculations by applying equation (8) to the time series of the added feature point as t+1, and if equation (8) is satisfied, repeat the process of interpolating the new feature point.

[0088] For example, the stroke data processing unit 1141 extracts the data of the first stroke (data of time series numbers 1 to 13) from the first normalized stroke data shown in FIG. 8(c) and performs the above-mentioned feature point interpolation process, and further extracts the data of the second stroke (data of time series numbers 14 to 22) and performs the above-mentioned feature point interpolation process.

[0089] The stroke data processing unit 1141 also performs the above feature point interpolation process between the strokes to fill in the gaps with feature points at a predetermined interval. For example, the stroke data processing unit 1141 performs the above feature point interpolation process between the last time-series feature point of the first stroke and the first time-series feature point of the second stroke, thereby filling in the gaps between the end of the first stroke and the beginning of the second stroke with feature points at a predetermined interval. In other words, the stroke data processing unit 1141 performs feature point addition process on the gap between the end of the first stroke and the beginning of the second stroke as one stroke.

[0090] Furthermore, the stroke data processing unit 1141 assigns information on the pen tip state to each feature point. Specifically, the stroke data processing unit 1141 assigns "1" representing the contact state to the pen tip information of the feature points in the contact state (feature points belonging to each stroke), and assigns "0" representing the non-contact state to the pen tip information of the feature points in the non-contact state (feature points in the section between strokes).

[0091] FIG. 10 is a diagram showing a process of performing the above-mentioned feature point interpolation process on the thinned-out first normalized stroke data shown in FIG. 9 to obtain second normalized stroke data.

[0092] In Fig. 10, feature points with time series numbers 1 to 40 are set as the first stroke, and feature points with time series numbers 63 to 102 are set as the second stroke. In Fig. 10, feature points with time series numbers 41 to 62 between the first and second strokes are set as a non-contact state section.

[0093] Next, the stroke data processing unit 1141 generates data (hereinafter referred to as "third normalized stroke data") by extracting N+1 feature points from the second normalized stroke data (S105).

[0094] Here, the number of feature points in the second normalized stroke data is represented as "C."

[0095] If C>N+1, the stroke data processing unit 1141 extracts (selects) N+1 feature points from the second normalized stroke data to generate third normalized stroke data. Furthermore, when N+1>C, the stroke data processing unit 1141 selects some or all of the feature points of the second normalized stroke data multiple times to extract data of N+1 feature points and generate third normalized stroke data. It is desirable that the third normalized stroke data have as little bias in the distances between the feature points as possible. The stroke data processing unit 1141 is not limited to a particular method for extracting N+1 feature points from the second normalized stroke data, but it is possible to reduce bias in the distances between the feature points in the third normalized stroke data by performing the following process, for example.

[0096] Here, the stroke data processing unit 1141 selects the D(i)th time-series feature point of the second normalized stroke data as the i-th feature point (i is an integer from 1 to N) of the third normalized stroke data. For example, the following formula (9) can be applied as D(i). In other words, D(i) is an integer obtained by rounding down the decimal point of C / (N+1) multiplied by i. D(i) = [{C / (N + 1)} * (i ― 1)] + 1 …(9)

[0097] FIG. 11 is a diagram showing an example of third normalized stroke data obtained as a result of extracting N + 1 feature points from the second normalized stroke data shown in FIG. 10.

[0098] FIG. 12 is a diagram showing the third normalized stroke data shown in FIG. 11 in the form of an image.

[0099] For example, since the second normalized stroke data shown in FIG. 10 is composed of 102 feature points, 101 points are extracted from this using equation (9). For example, D(1) = 1, D(2) = 2, ···, D(99) = 99, D(100) = 100, D(101) = 102, so only the 101st feature point of the second normalized stroke data is not selected (extracted) (skipped).

[0100] Next, an example where N + 1 < C will be described. Assuming C = 30, it becomes D(1) = 1, D(2) = 1, D(2) = 1, D(3) = 1, D(4) = 2, ···, D(99) = 29, D(100) = 29, D(101) = 30.

[0101] By using equation (9) as described above, the i-th feature point of the third normalized stroke data can be efficiently picked up from the second normalized stroke data.

[0102] Next, the stroke data processing unit 1141 converts the second normalized stroke data into input vector data and acquires it (S105).

[0103] The stroke data processing unit 1141 sets the X coordinate, Y coordinate, and pen tip state of each time series constituting the second normalized stroke data to VX, VY, and VT of each time series, respectively. Then, the stroke data processing unit 1141 can obtain VR(t), VU(t), VL(t), and VD(t) based on VX(t), VX(t+1), VY(t), and VY(t+1) as described above. In this way, the stroke data processing unit 1141 can obtain V(1) to V(100).

[0104] FIG. 13 shows an example of input vector data acquired based on the second normalized stroke data shown in FIG.

[0105] Next, a process in which the stroke data processing unit 1141 generates input image data for offline character recognition processing from input stroke data will be described with reference to the flowchart of FIG.

[0106] First, it is assumed that the stroke data processing unit 1141 holds input stroke data for one character (S201).

[0107] Next, the stroke data processing unit 1141 obtains data (hereinafter referred to as "fourth normalized stroke data") normalized so that the input stroke data fits exactly within a normalized area of ​​a predetermined resolution (S202).

[0108] FIG. 15 is a diagram showing normalization processing in the process in which the stroke data processing unit 1141 generates input vector data for online character recognition processing.

[0109] FIG. 15(a) is an image obtained by cutting out only the area where the feature points (sample positions) are drawn from the input stroke data shown in FIG.

[0110] FIG. 15(b) shows an image obtained by converting the image in FIG. 15(a) into a normalized region of 64 pixels×64 pixels (a region with an aspect ratio of 1:1).

[0111] FIG. 15(c) is a diagram showing normalized stroke data (fourth normalized stroke data) corresponding to each feature point (each pixel) of the image in the normalized region of FIG. 15(b).

[0112] As shown in FIG. 15(b) and FIG. 15(c), the stroke data processing unit 1141 normalizes the input stroke data to a normalized area of ​​64 pixels×64 pixels by providing two-pixel margins at the top, bottom, left, and right edges. That is, the stroke data processing unit 1141 essentially performs processing to normalize the input stroke data to an area of ​​60 pixels×60 pixels. In the example of FIG. 15, the stroke data processing unit 1141 obtains the coordinates of each feature point after performing a resolution conversion process to convert the image of the input stroke data (an image of 208 pixels×270 pixels) to an image of 60 pixels×60 pixels (an image with an aspect ratio of 1:1), thereby obtaining the fourth normalized stroke data shown in FIG. 15(c). At this time, various image processing methods can be applied to the specific method of the resolution conversion process performed by the stroke data processing unit 1141, so a description of the specific process will be omitted.

[0113] Next, the stroke data processing unit 1141 performs processing to thin out feature points from the fourth normalized stroke data so that the distance between each feature point and its adjacent feature point becomes equal to or greater than a predetermined distance (S203).

[0114] The stroke data processing unit 1141 may apply a process for thinning out feature points from the fourth normalized stroke data in a manner similar to the process for thinning out feature points from the first normalized stroke data described above. For example, the stroke data processing unit 1141 may repeat the thinning process for each stroke until equation (7) is no longer satisfied for all time-series feature points (because the distance between all feature points is equal to or greater than a predetermined value). In this case, it is preferable that the stroke data processing unit 1141 sets the SIZE when applying equation (7) to 64, which is the same as the resolution of the fourth normalized stroke data.

[0115] Next, the stroke data processing unit 1141 acquires input image data based on the fourth normalized stroke data after the feature points have been thinned out (S204).

[0116] For example, the stroke data processing unit 1141 may obtain data on feature points of each stroke from the fourth normalized stroke data after thinning processing in an image area of ​​64 pixels x 64 pixels, and obtain input image data by drawing lines connecting the feature points of each stroke in the above image area.

[0117] FIG. 16 is a diagram showing an image of the input image data obtained based on the fourth normalized stroke data shown in FIG.

[0118] Next, the process performed by the character recognition processing unit 114 in the learning mode will be described with reference to FIG.

[0119] Here, it is assumed that when the character recognition processing unit 114 is operating in the learning mode, the content processing unit 111 supplies the character recognition processing unit 114 with input stroke data for learning (input stroke data for one character) and a set of correct labels corresponding to the characters in the input stroke data.

[0120] First, the stroke data processing unit 1141 generates input vector data and input image data based on the supplied input stroke data, and supplies them to the online AI processing unit 1142 and the offline AI processing unit 1143, respectively (S301).

[0121] In the online AI processing unit 1142 of the character recognition processing unit 114 operating in the learning mode, learning processing is carried out based on the supplied input vector data and correct answer labels (S302).

[0122] Furthermore, in the offline AI processing unit 1143 of the character recognition processing unit 114 operating in the learning mode, a learning process is carried out based on the supplied input image data and the correct label (S303).

[0123] As described above, every time learning data is supplied from content processing unit 111 to character recognition processing unit 114, learning processing is performed using the learning data set.

[0124] Next, the process performed by the character recognition processor 114 in the recognition process mode will be described with reference to FIG.

[0125] Here, it is assumed that input stroke data for learning (input stroke data for one character) is supplied from content processing unit 111 to character recognition processing unit 114 when character recognition processing unit 114 is operating in the character recognition mode.

[0126] First, the stroke data processing unit 1141 generates input vector data and input image data based on the supplied input stroke data, and supplies them to the online AI processing unit 1142 and the offline AI processing unit 1143, respectively (S401).

[0127] The online AI processing unit 1142 of the character recognition processing unit 114 operating in character recognition mode performs character determination processing using the stored learning model based on the supplied input vector data, and supplies the determination result (online determination result) together with the reliability to the character recognition result output unit 1144 (S402).

[0128] The offline AI processing unit 1143 of the character recognition processing unit 114 operating in character recognition mode performs character determination processing using the stored learning model based on the supplied input image data, and supplies the determination result (offline determination result) together with the reliability to the character recognition result output unit 1144 (S403).

[0129] Next, the character recognition result output unit 1144 compares the reliability of the online determination result with that of the offline determination result, selects the determination result with the greater reliability (S404), and outputs it (S405).

[0130] As described above, the character recognition processing unit 114 outputs a determination result every time input stroke data is supplied from the content processing unit 111.

[0131] (A-3) Effects of the First Embodiment According to this embodiment, the following effects can be obtained.

[0132] (A-3-1) First, the inventor actually constructed the character recognition processing unit 114 of the recognition processing system 1 and conducted an experiment (hereinafter referred to as "this experiment") on the character recognition accuracy (correct rate of the judgment result) when learning processing and character recognition processing were performed. The contents and results of this experiment are described below.

[0133] In this experiment, learning and recognition processes were performed using "katakana," "hiragana," and "JIS level 1 kanji" characters (a total of 3107 types of characters) as sample fonts (hereinafter referred to as "sample fonts"). In this experiment, 160 samples of input stroke data (input stroke data when a human inputs into the pen tablet 20 using the electronic pen 30) and correct answer labels were prepared for each character of the sample font, and supplied to the character recognition processing unit 114 operating in the learning mode. As a result, the process of the flowchart in FIG. 17 is performed for each sample of each sample font, and a learning model is obtained in the online AI processing unit 1142 and the offline AI processing unit 1143, respectively.

[0134] In this experiment, in order to confirm the recognition accuracy after the learning process is completed in the character recognition processing unit 114, 160 samples of input stroke data and correct answer labels were prepared for each character of the sample font and supplied to the character recognition processing unit 114 operating in the recognition processing mode. As a result, the recognition process of the flowchart in FIG. 18 was performed for each sample of each sample font. At this time, the inventor confirmed the accuracy rate of each of the online judgment result and the offline judgment result, and the accuracy rate of the judgment result output from the character recognition result output unit 1144 (a result of comprehensively judging the online judgment result and the offline judgment result). As a result of this experiment, the accuracy rate of the online judgment result alone was about 94%, and the accuracy rate of the offline judgment result alone was about 90%. And the judgment result output from the character recognition result output unit 1144 was about 98%. In other words, it was found that the recognition accuracy is higher when the final judgment result is output taking into account both the online judgment result and the offline judgment result.

[0135] (A-3-2) In the information processing terminal 10 (character recognition processing unit 114) of the first embodiment, input vector data used in the online character recognition learning process and character recognition process is normalized and processed so that all of the input vector data has N feature points. This allows the information processing terminal 10 (character recognition processing unit 114) of the first embodiment to generate and process fixed-length input vector data for all characters, regardless of the number of strokes, etc. This is because, in general, in machine learning processing using a neural network, it is desirable to input fixed-length data. It is possible to consider using an AI engine that supports variable-length input layers, or configuring a neural network with a fixed-length input layer that matches the longest data length (effectively using a fixed-length input layer with variable length), but there is a risk that processing efficiency and recognition accuracy will be unstable compared to when the process is completed with fixed-length input and output.

[0136] (A-3-3) In the information processing terminal 10 (character recognition processing unit 114) of the first embodiment, not only coordinates (VX, VY) but also parameters related to the motion vector (VR, VU, VL, VD) and the state of the pen tip of the electronic pen 30 (contact state or non-contact state) are introduced as feature quantities constituting the input vector data. As a result, in the information processing terminal 10 (character recognition processing unit 114) of the first embodiment, information on strokes while the pen tip of the electronic pen 30 is in a non-contact state is also acquired as feature quantities. In addition, in the information processing terminal 10 (character recognition processing unit 114) of the first embodiment, it is not necessary to prepare a standard pattern for each character in advance as in the technology described in Patent Document 1, or to perform processing such as matching feature points with all standard patterns during character recognition. As described above, in the information processing terminal 10 (character recognition processing unit 114) of the first embodiment, efficient character recognition processing can be performed while increasing the amount of information acquired about strokes during character input.

[0137] (B-1) Second embodiment Hereinafter, a second embodiment of a recognition processing device, a recognition processing program, a recognition processing method, and a recognition processing system according to the present invention will be described in detail with reference to the drawings. In this embodiment, an example in which an information processing terminal is configured as the recognition processing device of the present invention will be described.

[0138] Characters according to the second embodiment can also be shown using Fig. 1. Note that the reference characters in parentheses in Fig. 1 are used only in the second embodiment.

[0139] The following describes the differences between the characters in the second embodiment and the first embodiment.

[0140] The recognition processing system 1A of the second embodiment differs from the first embodiment in that the information processing terminal 10 is replaced with an information processing terminal 10A. Also, in the information processing terminal 10A of the second embodiment, the control unit 11 is replaced with a control unit 11A. Furthermore, in the control unit 11A of the second embodiment, the content processing unit 111 and the character recognition processing unit 114 are replaced with a content processing unit 111A and a character recognition processing unit 114A, respectively. Furthermore, in the character recognition processing unit 114A of the second embodiment, the stroke data processing unit 1141 is replaced with a stroke data processing unit 1141A, which is different from the first embodiment.

[0141] Incidentally, in the first embodiment, the pen tip state of the electronic pen 30 has been described as being either a contact state in which the pen tip 31 of the electronic pen 30 is in contact with the display panel 21 of the pen tablet 20, or a non-contact state in which the pen tip 31 of the electronic pen 30 is not in contact with the display panel 21 of the pen tablet 20. However, depending on the combination of the device applied to the pen tablet 20 and the electronic pen 30, there are some combinations that allow tracking of the lateral position of the pen tip 31 of the electronic pen 30 even in the non-contact state. For example, when applying a combination of a pen tablet and a stylus pen manufactured by Wacom (registered trademark), even if the stylus pen is in the non-contact state, the position (lateral position) of the pen tip can be tracked on the pen tablet as long as the height of the pen tip is equal to or less than a predetermined value.

[0142] Therefore, in this embodiment, in the tablet 20, even when the pen tip 31 of the electronic pen 30 is in a non-contact state, if the height of the pen tip 31 is below a predetermined value, it is assumed that the lateral position (coordinates) of the pen tip 31 can be detected. In the second embodiment, when the pen tip state of the electronic pen 30 is in a non-contact state and the tablet 20 can track the lateral position of the pen tip 31, that state (pen tip state) is referred to as the "hover state". Also, in the second embodiment, when the pen tip state of the electronic pen 30 is in a non-contact state and the tablet 20 cannot track the lateral position of the pen tip 31, that state (pen tip state) is referred to as the "lost state".

[0143] FIG. 19 and FIG. 20 are timing charts showing the height and pen tip state of the pen tip 31 for each time series when the user uses the electronic pen 30 to draw a Chinese character (for example, "eight", etc.) with two strokes on the tablet 20.

[0144] In FIG. 19, the horizontal axis represents time t, and the vertical axis represents the height (height for each time series) of the pen tip 31 of the electronic pen 30. In FIG. 19, the position of the pen tip 31 at each time from t0 to t21 is represented by the position of the tip of a wedge-shaped (wedge with the tip facing downward) symbol. Here, it is explained that t0 to t21 respectively indicate the timing for detecting (sampling) coordinates and the like of the electronic pen 30 (pen tip 31) on the tablet 20.

[0145] Also, in FIG. 19, the symbol at the time when the pen tip 31 is in the contact state is colored black, the symbol at the time when the pen tip 31 is in the hover state is hatched (diagonal lines), and the outline of the symbol at the time when the pen tip 31 is in the lost state is a dashed line.

[0146] 19, times t0 to t4 indicate drawing the first stroke, and the pen tip 31 of the electronic pen 30 is in a contact state. During the contact state, the coordinates and writing pressure of the electronic pen 30 (pen tip 31) are acquired by a sensor in the pen tablet 20.

[0147] In Fig. 19, from time t5 to t7, the pen tip 31 of the electronic pen 30 is in a hover state after the first stroke has been drawn. As described above, with a pen tablet manufactured by Wacom, it is possible to acquire coordinates and detect the presence of a stylus pen even when the stylus pen is floating a certain distance above the panel of the pen tablet. In the case of a hover state, a pen pressure value of "0" (that is, a value indicating a hover) is acquired with a pen tablet manufactured by Wacom.

[0148] In FIG. 19, from time t8 to time t11, the user further removes the pen tip 31 of the electronic pen 30 from the pen tablet 20, and the electronic pen 30 (pen tip 31) is in the loss state. During the loss state, the pen tablet 20 cannot detect the coordinates of the electronic pen 30 (pen tip 31).

[0149] 19, from time t12 to t13, the user again approaches the electronic pen 30 (pen tip 31) to the pen tablet 20 in order to draw a second stroke, resulting in a hover state. Then, from the following time t14 to t19, the user starts drawing a second stroke, so the electronic pen 30 (pen tip 31) comes into contact with the pen tablet 20, resulting in a contact state.

[0150] The example in FIG. 20 is the same as the example in FIG. 19, except that the timings t8 to t12 are in a hover state rather than a loss state.

[0151] FIG. 21 shows the aggregation result of the pen tip states at each sample (feature point) shown in the timing chart of FIG. 19. As shown in FIG. 21, in the example of FIG. 19, during the time when 20 samples have elapsed, 16 samples (coordinates of the electronic pen 30) are obtained excluding the period in the loss state (times t8 to t11). Also, as shown in FIG. 21, in the example of FIG. 19, out of the 16 obtained samples, 11 samples are in the contact state and 5 samples are in the hover state.

[0152] FIG. 22 is a diagram showing each sample (feature point) shown in the timing chart of FIG. 19. FIG. 22 shows the case where the user draws the Chinese character '八'.

[0153] In FIG. 22, the feature points in the contact state are indicated by circular (○) symbols, the feature points in the hover state are indicated by triangular (△) symbols, and the positions in the loss state are indicated by square (□) symbols. Note that in the case of the loss state, the coordinates of the feature points cannot be obtained by the tablet 20, but in FIG. 22, the position is illustrated by a square symbol assuming that the coordinates (the horizontal position of the pen tip 31) could be obtained. Also, hereinafter, the x - coordinates of the feature points from t0 to t20 are represented as x0 to x20, and the y - coordinates are represented as y0 to y20.

[0154] As described above, in the tablet 20 of the second embodiment, it is assumed that it is possible to detect any one of the contact state, the hover state, and the loss state for the electronic pen 30.

[0155] In the information processing terminal 10 (control unit 11) of the first embodiment, input stroke data consisting of only samples (coordinates) of the contact state is normalized to generate input vector data for the online AI processing unit 1142. In contrast, the information processing terminal 10A (control unit 11A) of the second embodiment differs from the second embodiment in that the input stroke data for the online AI processing unit 1142 can reflect three statuses, namely, the contact state, the hover state, and the loss state. In the second embodiment, the configuration and normalization process of the input stroke data for the offline AI processing unit 1143 can be applied in the same manner as in the first embodiment, and therefore a description thereof will be omitted here.

[0156] Next, an example of a method for configuring and normalizing input stroke data for the online AI processing unit 1142 in the second embodiment will be described. In the second embodiment, the following five methods, for example, can be used as a method for configuring and normalizing input stroke data for the online AI processing unit 1142.

[0157] [First normalization method] In the first normalization method, input stroke data is constructed using only the feature points (coordinates) of only the contact state, and normalized to N+1 feature points. That is, in the example of Fig. 19 and Fig. 21, the input stroke data is expressed using a total of nine sample feature points from t0 to t4 and t14 to t19. In this case, assuming that normalization is performed with N=100, the feature points of the input stroke data are upsampled by about 11 times. In the first normalization method, as in the first embodiment, the input stroke data is processed (thinning out and interpolating feature points) for each stroke (section in which the pen tip state is in contact) and normalized to obtain input vector data.

[0158] [Second normalization method] In the second normalization method, input stroke data is constructed by extracting and adding a portion (e.g., only one sample) of the hover state samples to the contact state samples (feature points), and normalized. In the second normalization method, for example, from the hover state samples, a sample at either the timing before or after the period of the loss state (e.g., the sample at time t7 or t12 in the example of FIG. 19) may be extracted and added to the input stroke data, or samples both before and after the period of the loss state (e.g., the samples at times t7 and t12 in the example of FIG. 19) may be extracted and added to the input stroke data.

[0159] FIG. 23 is a diagram showing an example of input stroke data when the second normalization method is applied to each sample (feature point) shown in the timing chart of FIG. Figure 23 shows an example of input stroke data constructed by adding samples from both before and after the period of the loss state (samples at times t7 and t12) to the contact state sample. In Figure 23, the hover state is represented by "0" in the pen tip state item.

[0160] In the stroke data processing unit 1141A, when the second normalization method is applied, the section (one section consisting of the hover state and the loss state) between strokes (between contact state sections) may also be normalized in the same way as for strokes (contact state section) to obtain input vector data. For example, in the case of input stroke data as shown in FIG. 23, the stroke data processing unit 1141A may perform normalization processing in the same way as for one stroke for the section between the last feature point of the first stroke and the first feature point of the second stroke.

[0161] In the stroke data processing unit 1141A, when the second normalization method is applied, when feature points are interpolated during a non-contact state period (hover state and loss state period) or immediately before or after the non-contact state period, the pen tip state of those feature points in the input vector data is set to "0".

[0162] [Third normalization method] In the third normalization method, input stroke data is normalized by adding all hover state samples to the contact state samples. For example, in the examples of Figures 19 and 21, 16 samples of the contact state or hover state may be included in the input stroke data.

[0163] In the stroke data processing unit 1141A, when the third normalization method is applied, the section between strokes (between contact state sections) (a section consisting of a hover state and a loss state) may also be normalized in the same way as the strokes (contact state sections) to obtain input vector data.

[0164] Fig. 24 is a diagram showing an example of input stroke data when the third normalization method is applied to each feature point shown in the timing chart of Fig. 19. In Fig. 24, the pen tip state in the hover state is set to "0".

[0165] In the stroke data processing unit 1141A, when the second normalization method is applied, the section (one section consisting of the hover state and the loss state) between strokes (between contact state sections) may also be normalized in the same way as for strokes (contact state section) to obtain input vector data. For example, in the case of input stroke data as shown in FIG. 23, the stroke data processing unit 1141A may perform normalization processing in the same way as for one stroke for the section between the last feature point of the first stroke and the first feature point of the second stroke.

[0166] In the third normalization method, when feature points are interpolated during a hover state period and before and after a hover state period, the pen tip states of those feature points in the input vector data are set to "0." In the stroke data processing unit 1141A, when the third normalization method is applied, when feature points are interpolated during a non-contact state period (hover state and loss state period) or immediately before or after the non-contact state period, the pen tip states of those feature points in the input vector data are set to "0."

[0167] [The fourth normalization method] In the fourth normalization method, the input stroke data is normalized by configuring all samples (contact state, hover state, and loss state). For example, in the examples of Figures 19 and 21, all 20 samples may be included in the input stroke data.

[0168] FIG. 25 is a diagram showing an example of input stroke data when the fourth normalization method is applied to each feature point shown in the timing chart of FIG.

[0169] In FIG. 25, the pen tip state in the hover state is set to "0", and the pen tip state in the loss state is set to "2". In FIG. 25, the x coordinates of times t8 to t11 in the loss state are illustrated as c_x8 to c_x11, and the y coordinates are illustrated as c_y8 to c_y11. In FIG. 25, each coordinate (X coordinate and Y coordinate) of times t8 to t11 in the loss state may be set to the coordinates of a position interpolated (linearly interpolated) between the coordinates of the feature points of the previous and next hover periods. In the example of FIG. 25, each coordinate of t8 to t11 may be set at equal intervals on a line connecting the coordinates (x7, y7) of t7 immediately before the loss state period and the coordinates (x12, y12) immediately after the loss state period.

[0170] When the fourth normalization method is applied, the stroke data processing unit 1141A performs normalization processing (thinning out and interpolating feature points) on the hover state section and the loss state section in the same manner as the image section (contact state section) to acquire input vector data. For example, in the case of input stroke data as shown in FIG. 25, the stroke data processing unit 1141A may perform normalization processing (thinning out and interpolating feature points) on the hover state section from time t5 to t7, the loss state section from time t8 to t11, and the hover section from time t12 to t14 in the same manner as the image section (contact state section) to connect them.

[0171] In addition, in the fourth normalization method, when feature points are interpolated during a hover state period or a loss state period, the value of the pen tip state on the input vector data may be set according to the following rule. In the fourth normalization method, when feature points are interpolated during a loss state period and immediately before or immediately after a loss state period, the pen tip state of those feature points may be set to "2" (loss state) in the input vector data. Furthermore, in the fourth normalization method, when feature points are interpolated during a hover state period immediately before a loss state period and immediately before the hover state period, the pen tip state of those feature points may be set to "0" (hover state) in the input vector data. In addition, in the fourth normalization method, when feature points are interpolated during a hover state period immediately after a loss state period and immediately after the hover state period, the pen tip state of those feature points may be set to "0" (hover state) in the input vector data.

[0172] As described above, when the fourth normalization method is applied, in the example of the timing chart in FIG. 19, input stroke data is generated using all samples (20 samples) including samples in the loss state, so that during normalization, only about five times the feature point interpolation (dilution) is required, and more accurate information (more information amount) can be included in the input vector data. In other words, when the fourth normalization method is applied, in the example of the timing chart in FIG. 19, information distinguishing between the hover state and the loss state can be included in the input vector data. As a result, the fourth normalization method can increase the amount of information in the input vector data, which can improve the recognition accuracy depending on the learning environment and recognition environment.

[0173] [Fifth normalization method] In the fifth normalization method, input stroke data is constructed and normalized for all samples (contact state, hover state, and loss state) in the same way as in the fourth normalization method. However, the fifth normalization method differs from the fourth normalization method in that the loss state and hover state are treated uniformly. Here, the fifth normalization method will be described assuming that the loss state and hover state are all treated as the hover state.

[0174] FIG. 26 is a diagram showing an example of input stroke data when the fifth normalization method is applied to each feature point shown in the timing chart of FIG.

[0175] FIG. 26 differs from FIG. 25 (fourth normalization method) in that the pen tip state during the loss state is also set to “0” like the hover state.

[0176] In the fifth normalization method, when feature points are interpolated during a period consisting of a hover state and a loss period (in FIG. 19, the period from time t5 to t13) and immediately before or after that period, the pen tip states of those feature points in the input vector data are set to “0”.

[0177] As a result, in the fifth normalization method, even if the height (detection range) at which coordinates can be obtained in a hover state varies due to differences in the capabilities of the sensor (sensor that detects the electronic pen 30) of the pen tablet 20 and the state of Figure 19 and the state of Figure 20 are mixed, the contents of the input vector data will be approximately the same, making it possible to perform stable learning and recognition processes.

[0178] (C) Other embodiments The present invention is not limited to the above-described embodiments, and may include modified embodiments such as those exemplified below.

[0179] (C-1) In each of the above embodiments, the information processing terminal 10 and the pen tablet 20 (display panel 21) are configured as separate devices, but they may be configured as an integrated device. For example, the information processing terminal 10 may be configured using a computer (for example, a tablet terminal or a smartphone) equipped with a touch panel display.

[0180] (C-2) In each of the above embodiments, the character recognition processing unit 114 includes both the online AI processing unit 1142 and the offline AI processing unit 1143, but may also be configured to include only the online AI processing unit 1142. [Explanation of symbols]

[0181] 1...recognition processing system, 10...information processing terminal, 11...control unit, 111...content processing unit, 112...display driver, 113...pen tablet driver, 114...character recognition processing unit, 1141...stroke data processing unit, 1142...online AI processing unit, 1143...offline AI processing unit, 1144...character recognition result output unit, 12...video IF, 13...USB port, 20...pen tablet, 21...display panel, 30...electronic pen, 31...pen tip.

Claims

1. a normalization means for acquiring input stroke data indicating an input pattern of strokes made by an electronic pen in a time series order for each input character, the input stroke data including position information of feature points in the time series order of the input pattern, and performing a normalization process for normalizing the acquired input stroke data into an input pattern having a fixed number of samples to acquire normalized stroke data; an input vector data acquisition means for converting the normalized stroke data normalized by the normalization means into input vector data expressed by the feature amount of the fixed number of samples; a character recognition processing means for performing character recognition processing on the input vector data acquired by the input vector data acquisition means, using a learning model that has been machine-learned using the input vector data acquired by the input vector data acquisition means, The normalization process performed by the normalization means includes the following steps: a first normalization process for thinning out feature points from the input stroke data so that intervals between all feature points adjacent in time series are equal to or greater than a predetermined threshold value, thereby acquiring intermediate stroke data; and a second normalization process for acquiring data obtained by interpolating feature points of the intermediate stroke data such that intervals between all feature points adjacent in chronological order are equal to or smaller than the threshold value.

2. A recognition processing device comprising:

2. The recognition processing device according to claim 1, characterized in that each feature constituting the input vector data includes a coordinate parameter indicating a coordinate corresponding to a time series of the feature, a motion vector parameter indicating a motion vector from a time series immediately preceding the feature, and a pen tip state parameter indicating a state of the pen tip of the electronic pen corresponding to the time series of the feature.

3. 3. The recognition processing apparatus according to claim 2, wherein each of the feature amounts constituting the input vector data includes a pen tip state parameter indicating a state of the pen tip of the electronic pen corresponding to a time series of the feature amount.

4. The input vector data acquisition means further generates input image data drawn based on the input stroke data, the character recognition processing means performs character recognition processing on the input vector data acquired by the input vector data acquisition means using a first learning model previously trained by machine learning using input vector data to acquire a first character recognition result, and further performs character recognition processing on the input image data acquired by the input vector data acquisition means using a second learning model previously trained by machine learning using an input image to acquire a second character recognition result; The apparatus further includes a character recognition result output means for selecting either the first character recognition result or the second character recognition result by the character recognition processing means and outputting the selected result as a final character recognition processing result.

4. The recognition processing device according to claim 1,

5. the character recognition processing means acquires a reliability of the first character recognition result or the second character recognition result when acquiring the first character recognition result or the second character recognition result; The character recognition result output means selects the one having a higher reliability from the first character recognition result and the second character recognition result by the character recognition processing means, and outputs the selected result as a final character recognition processing result.

5. The recognition processing device according to claim 4.

6. Computer, a normalization means for acquiring input stroke data indicating an input pattern of strokes made by an electronic pen in a time series order for each input character, the input stroke data including position information of feature points in the time series order of the input pattern, and performing a normalization process for normalizing the acquired input stroke data into an input pattern having a fixed number of samples to acquire normalized stroke data; an input vector data acquisition means for converting the normalized stroke data normalized by the normalization means into input vector data expressed by the feature amount of the fixed number of samples; a learning model that has been machine-learned using the input vector data acquired by the input vector data acquisition means, the learning model functioning as a character recognition processing means that performs character recognition processing on the input vector data acquired by the input vector data acquisition means; The normalization process performed by the normalization means includes the following steps: a first normalization process for thinning out feature points from the input stroke data so that intervals between all feature points adjacent in time series are equal to or greater than a predetermined threshold value, thereby acquiring intermediate stroke data; and a second normalization process for acquiring data obtained by interpolating feature points of the intermediate stroke data such that intervals between all feature points adjacent in chronological order are equal to or smaller than the threshold value. A recognition processing program comprising:

7. A recognition processing method performed by a recognition processing device, The recognition processing device includes a normalization means, an input vector data acquisition means, a character recognition processing means, and a character recognition result output means, the normalization means acquires input stroke data indicating an input pattern of strokes made by an electronic pen in a time series order for each input character and including position information of feature points in the time series order of the input pattern, and acquires normalized stroke data by performing a normalization process for normalizing the acquired input stroke data into an input pattern with a fixed number of samples; the input vector data acquisition means converts the normalized stroke data normalized by the normalization means into input vector data expressed by the feature amount of the fixed number of samples; the character recognition processing means performs character recognition processing on the input vector data acquired by the input vector data acquisition means, using a learning model that has been machine-learned using the input vector data acquired by the input vector data acquisition means; The normalization process performed by the normalization means includes the following steps: a first normalization process for thinning out feature points from the input stroke data so that intervals between all feature points adjacent in time series are equal to or greater than a predetermined threshold value, thereby acquiring intermediate stroke data; and a second normalization process step of acquiring data by interpolating feature points of the intermediate stroke data so that intervals between all feature points adjacent in chronological order are equal to or less than the threshold value.

8. A recognition processing system having a pen tablet capable of receiving input using an electronic pen and a front electronic pen, and a recognition processing device that recognizes characters written on the pen tablet by a user with the electronic pen, characterized in that the recognition processing system is characterized in that a recognition processing device described in any one of claims 1 to 5 is applied as the recognition processing device.

Images