Handwriting recognition method and system for mathematical formula

Abstract

The invention discloses a mathematical formula handwriting recognition method and system, and the method comprises the steps: carrying out the feature extraction of a low level, a middle level and a high level of obtained data of a handwritten mathematical formula, extracting detail parameters of the formula, such as stroke attributes, a space structure, an overall structure and a logic relation, and carrying out the formula conversion of an extraction result, and a high-precision handwritten mathematical formula identification result can be obtained. According to the method, hierarchical feature extraction is performed on the handwritten mathematical formula data, and the extracted detail parameters are taken as the identification basis of the complex mathematical formula, so that the identification accuracy of the complex handwritten mathematical formula can be improved, and the identification level of the complex handwritten mathematical formula can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of handwritten mathematical formula recognition, and in particular to a method and system for handwritten mathematical formula recognition. Background Art

[0002] With the increasing development of educational informatization, the use of electronic whiteboards for teaching has become common. However, most existing electronic whiteboards only support the recognition of simple handwritten mathematical formulas, such as fractions, four arithmetic operations and other mathematical formulas. Handwritten mathematical formula recognition technology is the technical basis for electronic whiteboards to recognize handwritten mathematical formulas. Specifically, it converts handwritten mathematical formulas (i.e., handwritten mathematical symbols and structured mathematical expressions) into digital data through computer vision and machine learning methods to achieve automatic recognition and understanding. Among the publicly available mathematical formula handwriting recognition technologies, there are technologies that divide mathematical formulas into individual characters for recognition or directly recognize the formula as a whole.

[0003] However, in further mathematics teaching and research, the existing electronic whiteboard cannot meet the needs of recognizing complex handwritten mathematical formulas. The existing electronic whiteboard recognition technology for handwritten mathematical formulas has problems such as low accuracy and low efficiency when facing complex handwritten mathematical formulas with difficult structure analysis, complex character relationships, and large writing variation.

[0004] Therefore, in order to realize the recognition of complex handwritten mathematical formulas by the electronic whiteboard, it is necessary to improve the recognition level of complex handwritten mathematical formulas. Summary of the invention

[0005] The present invention provides a method and system for handwritten mathematical formula recognition to solve the above technical problems. By performing hierarchical feature extraction on the acquired handwritten mathematical formula data, high-precision handwritten mathematical formula recognition results can be obtained, effectively improving the recognition level of complex handwritten mathematical formulas.

[0006] In order to solve the above technical problems, the present invention provides a mathematical formula handwriting recognition method, comprising the following steps:

[0007] Get handwritten math formula data;

[0008] Acquire formula stroke images based on handwritten mathematical formula data;

[0009] Perform low-level feature extraction based on the formula stroke image, extract the basic attributes of the strokes, and obtain the stroke attribute matrix;

[0010] Based on the stroke attribute matrix, the mid-level feature extraction is performed to extract the geometric shape and topological structure of the symbol and obtain the symbol structure matrix;

[0011] Based on the symbol structure matrix, high-level feature extraction is performed, and the overall structure and logical relationship of the formula are extracted by combining the context information in the handwritten mathematical formula data and the semantic information of the symbol to obtain the formula structure matrix;

[0012] Perform feature matching and symbol recognition based on the formula structure matrix to obtain a symbol recognition matrix;

[0013] Parse and transform mathematical formulas based on symbol recognition matrix to obtain formula representation matrix;

[0014] A graphical representation corresponding to the acquired handwritten mathematical formula data is obtained based on the formula representation matrix.

[0015] Through the above scheme, by performing hierarchical feature extraction on the acquired handwritten mathematical formula data, extracting the formula's stroke attributes, spatial structure, overall structure, logical relationship and other detail parameters, and further converting the extraction results into formulas, a high-precision handwritten mathematical formula recognition result can be obtained. Furthermore, this method can improve the accuracy of complex handwritten mathematical formula recognition by performing hierarchical feature extraction on handwritten mathematical formula data and using the extracted detail parameters as the recognition basis for complex mathematical formulas, thereby effectively improving the recognition level of complex handwritten mathematical formulas.

[0016] Preferably, the process of obtaining the formula stroke image based on the handwritten mathematical formula data is specifically as follows:

[0017] The symbols in the handwritten mathematical formula data are subjected to stroke denoising, stroke segmentation and image format conversion processing to obtain stroke images corresponding to the symbols in the handwritten mathematical formula data.

[0018] In the above scheme, the accuracy of symbol recognition in the mathematical formula data can be improved through intelligent preprocessing methods including but not limited to stroke denoising, stroke segmentation, stroke splicing, etc., and then converted into corresponding stroke images through image format conversion processing for subsequent recognition.

[0019] Preferably, during the hierarchical feature extraction process, if the extraction result does not meet the judgment condition, a corresponding correction is performed, specifically:

[0020] When extracting low-level features, it is determined whether the stroke data is complete. If it is incomplete, the data is completed.

[0021] When extracting mid-level features, it is determined whether the geometric shape of the symbol is correct. If not, the structure is adjusted.

[0022] When extracting high-level features, it is determined whether the contextual information and semantic information of the formula match. If not, semantic correction is performed.

[0023] Preferably, feature matching and symbol recognition are performed based on the formula structure matrix to obtain a symbol recognition matrix, specifically:

[0024] Using a preset database, feature matching and symbol recognition are performed based on a formula structure matrix to obtain a symbol recognition matrix;

[0025] Preferably, the preset database has a symbol template update mechanism, specifically:

[0026] Recognize the symbols in the acquired stroke image, compare the recognized symbols with the existing symbol templates in the preset database, and recognize new symbols;

[0027] Identify the new symbol and obtain its corresponding parameter;

[0028] In response to corresponding parameters of the new symbol, standardize the new symbol to form a symbol template;

[0029] Adds a new symbol template to the database.

[0030] The symbol template update mechanism mentioned in the above scheme is one aspect of the dynamic database update mechanism set by the present invention. In the process of identifying the symbols in the acquired stroke image, a pattern matching algorithm is used to identify the differences between the common symbols in the preset database and the symbols in the stroke image, and detect the appearance of new symbols and changes in user writing habits; the corresponding parameters of the new symbols are obtained by detection, and the new symbol data is standardized and labeled to form a symbol template, and the template is added to the database to improve the efficiency of subsequent matching and recognition, thereby realizing dynamic optimization of the database. On the other hand, the dynamic database update mechanism also includes an update trigger and feedback mechanism, which triggers database updates according to the frequency and accuracy of user input, and displays the update results and recognition results to the user for feedback.

[0031] Furthermore, the above-mentioned use of the preset database performs feature matching and symbol recognition based on the formula structure matrix to obtain a symbol recognition matrix, which is specifically:

[0032] Each symbol in the formula structure matrix is ​​matched with the existing symbol template in the preset database, and the symbol is identified by the pattern matching algorithm to obtain the symbol recognition matrix.

[0033] In the above scheme, each symbol in the formula structure matrix obtained after high-level feature extraction is matched with the symbol template in the database to determine whether the symbol matching degree is high enough. If not, the template is expanded. Each symbol is identified by the pattern matching algorithm, and the identification results are stored in the symbol recognition matrix. The symbol recognition matrix is ​​initialized as the basis for subsequent formula transformation. Each row in the matrix is ​​a recognized symbol and its position in the formula.

[0034] Preferably, the mathematical formula is parsed and converted based on the symbol recognition matrix to obtain the formula representation matrix, and the specific steps are:

[0035] Get the symbol recognition results corresponding to each row of the formula representation matrix, combine the rows in the order of writing, and get the complete mathematical formula corresponding to each row;

[0036] Based on the complete mathematical formula corresponding to each row, the formula is parsed and converted row by row to obtain the standard mathematical representation corresponding to each row and obtain the formula representation matrix.

[0037] In the above scheme, the complete mathematical formula is parsed and converted to obtain a corresponding standard mathematical representation, whose format includes but is not limited to LaTeX format; secondly, in the process of parsing and converting the mathematical formula based on the symbol recognition matrix, it will be determined whether the formula structure is correct, and if not, the structure will be corrected.

[0038] Preferably, the graphical representation corresponding to the acquired handwritten mathematical formula data is obtained based on the formula representation matrix, and the specific steps are:

[0039] Obtain the standard mathematical representation corresponding to each row in the formula representation matrix, perform calculations by row using a mathematical calculation engine, and obtain the mathematical calculation results corresponding to each row;

[0040] In response to the mathematical calculation results corresponding to each row, the drawing engine is used to draw the pictures in accordance with the rows to obtain the graphic representation results corresponding to each row;

[0041] Based on the graphic representation result corresponding to each row, a graphic representation corresponding to the acquired handwritten mathematical formula data is obtained.

[0042] Preferably, the mathematical computing engine includes but is not limited to the SymPy computing engine; preferably, the drawing engine includes but is not limited to the Matplotlib drawing engine;

[0043] Preferably, when the mathematical calculation engine or the drawing engine is used for calculation, it is determined whether the formula is operable, and if it is not operable, the user is prompted to re-enter the formula.

[0044] In order to solve the above technical problems, the present invention also provides a mathematical formula handwriting recognition system, which includes a data processing module, a feature extraction module, a recognition module, a conversion module, and an operation module, specifically:

[0045] A data processing module, including a data acquisition submodule, which is used to acquire handwritten mathematical formula data;

[0046] The feature extraction module includes a low-level feature extraction submodule, a mid-level feature extraction module and a high-level feature extraction submodule, wherein:

[0047] A low-level feature extraction submodule is used to extract low-level features based on the formula stroke image, extract the basic attributes of the strokes, and obtain a stroke attribute matrix;

[0048] The mid-level feature extraction submodule is used to extract mid-level features based on the stroke attribute matrix, extract the geometric shape and topological structure of the symbol, and obtain the symbol structure matrix;

[0049] A high-level feature extraction submodule is used to extract high-level features based on the symbol structure matrix, and extract the overall structure and logical relationship of the formula by combining the context information in the acquired handwritten mathematical formula data and the semantic information of the symbol to obtain the formula structure matrix;

[0050] A recognition module is used to perform feature matching and symbol recognition based on the formula structure matrix and a preset database to obtain a symbol recognition matrix;

[0051] A conversion module, used for parsing and converting mathematical formulas based on the symbol recognition matrix to obtain a formula representation matrix;

[0052] The operation module is used to obtain the graphical representation corresponding to the acquired handwritten mathematical formula data based on the formula representation matrix.

[0053] In the above scheme, the present invention provides a mathematical formula handwriting recognition system, which extracts hierarchical features from the acquired handwritten mathematical formula data, gradually extracts the detailed parameters of the formula, and further transforms the formula to obtain a high-precision handwritten mathematical formula recognition result. Furthermore, using the high-precision handwritten mathematical formula recognition result, the system can achieve accurate recognition of complex handwritten mathematical formulas, effectively improving the system's recognition level for complex handwritten mathematical formulas.

[0054] Preferably, the basic attributes of the strokes extracted by the low-level feature extraction submodule include but are not limited to length, angle, order and other attributes. The role of the order is to record the mathematical order of each stroke to help identify the overall structure of symbols and formulas.

[0055] Preferably, the geometric shape and topological structure of the symbol extracted by the mid-level feature extraction submodule include but are not limited to properties such as intersections and closed areas. The geometric shape and topological structure of the graphic are identified using a graphic analysis algorithm, and a symbol topological map is constructed to facilitate subsequent identification.

[0056] Preferably, the overall structure and logical relationship of the formula extracted by the high-level feature extraction submodule include but are not limited to context information and semantic information. The grammatical and semantic relationships in the formula are analyzed using natural language processing (NLP) technology to construct a formula logical structure diagram for subsequent identification.

[0057] Preferably, the data processing module further includes a preprocessing submodule, specifically:

[0058] The preprocessing submodule is used to perform stroke denoising, stroke segmentation and image format conversion processing on the symbols in the handwritten mathematical formula data, and obtain the stroke images corresponding to the symbols in the handwritten mathematical formula data.

[0059] Furthermore, the mathematical formula handwriting recognition system provided by the present invention further includes a database module for establishing and updating a preset database using a symbol template update mechanism, specifically:

[0060] Recognize the symbols in the acquired stroke image, compare the recognized symbols with the existing symbol templates in the preset database, and recognize new symbols;

[0061] Identify the new symbol and obtain its corresponding parameter;

[0062] In response to corresponding parameters of the new symbol, standardize the new symbol to form a symbol template;

[0063] Adds a new symbol template to the database.

[0064] In the above scheme, the database module recognizes the symbols in the acquired stroke image, and in the process of comparing the symbols with the symbols in the preset database to obtain new symbols, user factors such as changes in user writing habits will be considered to improve the accuracy of the system in recognizing new symbols. Secondly, the new symbols are standardized and added to the database as templates, which is a dynamic update of the database. Secondly, the database module will trigger the database update according to the frequency and accuracy of user input, and feedback the update effect and recognition results to the user.

[0065] Furthermore, the above recognition module is used to perform feature matching and symbol recognition based on the formula structure matrix and a preset database to obtain a symbol recognition matrix, specifically:

[0066] Each symbol in the formula structure matrix is ​​matched with the existing symbol template in the preset database, and the symbol is identified by the pattern matching algorithm to obtain the symbol recognition matrix.

[0067] Preferably, the conversion module includes a formula combination submodule and a formula conversion submodule, wherein:

[0068] The formula combination submodule is used to obtain the symbol recognition results corresponding to each row of the symbol recognition matrix, and combine them according to the writing order to obtain the complete mathematical formula corresponding to each row;

[0069] The formula conversion submodule is used to parse and convert the complete mathematical formula corresponding to each row, obtain the standard mathematical representation corresponding to each row, and obtain the formula representation matrix.

[0070] Preferably, the operation module includes a mathematical calculation submodule and a drawing submodule, wherein:

[0071] The mathematical calculation submodule is used to obtain the standard mathematical representation corresponding to each row in the formula representation matrix, and use the mathematical calculation engine to calculate according to the row to obtain the mathematical calculation result corresponding to each row;

[0072] The drawing submodule is used to respond to the mathematical calculation result corresponding to each row, and use the drawing engine to draw according to the row to obtain the graphic representation result corresponding to each row, and obtain the graphic representation corresponding to the acquired handwritten mathematical formula data.

[0073] In the above scheme, after obtaining the graphical representation corresponding to the formula, the system can display the recognized mathematical formula and calculation results on the screen of the visualization device, including but not limited to the electronic whiteboard; the user can correct and modify the recognition results, and the system will update the calculation and drawing results in real time. Secondly, the system provides a variety of interactive methods, including but not limited to handwriting input, gesture operation, voice commands, etc., to enhance the user experience.

[0074] In summary, the embodiments of the present invention provide a method and system for mathematical formula handwriting recognition, which have the following beneficial effects:

[0075] By performing low, medium and high level feature extraction on the acquired handwritten mathematical formula data, extracting the formula's stroke attributes, spatial structure, overall structure and logical relationship and other detail parameters, and further performing feature matching and symbol recognition, mathematical formula analysis and conversion, engine calculation and other formula conversion processes on the extracted results, a high-precision handwritten mathematical formula recognition result can be obtained. Furthermore, the present invention can improve the accuracy of complex handwritten mathematical formula recognition by performing hierarchical feature extraction on handwritten mathematical formula data and using the extracted detail parameters as the recognition basis for complex mathematical formulas, thereby effectively improving the recognition level of complex handwritten mathematical formulas.

[0076] In addition, the mathematical formula handwriting recognition system provided by the present invention has multiple interactive modes to enhance the user experience, specifically:

[0077] (1) The handwritten mathematical formula recognition results obtained by the system are displayed in real time on the screen of the visualization device;

[0078] (2) Users can correct and modify the recognition results, and the system will update the calculation and drawing results in real time;

[0079] (3) Users can operate the system by setting handwriting input, gesture operation, voice command, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a flow chart of an embodiment of the mathematical formula handwriting recognition method provided by the present invention;

[0081] Figure 2 It is a flowchart of an embodiment of a process of performing hierarchical feature extraction on acquired handwritten mathematical formula data provided by the present invention;

[0082] Figure 3 It is a structural schematic diagram of an embodiment of the mathematical formula handwriting recognition system provided by the present invention. DETAILED DESCRIPTION

[0083] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] Example 1

[0085] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of an embodiment of the method for handwriting recognition of mathematical formulas provided by the present invention. Figure 1 The method includes steps S001-S008, each of which is as follows:

[0086] Step S001: Acquire handwritten mathematical formula data.

[0087] In this embodiment, the source of obtaining the handwritten mathematical formula data includes but is not limited to capturing the user's writing trajectory using a high-precision stylus.

[0088] Step S002: Acquire a formula stroke image based on the handwritten mathematical formula data.

[0089] In this embodiment, the process of obtaining the formula stroke image based on the handwritten mathematical formula data is specifically as follows:

[0090] The symbols in the handwritten mathematical formula data are subjected to stroke denoising, stroke segmentation and image format conversion processing to obtain the stroke images corresponding to the symbols in the handwritten mathematical formula data.

[0091] Preferably, the accuracy of symbol recognition in the mathematical formula data can be improved through intelligent preprocessing methods including but not limited to stroke denoising, stroke segmentation, stroke splicing, etc., and then converted into corresponding stroke images through image format conversion processing for subsequent recognition.

[0092] Step S003: performing low-level feature extraction based on the formula stroke image, extracting basic attributes of the strokes, and obtaining a stroke attribute matrix.

[0093] Step S004: Perform mid-level feature extraction based on the stroke attribute matrix to extract the geometric shape and topological structure of the symbol to obtain a symbol structure matrix.

[0094] Step S005: Perform high-level feature extraction based on the symbol structure matrix, combine the context information in the handwritten mathematical formula data and the semantic information of the symbol, extract the overall structure and logical relationship of the formula, and obtain the formula structure matrix.

[0095] In this embodiment, during the feature extraction process described in steps S003 to S005, if the extraction result does not meet the judgment condition, a corresponding correction is performed, specifically:

[0096] When extracting low-level features, determine whether the stroke data is complete, and if it is incomplete, complete the data;

[0097] When extracting mid-level features, determine whether the geometric shape of the symbol is correct, and if not, make structural adjustments;

[0098] When extracting high-level features, it is determined whether the contextual information and semantic information of the formula match. If not, semantic correction is performed.

[0099] Step S006: Perform feature matching and symbol recognition based on the formula structure matrix to obtain a symbol recognition matrix.

[0100] In this embodiment, feature matching and symbol recognition are performed based on the formula structure matrix to obtain a symbol recognition matrix, which is specifically:

[0101] Using the preset database, feature matching and symbol recognition are performed based on the formula structure matrix to obtain a symbol recognition matrix.

[0102] Preferably, the preset database has a symbol template update mechanism, specifically:

[0103] Recognize the symbols in the acquired stroke image, compare the recognized symbols with the existing symbol templates in the preset database, and recognize new symbols;

[0104] Identify the new symbol and obtain its corresponding parameter;

[0105] In response to corresponding parameters of the new symbol, standardize the new symbol to form a symbol template;

[0106] Adds a new symbol template to the database.

[0107] In this embodiment, the symbol template update mechanism mentioned above is one aspect of the dynamic database update mechanism set by the present invention. In the process of identifying the symbols in the acquired stroke image, a pattern matching algorithm is used to identify the differences between the common symbols in the preset database and the symbols in the stroke image, and to detect the appearance of new symbols and changes in the user's writing habits. The corresponding parameters of the new symbol are obtained by detection, and the new symbol data is standardized and labeled to form a symbol template. The template is added to the database to improve the efficiency of subsequent matching and recognition, thereby realizing dynamic optimization of the database. On the other hand, the dynamic database update mechanism also includes an update trigger and feedback mechanism, which triggers the database update according to the frequency and accuracy of the user input, and displays the update results and recognition results to the user for feedback.

[0108] Furthermore, the above-mentioned use of the preset database performs feature matching and symbol recognition based on the formula structure matrix to obtain a symbol recognition matrix, which is specifically:

[0109] Each symbol in the formula structure matrix is ​​matched with the existing symbol template in the preset database, and the symbol is identified by the pattern matching algorithm to obtain the symbol recognition matrix.

[0110] Preferably, each symbol in the formula structure matrix obtained after high-level feature extraction is matched with the symbol template in the database to determine whether the symbol matching degree is high enough. If not, the template is expanded. Each symbol is identified by the pattern matching algorithm, and the identification results are stored in the symbol identification matrix. The symbol identification matrix is ​​initialized as the basis for subsequent formula transformation. Each row in the matrix is ​​a recognized symbol and its position in the formula.

[0111] Step S007: parsing and converting the mathematical formula based on the symbol recognition matrix to obtain a formula representation matrix.

[0112] In this embodiment, the mathematical formula is parsed and converted based on the symbol recognition matrix to obtain the formula representation matrix, and the specific steps are:

[0113] Get the symbol recognition results corresponding to each row of the formula representation matrix, combine the rows in the order of writing, and get the complete mathematical formula corresponding to each row;

[0114] Based on the complete mathematical formula corresponding to each row, the formula is parsed and converted row by row to obtain the standard mathematical representation corresponding to each row and obtain the formula representation matrix.

[0115] In the above scheme, the complete mathematical formula is parsed and converted to obtain the corresponding standard mathematical representation, whose format includes but is not limited to LaTeX format. Secondly, in the process of parsing and converting the mathematical formula based on the symbol recognition matrix, it is determined whether the formula structure is correct, and if not, the structure is corrected.

[0116] Preferably, the graphical representation corresponding to the acquired handwritten mathematical formula data is obtained based on the formula representation matrix, and the specific steps are:

[0117] Obtain the standard mathematical representation corresponding to each row in the formula representation matrix, perform calculations by row using a mathematical calculation engine, and obtain the mathematical calculation results corresponding to each row;

[0118] In response to the mathematical calculation result corresponding to each row, a drawing engine is used to draw the corresponding lines to obtain the graphic representation result corresponding to each line.

[0119] Preferably, the mathematical computing engine includes but is not limited to the SymPy computing engine.

[0120] Preferably, the drawing engine includes but is not limited to the Matplotlib drawing engine.

[0121] Step S008: obtaining a graphical representation corresponding to the handwritten mathematical formula data based on the formula representation matrix.

[0122] In this embodiment, when the mathematical calculation engine or the drawing engine is used for calculation, it is determined whether the formula is operable. If it is not operable, the user is prompted to re-enter the formula.

[0123] Example 2

[0124] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of an embodiment of the process of performing hierarchical feature extraction on the acquired handwritten mathematical formula data provided by the present invention. Figure 2 The method includes steps S101-S103, each of which is as follows:

[0125] Step S101: extract low-level features, extract stroke attributes, and initialize the stroke attribute matrix.

[0126] In this embodiment, the specific process of low-level feature extraction is as follows: after obtaining the preprocessed stroke image, the basic attributes of the strokes are extracted, including but not limited to length, angle, stroke order, etc. Further, the stroke attribute matrix is ​​initialized using the extracted basic stroke attributes, and each row in the matrix is ​​an attribute vector of a stroke, including attributes such as length (L), angle (θ), order (S), etc. The role of the order (S) is to record the writing order of each stroke to help identify the overall structure of symbols and formulas.

[0127] The extraction algorithm is as follows:

[0128]

[0129] Where: L is the stroke length; x1, x2 and y1, y2 are the coordinates of the stroke start and end points.

[0130] Step S102: mid-level feature extraction, extracting the geometric and topological properties of the symbol and initializing the symbol structure matrix.

[0131] In this embodiment, the specific process of mid-level feature extraction is: obtain the stroke attribute matrix after low-level feature extraction, extract the geometric shape and topological structure of the symbol, such as intersections, closed areas, etc. Further, use the extracted geometric shape and topological structure to initialize the symbol structure matrix, each row in the matrix is ​​the geometric and topological attributes of a symbol, including intersections (P), closed areas (C) and other attributes. Use the graphic analysis algorithm to identify the geometric shape and topological structure of the graphic and construct a symbol topology map.

[0132] The algorithm is as follows:

[0133] P=(xi,yi)

[0134]

[0135] Among them, P is the coordinate of the intersection point; x1, y1 are the coordinate values ​​of the intersection point.

[0136] Step S103: extract high-level features, extract the overall structure and logical relationship of the formula, and initialize the formula structure matrix.

[0137] In this embodiment, the specific process of high-level feature extraction is as follows: obtain the symbol structure matrix after the mid-level feature extraction. Combine the context information and the semantic information of the symbol to extract the overall structure and logical relationship of the formula. Initialize the formula structure matrix, each row in the matrix is ​​the structure and logical attributes of a formula, including context information (Ctx) and semantic information (Sem). Use natural language processing (NLP) technology to parse the grammatical and semantic relationships in the formula and construct the logical structure diagram of the formula.

[0138] The specific algorithm is as follows:

[0139]

[0140] Among them: Ctx is the contextual relationship of the symbol; Sem is the semantic information of the symbol.

[0141] In summary, by performing low, medium and high level feature extraction on the acquired handwritten mathematical formula data, and extracting detailed parameters such as the formula's stroke attributes, spatial structure, overall structure and logical relationships, high-precision feature extraction results can be obtained; further, by using the feature extraction results for subsequent formula transformation processes, high-precision handwritten mathematical formula recognition results can be obtained, thereby effectively improving the recognition level of complex handwritten mathematical formulas.

[0142] Example 3

[0143] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a structure of an embodiment of a mathematical formula handwriting recognition system provided by the present invention. Figure 3 The system includes a data processing module, a feature extraction module, a recognition module, a conversion module, and a calculation module, specifically:

[0144] A data processing module, including a data acquisition submodule, which is used to acquire handwritten mathematical formula data;

[0145] The feature extraction module includes a low-level feature extraction submodule, a mid-level feature extraction module and a high-level feature extraction submodule, wherein:

[0146] A low-level feature extraction submodule is used to extract low-level features based on the formula stroke image, extract the basic attributes of the strokes, and obtain a stroke attribute matrix;

[0147] The mid-level feature extraction submodule is used to extract mid-level features based on the stroke attribute matrix, extract the geometric shape and topological structure of the symbol, and obtain the symbol structure matrix;

[0148] A high-level feature extraction submodule is used to extract high-level features based on the symbol structure matrix, and extract the overall structure and logical relationship of the formula by combining the context information in the acquired handwritten mathematical formula data and the semantic information of the symbol to obtain the formula structure matrix;

[0149] A recognition module is used to perform feature matching and symbol recognition based on the formula structure matrix and a preset database to obtain a symbol recognition matrix;

[0150] A conversion module, used for parsing and converting mathematical formulas based on the symbol recognition matrix to obtain a formula representation matrix;

[0151] The operation module is used to obtain the graphical representation corresponding to the acquired handwritten mathematical formula data based on the formula representation matrix.

[0152] In this embodiment, the present invention provides a mathematical formula handwriting recognition system, which can extract the detailed parameters of the formula step by step by performing hierarchical feature extraction on the acquired handwritten mathematical formula data, and then further transform the formula to obtain a high-precision handwritten mathematical formula recognition result. Furthermore, using the high-precision handwritten mathematical formula recognition result, the system can realize the accurate recognition of complex handwritten mathematical formulas, effectively improving the system's recognition level for complex handwritten mathematical formulas.

[0153] Preferably, the basic attributes of the strokes extracted by the low-level feature extraction submodule include but are not limited to length, angle, order and other attributes. The role of the order is to record the mathematical order of each stroke to help identify the overall structure of symbols and formulas.

[0154] Preferably, the geometric shape and topological structure of the symbol extracted by the mid-level feature extraction submodule include but are not limited to properties such as intersections and closed areas. The geometric shape and topological structure of the graphic are identified using a graphic analysis algorithm, and a symbol topological map is constructed to facilitate subsequent identification.

[0155] Preferably, the overall structure and logical relationship of the formula extracted by the high-level feature extraction submodule include but are not limited to context information and semantic information. The grammatical and semantic relationships in the formula are analyzed using natural language processing (NLP) technology to construct a formula logical structure diagram for subsequent identification.

[0156] Preferably, the data processing module further includes a preprocessing submodule, specifically:

[0157] The preprocessing submodule is used to perform stroke denoising, stroke segmentation and image format conversion processing on the symbols in the handwritten mathematical formula data, and obtain the stroke images corresponding to the symbols in the handwritten mathematical formula data.

[0158] Furthermore, the mathematical formula handwriting recognition system provided by the present invention further includes a database module for establishing and updating a preset database using a symbol template update mechanism, specifically:

[0159] Recognize the symbols in the acquired stroke image, compare the recognized symbols with the existing symbol templates in the preset database, and recognize new symbols;

[0160] Identify the new symbol and obtain its corresponding parameter;

[0161] In response to corresponding parameters of the new symbol, standardize the new symbol to form a symbol template;

[0162] Adds a new symbol template to the database.

[0163] In this example, the database module recognizes the symbols in the acquired stroke image and compares them with the symbols in the preset database to obtain new symbols. In this process, user factors, such as changes in the user's writing habits, are considered to improve the accuracy of the system in recognizing new symbols. Adding new symbols to the database as templates after standardization is a dynamic update of the database. Secondly, the database module triggers database updates based on the frequency and accuracy of user input, and provides feedback to the user to show the update effect and recognition results.

[0164] Furthermore, the above recognition module is used to perform feature matching and symbol recognition based on the formula structure matrix and a preset database to obtain a symbol recognition matrix, specifically:

[0165] Each symbol in the formula structure matrix is ​​matched with the existing symbol template in the preset database, and the symbol is identified by the pattern matching algorithm to obtain the symbol recognition matrix.

[0166] Preferably, the conversion module includes a formula combination submodule and a formula conversion submodule, wherein:

[0167] The formula combination submodule is used to obtain the symbol recognition results corresponding to each row of the symbol recognition matrix, and combine them according to the writing order to obtain the complete mathematical formula corresponding to each row;

[0168] The formula conversion submodule is used to parse and convert the complete mathematical formula corresponding to each row, obtain the standard mathematical representation corresponding to each row, and obtain the formula representation matrix.

[0169] Preferably, the operation module includes a mathematical calculation submodule and a drawing submodule, wherein:

[0170] The mathematical calculation submodule is used to obtain the standard mathematical representation corresponding to each row in the formula representation matrix, and use the mathematical calculation engine to calculate according to the row to obtain the mathematical calculation result corresponding to each row;

[0171] The drawing submodule is used to respond to the mathematical calculation result corresponding to each row, and use the drawing engine to draw according to the row to obtain the graphic representation result corresponding to each row, and obtain the graphic representation corresponding to the acquired handwritten mathematical formula data.

[0172] In this embodiment, after obtaining the graphical representation corresponding to the formula, the system can display the recognized mathematical formula and calculation results on the screen of a visualization device, including but not limited to an electronic whiteboard; the user can correct and modify the recognition results, and the system will update the calculation and drawing results in real time. Secondly, the system provides a variety of interactive methods, including but not limited to handwriting input, gesture operation, voice commands, etc., to enhance the user experience.

[0173] In summary, the embodiments of the present invention provide a batch transfer method and system based on blockchain, which has the following beneficial effects:

[0174] By performing low, medium and high level feature extraction on the acquired handwritten mathematical formula data, extracting the formula's stroke attributes, spatial structure, overall structure and logical relationship and other detail parameters, and further performing feature matching and symbol recognition, mathematical formula analysis and conversion, engine calculation and other formula conversion processes on the extracted results, a high-precision handwritten mathematical formula recognition result can be obtained. Furthermore, the present invention can improve the accuracy of complex handwritten mathematical formula recognition by performing hierarchical feature extraction on handwritten mathematical formula data and using the extracted detail parameters as the recognition basis for complex mathematical formulas, thereby effectively improving the recognition level of complex handwritten mathematical formulas.

[0175] In addition, the mathematical formula handwriting recognition system provided by the present invention has multiple interactive modes to enhance the user experience, specifically:

[0176] (1) The handwritten mathematical formula recognition results obtained by the system are displayed in real time on the screen of the visualization device;

[0177] (2) Users can correct and modify the recognition results, and the system will update the calculation and drawing results in real time;

[0178] (3) Users can operate the system by setting handwriting input, gesture operation, voice command, etc.

[0179] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

[0180] Those skilled in the art can understand that all or part of the processes in the above embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above embodiments. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

Claims

1. A method for handwriting recognition of mathematical formulas, characterized in that: The following steps are involved: Get handwritten math formula data; Acquire a formula stroke image based on the handwritten mathematical formula data; Based on the formula stroke image, low-level feature extraction is performed to extract basic attributes of the strokes to obtain a stroke attribute matrix; Performing mid-level feature extraction based on the stroke attribute matrix to extract the geometric shape and topological structure of the symbol to obtain a symbol structure matrix; Based on the symbol structure matrix, high-level feature extraction is performed, and the overall structure and logical relationship of the formula are extracted by combining the context information in the handwritten mathematical formula data and the semantic information of the symbol to obtain the formula structure matrix; Perform feature matching and symbol recognition based on the formula structure matrix to obtain a symbol recognition matrix; Parsing and converting mathematical formulas based on the symbol recognition matrix to obtain a formula representation matrix; A graphical representation corresponding to the handwritten mathematical formula data is obtained based on the formula representation matrix.

2. A mathematical formula handwriting recognition method as claimed in claim 1, characterized in that: The step of acquiring a formula stroke image based on the handwritten mathematical formula data is specifically as follows: The symbols in the handwritten mathematical formula data are subjected to stroke denoising, stroke segmentation and image format conversion processing to obtain stroke images corresponding to the symbols in the handwritten mathematical formula data.

3. A mathematical formula handwriting recognition method as claimed in claim 2, characterized in that: The feature matching and symbol recognition are performed based on the formula structure matrix to obtain a symbol recognition matrix, which is specifically: Using a preset database, feature matching and symbol recognition are performed based on the formula structure matrix to obtain a symbol recognition matrix; The preset database has a symbol template update mechanism, specifically: Recognize the symbols in the stroke image, compare the recognized symbols with the existing symbol templates in the preset database, and recognize new symbols; Identify the new symbol and obtain corresponding parameters of the new symbol; In response to the corresponding parameters, the new symbol is standardized to form a symbol template; Add the symbol template to the database.

4. A mathematical formula handwriting recognition method as claimed in claim 3, characterized in that: The method of using a preset database to perform feature matching and symbol recognition based on the formula structure matrix to obtain a symbol recognition matrix is ​​specifically: Each symbol in the formula structure matrix is ​​matched with a symbol template existing in the preset database, and the symbol is identified by a pattern matching algorithm to obtain the symbol identification matrix.

5. A mathematical formula handwriting recognition method as claimed in claim 4, characterized in that: The mathematical formula is parsed and converted based on the symbol recognition matrix to obtain a formula representation matrix, which is specifically: Obtain the symbol recognition results corresponding to each row of the formula representation matrix, and combine the rows in the order of writing to obtain the complete mathematical formula corresponding to each row; The complete mathematical formula corresponding to each row is parsed and converted row by row to obtain the standard mathematical representation corresponding to each row, thereby obtaining the formula representation matrix.

6. A mathematical formula handwriting recognition method as claimed in claim 5, characterized in that: The graphical representation corresponding to the handwritten mathematical formula data is obtained based on the formula representation matrix, specifically: Obtain the standard mathematical representation corresponding to each row in the formula representation matrix, and perform calculations by row using a mathematical calculation engine to obtain a mathematical calculation result corresponding to each row; In response to the mathematical calculation result corresponding to each row, drawing by using a drawing engine row by row to obtain a graphic representation result corresponding to each row; Based on the graphic representation result corresponding to each row, a graphic representation corresponding to the handwritten mathematical formula data is obtained.

7. A mathematical formula handwriting recognition system, characterized in that: The system includes a data processing module, a feature extraction module, a recognition module, a conversion module, and a calculation module, specifically: The data processing module includes a data acquisition submodule; The data acquisition submodule is used to acquire handwritten mathematical formula data; The feature extraction module includes a low-level feature extraction submodule, a mid-level feature extraction module and a high-level feature extraction submodule; The low-level feature extraction submodule is used to perform low-level feature extraction based on the formula stroke image, extract basic attributes of the strokes, and obtain a stroke attribute matrix; The mid-level feature extraction submodule is used to extract mid-level features based on the stroke attribute matrix, extract the geometric shape and topological structure of the symbol, and obtain the symbol structure matrix; The high-level feature extraction submodule is used to perform high-level feature extraction based on the symbol structure matrix, extract the overall structure and logical relationship of the formula in combination with the context information and the semantic information of the symbol in the handwritten mathematical formula data, and obtain the formula structure matrix; The recognition module is used to perform feature matching and symbol recognition based on the formula structure matrix and a preset database to obtain a symbol recognition matrix; The conversion module is used to parse and convert the mathematical formula based on the symbol recognition matrix to obtain a formula representation matrix; The operation module is used to obtain the graphical representation corresponding to the handwritten mathematical formula data based on the formula representation matrix.

8. A mathematical formula handwriting recognition system as claimed in claim 7, characterized in that: The data processing module also includes a preprocessing submodule, specifically: The preprocessing submodule is used to perform stroke denoising, stroke segmentation and image format conversion processing on the symbols in the handwritten mathematical formula data, and obtain the stroke images corresponding to the symbols in the handwritten mathematical formula data.

9. A mathematical formula handwriting recognition system as claimed in claim 8, characterized in that: The system also includes a database module, which is used to establish and update the preset database using a symbol template update mechanism, specifically: Recognize the symbols in the stroke image, compare the recognized symbols with the existing symbol templates in the preset database, and recognize new symbols; Identify the new symbol and obtain corresponding parameters of the new symbol; In response to the corresponding parameters, the new symbol is standardized to form a symbol template; Add the symbol template to the database.

10. A mathematical formula handwriting recognition system as claimed in claim 9, characterized in that: The recognition module is used to perform feature matching and symbol recognition based on the formula structure matrix and a preset database to obtain a symbol recognition matrix, specifically: Each symbol in the formula structure matrix is ​​matched with a symbol template existing in the preset database, and the symbol is identified by a pattern matching algorithm to obtain the symbol identification matrix.

11. A mathematical formula handwriting recognition system as claimed in claim 10, characterized in that: The conversion module includes a formula combination submodule and a formula conversion submodule, specifically: The formula combination submodule is used to obtain the symbol recognition results corresponding to each row of the symbol recognition matrix, and combine them according to the writing order of each row to obtain the complete mathematical formula corresponding to each row; The formula conversion submodule is used to parse and convert the complete mathematical formula corresponding to each row row by row, obtain the standard mathematical representation corresponding to each row, and obtain the formula representation matrix.

12. A mathematical formula handwriting recognition system as claimed in claim 11, characterized in that: The operation module includes a mathematical calculation submodule and a drawing submodule, specifically: The mathematical calculation submodule is used to obtain the standard mathematical representation corresponding to each row in the formula representation matrix, and use the mathematical calculation engine to perform calculations on a row-by-row basis to obtain the mathematical calculation results corresponding to each row; The drawing submodule is used to respond to the mathematical calculation result corresponding to each row, and use the drawing engine to draw row by row to obtain the graphic representation result corresponding to each row, and obtain the graphic representation corresponding to the handwritten mathematical formula data.

Images