Mathematical formula identification coding method

Abstract

The invention discloses a mathematical formula identification coding method, and particularly relates to the technical field of formula coding. The method comprises the following steps: acquiring mathematical formula image data to be identified, and performing symbol boundary extraction and preprocessing to generate symbol feature expression data; and performing visual spatial layout analysis and symbol type semantic classification based on the symbol feature expression data to generate formula layout structure data and symbol semantic classification data. And through graph structure analysis based on a topological relation, determining an inter-symbol topological relation of the mathematical formula, and generating symbol topological relation data. And deriving a dimension constraint relationship and an operator dependency relationship between symbols through a mathematical meta-knowledge mining technology, and generating mathematical meta-knowledge constraint data. Initial mathematical formula structure expression data is generated through decoding of structure and semantic constraint fusion, and a formula coding sequence is generated through formula consistency verification and semantic constraint reconstruction. According to the invention, the accuracy and efficiency of mathematical formula identification can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of formula encoding technology, and more specifically, to a mathematical formula recognition and encoding method. Background Technology

[0002] Automatic recognition and encoding technology for mathematical formulas has wide applications in academic papers, textbooks, and electronic document processing. Existing mathematical formula recognition technologies typically rely on image processing techniques to extract formula information from documents and then identify the formula based on its structural and symbolic features.

[0003] Traditional methods still have certain limitations when dealing with the analysis of complex mathematical formulas. In particular, when handling complex formulas with multiple structural relationships, symbolic dependencies, and dimensional constraints, existing technologies struggle to fully uncover the hierarchical structure and semantic relationships between the symbols in the formula. This results in insufficient accuracy in formula structure analysis and symbol matching, thus affecting the accuracy and efficiency of formula recognition. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide a mathematical formula recognition and encoding method to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A mathematical formula recognition and encoding method includes the following steps:

[0007] S1: Obtain the image data of the mathematical formula to be identified, perform symbol boundary extraction and preprocessing on the mathematical formula image data, and generate symbol feature expression data;

[0008] S2: Based on the symbol feature expression data, perform visual spatial layout parsing and symbol type semantic classification respectively to generate formula layout structure data and symbol semantic classification data;

[0009] S3: Based on the formula layout structure data, determine the topological relationships between symbols in the mathematical formula through graph structure analysis based on topological relationships, and generate symbol topological relationship data;

[0010] S4: Based on the symbolic semantic classification data, through mathematical meta-knowledge mining, dimensional constraint relationships and operator dependency relationships between symbols are deduced to generate mathematical meta-knowledge constraint data;

[0011] S5: Based on symbolic topological relation data and mathematical meta-knowledge constraint data, generate initial mathematical formula structure expression data containing implicit mathematical meta-knowledge constraints through decoding that fuses structural and semantic constraints.

[0012] S6: Based on the initial mathematical formula structure, express the data, and generate the formula encoding sequence through formula consistency verification and semantic constraint reconstruction.

[0013] In a preferred embodiment, S1 specifically refers to:

[0014] Obtain the image data of the mathematical formula to be identified;

[0015] The image data of the mathematical formula to be identified is subjected to image grayscale conversion and adaptive binarization.

[0016] Based on the mathematical formula image data after adaptive binarization, the boundary regions of mathematical formula symbols are identified, and the position of each mathematical formula symbol in the mathematical formula image data is determined.

[0017] Feature extraction is performed on the boundary region of each mathematical formula symbol to extract the shape, size, and spatial location features of the mathematical formula symbol and generate symbol feature representation data.

[0018] In a preferred embodiment, S2 specifically refers to:

[0019] Based on the shape, size, and spatial position characteristics of mathematical formula symbols in the symbol feature representation data, the relative positional relationships between mathematical formula symbols are identified, the two-dimensional layout structure of symbols in mathematical formulas is determined, and formula layout structure data is generated.

[0020] Based on the shape and size features of mathematical formula symbols in the symbol feature representation data, the categories of mathematical formula symbols are identified and classified, the mathematical symbol type corresponding to each mathematical formula symbol is determined, and symbol semantic classification data is generated.

[0021] In a preferred embodiment, S3 specifically refers to:

[0022] Based on the formula layout structure data, the connection relationship between mathematical formula symbols is established by utilizing the relative positional relationship between mathematical formula symbols;

[0023] Based on the connections between mathematical formula symbols, construct a topological graph with each mathematical formula symbol as a node;

[0024] Based on the connection relationships between nodes in the topological graph, determine the hierarchical, adjacency, and nesting relationships between mathematical formula symbol nodes in the topological graph;

[0025] Based on the hierarchical, adjacency, and nesting relationships between the mathematical formula symbol nodes in the topological relationship diagram, symbolic topological relationship data is generated.

[0026] In a preferred embodiment, S4 specifically refers to:

[0027] Based on the mathematical symbol type of each mathematical formula symbol in the symbol semantic classification data, query the preset meta-knowledge rule base to obtain the physical dimension information, operator meta-number information and operation priority information corresponding to the mathematical symbol type, and generate the initial table of symbol meta-knowledge.

[0028] Based on the initial table of symbolic meta-knowledge, the operation rules are matched for symbol combinations in the symbolic semantic classification data to determine the operation associations between symbols and generate an operator dependency table.

[0029] For each operational association in the operator dependency table, perform dimensional consistency derivation to establish dimensional constraint entries between symbols;

[0030] Integrate the operator dependency table and dimension constraint entries to encode and generate mathematical meta-knowledge constraint data.

[0031] In a preferred embodiment, S5 specifically refers to:

[0032] Based on the hierarchical, adjacency, and nesting relationships between mathematical formula symbol nodes in the symbolic topology data, an initial mathematical formula structure tree is constructed.

[0033] Based on the operator dependency table and dimension constraint entries in the mathematical meta-knowledge constraint data, constraint matching is performed on the symbol nodes in the initial mathematical formula structure tree to determine the effective connection methods between symbol nodes that conform to operator dependencies and dimension constraints.

[0034] Based on effective connection methods, the initial mathematical formula structure tree is modified to generate a mathematical formula structure tree that integrates mathematical meta-knowledge constraints.

[0035] Based on the mathematical formula structure tree that integrates mathematical meta-knowledge constraints, tree structure traversal and node relationship transformation are used to generate initial mathematical formula structure expression data containing implicit mathematical meta-knowledge constraints.

[0036] In a preferred embodiment, S6 specifically refers to:

[0037] Based on the mathematical formula structure tree that integrates mathematical meta-knowledge constraints, the operation dependency and dimensional constraint information of each mathematical formula symbol node are extracted to generate a mathematical formula node constraint verification table.

[0038] Based on the mathematical formula node constraint verification table, the formula consistency verification is performed on each symbol node in the mathematical formula structure tree that integrates mathematical meta-knowledge constraints, and constraint conflicts of symbol nodes in the mathematical formula structure tree are identified and marked.

[0039] Perform semantic constraint reconstruction on the marked constraint conflict nodes to regenerate the final mathematical formula structure tree that satisfies the constraint conditions;

[0040] Based on the final mathematical formula structure tree that satisfies the constraints, the encoding transformation of the nodes in the mathematical formula structure tree is performed to generate a formula encoding sequence to represent the mathematical formula.

[0041] The technical effects and advantages of the mathematical formula recognition and encoding method of the present invention are as follows:

[0042] By extracting and preprocessing symbol boundaries from mathematical formula image data, the accuracy and noise resistance of symbol detection can be effectively improved. Visual spatial layout parsing and semantic classification of symbol types are performed based on symbol feature representation data, accurately identifying two-dimensional structures such as subscripts, superscripts, fractions, and matrices, as well as distinguishing symbol types such as numbers, letters, and operators. Topological graph structure analysis effectively depicts the hierarchical, adjacent, and nested relationships between symbol nodes, enhancing the completeness of structural representation. Mathematical meta-knowledge mining using a mathematical meta-knowledge rule base can deduce dimensional constraints and operator dependencies, improving the consistency of formula semantics. Through decoding that fuses structural and semantic constraints, initial formula structure representation data containing implicit constraint information is generated. Combined with formula consistency verification and semantic constraint reconstruction, conflicting symbols are corrected, and an encoding sequence conforming to mathematical logic and physical dimensions is generated, thereby improving the accuracy and robustness of mathematical formula recognition. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of a mathematical formula recognition and encoding method according to the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0045] Example

[0046] Figure 1 This invention provides a mathematical formula recognition and encoding method, which includes the following steps:

[0047] S1: Obtain the image data of the mathematical formula to be identified, perform symbol boundary extraction and preprocessing on the mathematical formula image data, and generate symbol feature expression data;

[0048] S2: Based on the symbol feature expression data, perform visual spatial layout parsing and symbol type semantic classification respectively to generate formula layout structure data and symbol semantic classification data;

[0049] S3: Based on the formula layout structure data, determine the topological relationships between symbols in the mathematical formula through graph structure analysis based on topological relationships, and generate symbol topological relationship data;

[0050] S4: Based on the symbolic semantic classification data, through mathematical meta-knowledge mining, dimensional constraint relationships and operator dependency relationships between symbols are deduced to generate mathematical meta-knowledge constraint data;

[0051] S5: Based on symbolic topological relation data and mathematical meta-knowledge constraint data, generate initial mathematical formula structure expression data containing implicit mathematical meta-knowledge constraints through decoding that fuses structural and semantic constraints.

[0052] S6: Based on the initial mathematical formula structure, express the data, and generate the formula encoding sequence through formula consistency verification and semantic constraint reconstruction.

[0053] S1: Obtain the image data of the mathematical formula to be identified, perform symbol boundary extraction and preprocessing on the mathematical formula image data, and generate symbol feature representation data, including:

[0054] Obtain the image data of the mathematical formula to be identified;

[0055] Mathematical formula image data refers to the original image of a mathematical formula presented in a paper or electronic document. Examples include, but are not limited to, digital images obtained by scanning a paper mathematical document with a high-resolution scanner, or digital images generated by cropping the area displaying the mathematical formula on a screen using a digital device. The acquired mathematical formula image data format includes, but is not limited to, JPEG, BMP, or PNG formats.

[0056] The image data of the mathematical formula to be identified is subjected to image grayscale conversion and adaptive binarization.

[0057] The image grayscale conversion process is as follows: For each pixel in the acquired mathematical formula image data, the corresponding grayscale value is calculated using the grayscale conversion formula based on the pixel values ​​of the red, green, and blue channels. The grayscale conversion formula can be expressed as: Grayscale value = 0.299 × red channel value + 0.587 × green channel value + 0.114 × blue channel value. The calculated grayscale value is then used to replace the original pixel value, ultimately generating the grayscale-converted mathematical formula image data.

[0058] The adaptive binarization process involves calculating the average grayscale value of a local pixel region centered on the pixel for each pixel in the converted mathematical formula image data. This average grayscale value is then compared to a preset binarization threshold to determine the final binarized value of the pixel. The size of the local pixel region used in the adaptive binarization process is determined by the window size, such as a 5×5 pixel region or a 7×7 pixel region. The preset binarization threshold is determined by analyzing and statistically processing a large amount of typical mathematical formula image data, using the average grayscale value of each local region as the benchmark for the binarization threshold. For example, the benchmark threshold can be set between 120 and 140. When the average grayscale value of a local region is higher than the preset binarization threshold, the center pixel is set to white (value 255); when the average grayscale value of a local region is lower than or equal to the preset binarization threshold, the center pixel is set to black (value 0).

[0059] Based on the mathematical formula image data after adaptive binarization, the boundary regions of mathematical formula symbols are identified, and the position of each mathematical formula symbol in the mathematical formula image data is determined.

[0060] A contour detection algorithm is used to detect black pixel regions in mathematical formula image data, extracting the boundary contour information of each independent symbol region. The contour detection algorithm employs a pixel connectivity-based method: for binary image data, each pixel is scanned row by row from left to right and top to bottom. When a black pixel is detected, a depth-first search or breadth-first search method is used to traverse the pixel's adjacent regions, identifying all connected black pixels, forming the boundary region of the mathematical formula symbol. The coordinate information of the pixels in the boundary region is recorded to determine the position of the boundary region in the mathematical formula image data. This process is repeated for all symbol regions in the mathematical formula image data to identify all boundary regions of mathematical formula symbols within the image data.

[0061] For each identified mathematical formula symbol boundary region, the geometric center position of the mathematical formula symbol boundary region is calculated based on the coordinate information of all pixels in the boundary region. For example, by calculating the average horizontal and vertical coordinates of all pixels in the boundary region, two-dimensional coordinates representing the position of each mathematical formula symbol region are obtained.

[0062] Feature extraction is performed on the boundary region of each mathematical formula symbol to extract the shape, size and spatial location features of the mathematical formula symbol and generate symbol feature representation data;

[0063] The shape features of mathematical formula symbols are extracted as follows: based on the geometric contour of the symbol's boundary region, geometric indices such as curvature, perimeter, and area are calculated. For example, the ratio of perimeter to area determines the compactness of the shape, and Fourier descriptors are used to describe the contour curve features. The dimensional features of mathematical formula symbols are extracted as follows: the width and height of the bounding rectangle of the symbol's boundary region are calculated, and the aspect ratio of the rectangle is recorded. The spatial location features of mathematical formula symbols are extracted as follows: using defined two-dimensional coordinates, the coordinates of the geometric center of each mathematical formula symbol are recorded. The extracted shape, dimensional, and spatial location features are combined to form symbol feature representation data.

[0064] S2: Based on the symbol feature representation data, perform visual spatial layout analysis and symbol type semantic classification respectively to generate formula layout structure data and symbol semantic classification data, including:

[0065] Based on the shape, size, and spatial position characteristics of mathematical formula symbols in the symbol feature representation data, the relative positional relationships between mathematical formula symbols are identified, the two-dimensional layout structure of symbols in mathematical formulas is determined, and formula layout structure data is generated.

[0066] The relative positional relationship between mathematical formula symbols is expressed as the relative distance between the symbols in the horizontal and vertical directions: For each mathematical formula symbol in the symbol feature representation data, the position of the mathematical formula symbol in the mathematical formula image data is determined based on the geometric center coordinates of the symbol's spatial position features; the horizontal and vertical distances between the geometric center coordinates of any two mathematical formula symbols are calculated, thereby obtaining the relative spatial relationship between any two mathematical formula symbols. Through the above method, the relative spatial positional relationship between any two mathematical formula symbols in the mathematical formula image data can be obtained.

[0067] Based on the horizontal and vertical distances between the relative positions of mathematical formula symbols, the layout relationships between the symbols are determined, including but not limited to superscript, subscript, horizontal side-by-side, fractional, or matrix relationships. The layout determination rules are used to identify these relationships. These rules are as follows: when the geometric center of one mathematical formula symbol is above the geometric center of another symbol and the vertical distance is greater than a preset vertical threshold, while the horizontal distance is less than a preset horizontal threshold, the two symbols are determined to have a superscript / subscript layout relationship. The superscript / subscript determination is based on the relative size of the vertical coordinates. When the vertical distance between the geometric centers of two mathematical formula symbols is less than a preset vertical threshold, but the horizontal distance is greater than a preset horizontal threshold, the two symbols are determined to have a horizontal side-by-side layout relationship. When both the horizontal and vertical distances between the symbols are greater than their respective preset thresholds, the layout is determined to be either a matrix or fractional structure based on the relative positions of the symbols. For example, preset vertical and horizontal thresholds are determined by statistically analyzing the distances between symbols in superscript / subscript relationships, horizontal relationships, and matrix relationships within a large amount of image data of various types of mathematical formulas. Applying layout determination rules, all mathematical formula symbols are organized according to the identified two-dimensional layout structure relationships, and the two-dimensional layout structure is represented in the form of a tree structure or graph structure, generating formula layout structure data.

[0068] Based on the shape and size features of mathematical formula symbols in the symbol feature representation data, the categories of mathematical formula symbols are identified and classified, the mathematical symbol type corresponding to each mathematical formula symbol is determined, and symbol semantic classification data is generated.

[0069] A mathematical symbol type feature database is constructed, including typical shape and size features of various mathematical symbols, such as but not limited to numeric symbols, Greek letter symbols, regular Latin letter symbols, special mathematical symbols, operator symbols, and bracket symbols. The shape and size features of each mathematical formula symbol in the symbol feature expression data are compared with the typical shape and size features in the mathematical symbol type feature database. A similarity matching algorithm is used to determine the similarity between each mathematical formula symbol and its type. For example, the similarity matching algorithm can employ Euclidean distance or cosine similarity calculation. Taking Euclidean distance calculation as an example, the Euclidean distance is calculated between the shape feature vector of each mathematical formula symbol in the symbol feature expression data and the corresponding typical feature vector of each mathematical symbol type in the mathematical symbol type feature database. The mathematical symbol type with the smallest Euclidean distance is determined as the recognition type of the mathematical formula symbol. Through this classification process, the mathematical symbol type corresponding to each mathematical formula symbol is obtained, such as numeric type, Greek letter type, or operator type, and symbol semantic classification data is generated.

[0070] S3: Based on the formula layout structure data, determine the topological relationships between symbols in the mathematical formula through graph structure analysis based on topological relationships, and generate symbol topological relationship data, including:

[0071] Based on the formula layout structure data, the connection relationship between mathematical formula symbols is established by utilizing the relative positional relationship between mathematical formula symbols;

[0072] The connection relationships between mathematical formula symbols indicate whether there is a structural association between two mathematical formula symbols. The connection relationship is determined as follows: For two-dimensional layout structural relationships in the formula layout data, such as when one mathematical formula symbol is a superscript, subscript, or numerator and denominator of another mathematical formula symbol, a connection relationship is established between the two mathematical formula symbols, and this connection relationship is represented by a data structure. For example, the data structure for connection relationships can use matrix representation. If there are N mathematical formula symbols, an N×N connection matrix is ​​defined, where each element takes the value 0 or 1. When an element in the matrix is ​​1, it indicates a direct connection between the two mathematical formula symbols in the corresponding row and column; when an element is 0, it indicates no direct connection between the two mathematical formula symbols in the corresponding row and column. After this operation, a connection matrix representing all connection relationships between mathematical formula symbols is obtained.

[0073] Based on the connections between mathematical formula symbols, construct a topological graph with each mathematical formula symbol as a node;

[0074] A topological graph is a structured graphical representation of mathematical formula symbols as nodes and connections as edges. The construction of a topological graph involves defining nodes for each mathematical formula symbol identified in the mathematical formula image data, with each node corresponding to a specific mathematical formula symbol. Based on the connection relationship data in the connection matrix, edges are established between nodes corresponding to connections between mathematical formula symbols, representing the structural association between the two symbols. In the topological graph, the edges between nodes are directed edges, and their direction is determined according to the logical direction in a two-dimensional layout structure. For example, in a fraction structure, the directed edge points from the numerator symbol node to the denominator symbol node; in a superscript / subscript structure, the directed edge points from the main symbol node to the superscript or subscript symbol node; in a matrix structure, the direction of the directed edge is determined by the order of the matrix symbols in the row or column direction. For example, the directed edge between mathematical formula symbol nodes in the same row is from left to right, and the directed edge between mathematical formula symbol nodes in the same column is from top to bottom. This method is used to construct a topological graph of mathematical formula symbols.

[0075] Based on the connection relationships between nodes in the topological graph, determine the hierarchical, adjacency, and nesting relationships between mathematical formula symbol nodes in the topological graph;

[0076] Hierarchical relationships represent the hierarchical order between mathematical formula symbols. They are determined as follows: starting from the root node in the topological graph, traverse downwards. The root node is defined as a node in the mathematical formula structure that does not serve as a superscript, subscript, or fractional symbol for other mathematical formula symbols; the root node is at the top level. Nodes directly connected to and below the root node are determined as second-level nodes. This process continues downwards layer by layer until all nodes have determined their levels. Adjacency relationships represent the relative positions of mathematical formula symbol nodes at the same level. They are determined as follows: for mathematical formula symbol nodes at the same level in the topological graph, based on the two-dimensional coordinates of the symbols in the formula layout structure data, determine the adjacency order of nodes from left to right according to the coordinate values ​​of their geometric centers. Nested relationships represent the hierarchical relationship between mathematical formula symbol nodes with substructures. They are determined as follows: if one node in the topological graph is a direct child of another node, then the direct child node and its parent node have a nested relationship; if a node has multiple child nodes, and these child nodes have superscript, subscript, or fractional relationships, then the nested relationship between the child nodes is determined. The hierarchical, adjacency, and nesting relationships between nodes in the topological graph are determined using the methods described above.

[0077] Generate symbolic topology data based on the hierarchical, adjacency, and nesting relationships between mathematical formula symbol nodes in the topology graph;

[0078] For each mathematical formula symbol node in the topological graph, the hierarchical position of the node is recorded. For example, if the root node is set as the first level, the node's level is recorded as number 1, the second level node below it is number 2, and so on. The adjacent order of nodes within the same level is also recorded, for example, by marking the sequence numbers of adjacent nodes from left to right. The parent and child node information of each mathematical formula symbol node is recorded, that is, determining the corresponding mathematical formula symbol type and node position of the directly connected upper-level and lower-level nodes. For nested relationships between nodes, the nesting type between the parent node and all its child nodes is recorded, including but not limited to superscript nesting, subscript nesting, fraction nesting, or matrix nesting, for example, using a tree structure or linked list structure to record the symbolic topological relationship data. Through the above methods, the topological association information between each mathematical formula symbol node and other nodes is represented, thus forming symbolic topological relationship data. Symbolic topological relationship data includes the hierarchical position, adjacent order, parent and child node information, and nesting type information between nodes for each mathematical formula symbol node.

[0079] S4: Based on the symbolic semantic classification data, through mathematical meta-knowledge mining, dimensional constraint relationships and operator dependencies between symbols are derived, generating mathematical meta-knowledge constraint data, including:

[0080] Based on the mathematical symbol type of each mathematical formula symbol in the symbol semantic classification data, query the preset meta-knowledge rule base to obtain the physical dimension information, operator meta-number information and operation priority information corresponding to the mathematical symbol type, and generate the initial table of symbol meta-knowledge.

[0081] For each mathematical formula symbol in the symbol semantic classification data, a pre-defined meta-knowledge rule base is queried based on the mathematical symbol type. This meta-knowledge rule base is a pre-established rule database containing basic mathematical knowledge information for various mathematical symbol types, including physical dimension information, operator atom information, and operation priority information corresponding to each mathematical symbol type. Physical dimension information refers to the physical meaning or unit of measurement represented by the mathematical symbol, such as length, time, mass, and current. Operator atom information refers to the number of operands involved in the mathematical operation, such as unary, binary, or multivariate operators. Operation priority information refers to the order in which the operators represented by the mathematical symbol are executed during formula calculations, such as multiplication having higher priority than addition. Through these query operations, the meta-knowledge data corresponding to each mathematical formula symbol is obtained, forming an initial symbol meta-knowledge table.

[0082] Based on the initial table of symbolic meta-knowledge, the operation rules are matched for symbol combinations in the symbolic semantic classification data to determine the operation associations between symbols and generate an operator dependency table.

[0083] For the sequence of mathematical formula symbols identified in the symbol semantic classification data, the mathematical formula symbols are matched one by one or group by group according to mathematical operation rules and grammatical structure. Operation rules include, but are not limited to, arithmetic operation rules, function call rules, integral and differential operation rules, or matrix operation rules. For symbol combinations, it is first determined whether the mathematical formula symbols in the combination meet the atom number requirements of the operation rules, i.e., it is checked against the operator atom number information in the initial table of symbol atom knowledge. For example, for a binary operator type mathematical symbol, it is necessary to match two operand type mathematical formula symbols; when a match is successful, the priority order of operation execution is confirmed according to the operation priority information, thereby determining the operation association relationship between symbols. The operation association relationship records the operation order and combination relationship between mathematical formula symbols. For example, the operation association between a multiplication operator symbol and two numeric type symbols is that the numeric type symbols are operand symbols for multiplication, and the multiplication operator symbol is the main symbol of the operation rule. By matching the mathematical formula symbols according to the operation rules, the operation association relationships between all symbols are obtained, and finally, an operator dependency table is generated.

[0084] For each operational association in the operator dependency table, perform dimensional consistency derivation to establish dimensional constraint entries between symbols;

[0085] For each operational association in the operator dependency table, the physical dimensions of the mathematical formula symbols involved in the association are first determined. This involves extracting the physical dimensions of the operator symbol and each operand symbol from the initial symbolic knowledge table. Consistency is then determined using dimensional derivation formulas or dimensional analysis rules. For example, for multiplication of two operand symbols, the dimensional consistency derivation formula is: Dimensional result = Dimension of the first operand symbol × Dimension of the second operand symbol. For addition or subtraction of two operand symbols, the two operand symbols must have identical physical dimensions; otherwise, the dimensional consistency rule is not satisfied. Dimensional consistency is deduced for all operational associations using this method, generating corresponding dimensional constraint entries and recording whether each operational association conforms to the dimensional consistency rule.

[0086] Integrate the operator dependency table and dimension constraint entries, and encode to generate mathematical meta-knowledge constraint data;

[0087] By integrating the operator dependency table with each dimensional constraint entry, an information table or tree-like data structure with operational associations and dimensional consistency constraints is formed, thus obtaining mathematical meta-knowledge constraint data. Each mathematical formula symbol node records meta-knowledge constraint information such as symbol type, number of operator atomists, operation priority, operational associations, and dimensional consistency derivation results.

[0088] S5: Based on symbolic topological relation data and mathematical meta-knowledge constraint data, through decoding that fuses structural and semantic constraints, initial mathematical formula structure expression data containing implicit mathematical meta-knowledge constraints is generated, including:

[0089] Based on the hierarchical, adjacency, and nesting relationships between mathematical formula symbol nodes in the symbolic topology data, an initial mathematical formula structure tree is constructed.

[0090] A mathematical formula structure tree is a hierarchical data structure used to represent the internal symbolic organization of a mathematical formula. Each mathematical formula symbol corresponds to a node in the structure tree, and the connections between nodes are determined based on symbolic topology data. The process involves reading the node hierarchy information from the symbolic topology data and setting the node with the lowest hierarchical position as the root node of the tree. Following the hierarchical position downwards, nodes at each level are connected according to the parent-child node correspondence recorded in the symbolic topology data, forming directed edges from parent nodes to child nodes. For symbolic nodes with nested relationships, such as subscripts / superscripts, fractional structures, or matrix structures, nested subtrees are created below the parent node, and the nesting type identifier is recorded. Through this method, the mathematical formula symbol nodes are organized layer by layer, constructing the initial mathematical formula structure tree.

[0091] Based on the operator dependency table and dimension constraint entries in the mathematical meta-knowledge constraint data, constraint matching is performed on the symbol nodes in the initial mathematical formula structure tree to determine the effective connection methods between symbol nodes that conform to operator dependencies and dimension constraints.

[0092] For each non-leaf node in the initial mathematical formula structure tree, its corresponding mathematical formula symbol type is extracted, and the corresponding operator dependency table in the mathematical meta-knowledge constraint data is queried to determine whether the connection between the node and its child nodes conforms to the operator atom number information and operation priority information in the meta-knowledge constraints. For example, when the parent node is a binary operator type symbol, it is checked whether its number of child nodes is 2; if it is a unary operator type symbol, it is checked whether its number of child nodes is 1; if the atom number requirement is not met, it is marked as an invalid connection and temporarily disconnected from the structure tree. Based on the dimensional consistency information in the dimensional constraint entries, dimensional consistency verification is performed on each parent-child node connection relationship, that is, it is determined whether the physical dimensions of the symbol nodes at both ends of the connection meet the constraint conditions defined in the meta-knowledge rule base. For example, when the parent node is an addition operator symbol, the dimensions of the two child nodes must be exactly the same; when the parent node is a multiplication operator symbol, the dimensions of the two child nodes are allowed to be multiplied. Through the above constraint matching and dimensional consistency verification, valid connection methods between symbol nodes that conform to operator dependency and dimensional constraint conditions are selected.

[0093] Based on effective connection methods, the initial mathematical formula structure tree is modified to generate a mathematical formula structure tree that integrates mathematical meta-knowledge constraints.

[0094] Remove all nodes marked as invalid connections from the initial mathematical formula structure tree. Based on the valid connection records, re-establish directed connections between parent and child nodes. For new connections, such as indirect dependencies not explicitly recorded in the symbolic topological relation data but derived from the operator dependency table, add new edges to connect the corresponding nodes. For nodes with dimensional conflicts, adjust the node's dimensional attributes according to the dimensional constraint correction formula to ensure the consistency of the mathematical formula structure tree at both structural and semantic levels. In the corrected mathematical formula structure tree, the connections between nodes simultaneously satisfy both topological relational logical constraints and mathematical meta-knowledge constraints, exhibiting higher structural accuracy and logical completeness.

[0095] Based on the mathematical formula structure tree that integrates mathematical meta-knowledge constraints, tree structure traversal and node relationship transformation are used to generate initial mathematical formula structure expression data containing implicit mathematical meta-knowledge constraints.

[0096] A depth-first search (DFS) or breadth-first search (BFS) algorithm is employed, starting from the root node and sequentially visiting each node in the mathematical formula structure tree, recording the node visit order. During each visit, the mathematical symbol type, node hierarchy number, adjacent node index information, and connection type information with the parent node are extracted and encoded sequentially to generate a mathematical formula structure expression sequence. During node relationship transformation, the order of the nodes obtained through traversal is adjusted according to the operation priorities recorded in the meta-knowledge constraints to ensure that the symbol arrangement order in the expression sequence is consistent with the mathematical semantics. For example, for a mathematical formula containing addition and multiplication operations, if the multiplication operator has higher priority than the addition operator, then in the final generated structure expression sequence, the node arrangement order of the multiplication operator and its operands should take precedence over the arrangement order of the addition operator.

[0097] The generated initial mathematical formula structure representation data represents the overall structure and implicit constraints of the mathematical formula in the form of a symbol sequence. This data includes the numbering information, symbol type, operator atom, dimensional information, operation priority information, and connection relationship information for each mathematical formula symbol node. For example, using a nested parenthesis encoding format or a label-based structured data format, the parent-child node relationships and operator constraint information in the mathematical formula structure tree are transformed into a text-based representation sequence.

[0098] S6: Based on the initial mathematical formula structure data, generate a formula encoding sequence through formula consistency verification and semantic constraint reconstruction, including:

[0099] Based on the mathematical formula structure tree that integrates mathematical meta-knowledge constraints, the operation dependency and dimensional constraint information of each mathematical formula symbol node are extracted to generate a mathematical formula node constraint verification table.

[0100] During the traversal of the mathematical formula structure tree, each node in the structure tree is visited sequentially using either a depth-first search (DFS) or breadth-first search (BFS) algorithm. When traversing each node, information such as the node's mathematical symbol type, number of operators, operation priority, physical dimension information of the symbol, and connection type with parent and child nodes is extracted. For each symbol node, based on its symbol type and the corresponding entry in the meta-knowledge rule base provided in the mathematical meta-knowledge constraint data, it is determined whether the symbol node meets the operator number requirement for that type. The consistency of dimensions between symbol nodes is checked, i.e., whether the physical dimensions between symbol nodes are consistent. If there are inconsistent dimensions between symbol nodes or the operator number requirement is not met, the connection between the symbol nodes is marked as a constraint conflict. Through the above methods, a mathematical formula node constraint verification table is generated, recording the constraint verification results of all symbol nodes.

[0101] Based on the mathematical formula node constraint verification table, the formula consistency verification is performed on each symbol node in the mathematical formula structure tree that integrates mathematical meta-knowledge constraints, and constraint conflicts of symbol nodes in the mathematical formula structure tree are identified and marked.

[0102] The process iterates through each record in the constraint verification table of mathematical formula nodes. For symbolic nodes marked as having constraint conflicts, it analyzes their dependencies on other symbolic nodes. For example, when there are non-compliance rules on dimensional consistency or mismatched operator atom numbers between symbolic nodes, the connection between parent and child nodes is traced back to identify the constraint conflict, and the specific location and type of the conflict are recorded. Using this method, constraint conflicts of all symbolic nodes in the mathematical formula structure tree can be accurately identified.

[0103] Perform semantic constraint reconstruction on the marked constraint conflict nodes to regenerate the final mathematical formula structure tree that satisfies the constraint conditions;

[0104] Conflicting nodes are categorized based on their constraint conflict type. If the conflict stems from a mismatch in operator atom numbers, the connection order between symbol nodes is adjusted to conform to the operational rules. If the conflict arises from inconsistent dimensions, the dimensions of the conflicting nodes are adjusted by referring to the dimension correction rules in the mathematical meta-knowledge constraint data to ensure that the physical meaning of the mathematical formula is satisfied. For example, when the child nodes of an addition symbol node are operands with different physical dimensions, the dimensions can be unified according to the dimension derivation formula to conform to the operational rules. Through these methods, the connection methods between conflicting nodes are corrected, ultimately generating a mathematical formula structure tree that meets all constraints. In the corrected final mathematical formula structure tree, the connection relationships between all symbol nodes satisfy both topological constraints and mathematical meta-knowledge constraints.

[0105] Based on the final mathematical formula structure tree that satisfies the constraints, the encoding transformation of the nodes of the mathematical formula structure tree is performed to generate a formula encoding sequence to represent the mathematical formula.

[0106] The modified mathematical formula structure tree is traversed using a tree structure traversal algorithm. Based on the node visit order and constraints, the type, position, and operational relationships of each symbol node are encoded into a series of symbol sequences. According to the tree's hierarchical structure and the relative positions of the nodes, the nodes are arranged from top to bottom and left to right, and the symbol type, operator atom, dimensional information, and connection relationships of each symbol node are recorded. The order of the symbol nodes is adjusted by incorporating the operational priority information between them to ensure that the mathematical formula's expression conforms to mathematical logic. For example, when the formula contains addition and multiplication operations, the node order of the multiplication operator and its operands is adjusted according to operator priority, making it take precedence over the addition operator. Through this method, the final generated formula encoding sequence accurately reflects the structure and semantics of the mathematical formula, including the number of each symbol node, symbol type, operator atom, dimensional information, operational priority, and connection relationships between symbol nodes.

[0107] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0108] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0109] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0111] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0113] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0115] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A mathematical formula recognition and encoding method, characterized in that, Includes the following steps: S1: Obtain the image data of the mathematical formula to be identified, perform symbol boundary extraction and preprocessing on the mathematical formula image data, and generate symbol feature expression data; S2: Based on the symbol feature expression data, perform visual spatial layout parsing and symbol type semantic classification respectively to generate formula layout structure data and symbol semantic classification data; S3: Based on the formula layout structure data, determine the topological relationships between symbols in the mathematical formula through graph structure analysis based on topological relationships, and generate symbol topological relationship data; S4: Based on the symbolic semantic classification data, through mathematical meta-knowledge mining, dimensional constraint relationships and operator dependency relationships between symbols are deduced to generate mathematical meta-knowledge constraint data; S5: Based on symbolic topological relation data and mathematical meta-knowledge constraint data, generate initial mathematical formula structure expression data containing implicit mathematical meta-knowledge constraints through decoding that fuses structural and semantic constraints. S6: Based on the initial mathematical formula structure, express the data, and generate the formula encoding sequence through formula consistency verification and semantic constraint reconstruction.

2. The mathematical formula recognition and encoding method according to claim 1, characterized in that, S1, specifically: Obtain the image data of the mathematical formula to be identified; The image data of the mathematical formula to be identified is subjected to image grayscale conversion and adaptive binarization. Based on the mathematical formula image data after adaptive binarization, the boundary regions of mathematical formula symbols are identified, and the position of each mathematical formula symbol in the mathematical formula image data is determined. Feature extraction is performed on the boundary region of each mathematical formula symbol to extract the shape, size, and spatial location features of the mathematical formula symbol and generate symbol feature representation data.

3. The mathematical formula recognition and encoding method according to claim 2, characterized in that, S2, specifically: Based on the shape, size, and spatial position characteristics of mathematical formula symbols in the symbol feature representation data, the relative positional relationships between mathematical formula symbols are identified, the two-dimensional layout structure of symbols in mathematical formulas is determined, and formula layout structure data is generated. Based on the shape and size features of mathematical formula symbols in the symbol feature representation data, the categories of mathematical formula symbols are identified and classified, the mathematical symbol type corresponding to each mathematical formula symbol is determined, and symbol semantic classification data is generated.

4. The mathematical formula recognition and encoding method according to claim 3, characterized in that, S3, specifically: Based on the formula layout structure data, the connection relationship between mathematical formula symbols is established by utilizing the relative positional relationship between mathematical formula symbols; Based on the connections between mathematical formula symbols, construct a topological graph with each mathematical formula symbol as a node; Based on the connection relationships between nodes in the topological graph, determine the hierarchical, adjacency, and nesting relationships between mathematical formula symbol nodes in the topological graph; Based on the hierarchical, adjacency, and nesting relationships between the mathematical formula symbol nodes in the topological relationship diagram, symbolic topological relationship data is generated.

5. The mathematical formula recognition and encoding method according to claim 4, characterized in that, S4, specifically: Based on the mathematical symbol type of each mathematical formula symbol in the symbol semantic classification data, query the preset meta-knowledge rule base to obtain the physical dimension information, operator meta-number information and operation priority information corresponding to the mathematical symbol type, and generate the initial table of symbol meta-knowledge. Based on the initial table of symbolic meta-knowledge, the operation rules are matched for symbol combinations in the symbolic semantic classification data to determine the operation associations between symbols and generate an operator dependency table. For each operational association in the operator dependency table, perform dimensional consistency derivation to establish dimensional constraint entries between symbols; Integrate the operator dependency table and dimension constraint entries to encode and generate mathematical meta-knowledge constraint data.

6. The mathematical formula recognition and encoding method according to claim 5, characterized in that, S5, specifically: Based on the hierarchical, adjacency, and nesting relationships between mathematical formula symbol nodes in the symbolic topology data, an initial mathematical formula structure tree is constructed. Based on the operator dependency table and dimension constraint entries in the mathematical meta-knowledge constraint data, constraint matching is performed on the symbol nodes in the initial mathematical formula structure tree to determine the effective connection methods between symbol nodes that conform to operator dependencies and dimension constraints. Based on effective connection methods, the initial mathematical formula structure tree is modified to generate a mathematical formula structure tree that integrates mathematical meta-knowledge constraints. Based on the mathematical formula structure tree that integrates mathematical meta-knowledge constraints, tree structure traversal and node relationship transformation are used to generate initial mathematical formula structure expression data containing implicit mathematical meta-knowledge constraints.

7. The mathematical formula recognition and encoding method according to claim 6, characterized in that, S6, specifically: Based on the mathematical formula structure tree that integrates mathematical meta-knowledge constraints, the operation dependency and dimensional constraint information of each mathematical formula symbol node are extracted to generate a mathematical formula node constraint verification table. Based on the mathematical formula node constraint verification table, the formula consistency verification is performed on each symbol node in the mathematical formula structure tree that integrates mathematical meta-knowledge constraints, and constraint conflicts of symbol nodes in the mathematical formula structure tree are identified and marked. Perform semantic constraint reconstruction on the marked constraint conflict nodes to regenerate the final mathematical formula structure tree that satisfies the constraint conditions; Based on the final mathematical formula structure tree that satisfies the constraints, the encoding transformation of the nodes in the mathematical formula structure tree is performed to generate a formula encoding sequence to represent the mathematical formula.

Images