Handwritten mathematical formula identification method
By combining deep learning technologies such as CNN, RNN and GNN, symbolic recognition and relationship optimization of handwritten mathematical formulas is solved, and the problems of low recognition accuracy and difficult processing of complex formulas in the existing technology are achieved, and efficient handwritten mathematical formula recognition is achieved.
Patent Information
- Application Number
- CN202510242898.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-03
- Publication Date
- 2025-06-20
AI Technical Summary
The existing handwritten mathematical formula recognition methods have problems with low recognition accuracy and difficulty in dealing with symbol nesting and formula structure in complex formulas.
Convolutional neural network (CNN) and recurrent neural network (RNN) are used to symbol recognition and model training for offline and online handwritten mathematical formulas, and symbolic relationships are optimized in combination with graph neural network (GNN), and LaTeX expressions are generated through recursive transformation methods.
It improves the recognition accuracy of handwritten mathematical formulas, can effectively deal with symbol nesting and formula structures in complex formulas, and provides a fully automated and efficient recognition solution.
Smart Images

Figure CN120182988A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of optical character recognition, and more specifically, to a method for recognizing handwritten mathematical formulas. Background Art
[0002] The technology of handwritten mathematical formula recognition is one of the important research directions in the current fields of computer vision and artificial intelligence. With the rise of industries such as intelligent education, online learning, and automated academic publishing, the demand for automatic recognition of handwritten mathematical formulas is increasing day by day.
[0003] The existing technologies have the following deficiencies:
[0004] Existing handwritten mathematical formula recognition methods have great limitations, resulting in the recognition accuracy not meeting the actual application requirements. The symbol parsing technology is usually rule-based or template-based, lacking in-depth understanding of the complex relationships between symbols and being difficult to meet the parsing requirements of different mathematical formulas. Automatically generating standard LaTeX expressions remains a technical difficulty. Existing technologies fail to provide a fully automated and efficient solution and are difficult to handle symbol nesting and formula structures in complex formulas.
[0005] In view of the above problems, the present invention proposes a solution. Summary of the Invention
[0006] In order to overcome the above-mentioned defects of the existing technologies, embodiments of the present invention provide a method for recognizing handwritten mathematical formulas to solve the problems raised in the above background art.
[0007] To achieve the above object, the present invention provides the following technical solutions:
[0008] A method for recognizing handwritten mathematical formulas, comprising:
[0009] Step 1: Obtain data of a handwritten mathematical formula and determine the data reception type;
[0010] Step 2: After obtaining the handwritten mathematical formula data, preprocess the data, and perform offline image formula processing and online note formula processing on the data obtained in the offline mode and the online mode respectively;
[0011] Step 3: Use a convolutional neural network (CNN) and a recurrent neural network (RNN) to perform symbol recognition and model training on offline symbols and online symbols respectively, predict their corresponding mathematical symbol categories, and assign corresponding symbol labels;
[0012] Step 4: After completing the symbol recognition, perform symbol parsing on the symbols in the formula to form an expression that conforms to mathematical logic.
[0013] In a preferred embodiment, data acquisition is divided into offline mode acquisition and online mode acquisition.
[0014] In a preferred embodiment, offline image formula processing includes the following steps:
[0015] Perform image normalization on the acquired image data, and the specific solution is as follows:
[0016] Step A1: Divide the image into several regions, and calculate the rgb mean value in each region through the rgb values corresponding to each pixel point in the region;
[0017] The brightness L is calculated by weighted averaging the RGB mean values of each region. The specific reference formula is: L = 0.299×R + 0.587×G + 0.114×B:
[0018] Set the grayscale coefficient S1 = 1.5 for the dark area with lower brightness; the grayscale coefficient S2 = 1 for the normal area with moderate brightness; the grayscale coefficient S3 = 0.75 for the bright area with higher brightness; the overall grayscale coefficient is S0; calculate the overall grayscale coefficient of the image based on the grayscale coefficients of each region and the area ratio of the region division. The specific reference formula is: S0 = (S1 + S2 + S3) / 3; substitute the grayscale coefficients of each region to obtain the grayscale coefficient S0 = 1.08 of the overall image;
[0019] Step A2: Calculate the spatial resolution of each region according to the pixel size, physical size, number of pixels in each region, and area ratio of the image;
[0020] The spatial resolution is divided into horizontal spatial resolution and vertical spatial resolution. The specific basis formula is: number of pixels / size; according to the Nyquist sampling theorem, the maximum frequency (highest distinguishable frequency) of the image is half of the spatial resolution; the specific basis formula is: horizontal / vertical frequency resolution = horizontal / vertical spatial resolution / 2; obtain the horizontal frequency resolution of each region;
[0021] Specifically according to the following formula:
[0022] Detail retention degree = (spatial resolution + frequency resolution) / 2;
[0023] Calculate the binarization coefficient of each region by combining the brightness value and detail retention degree of each region. Specifically according to the following formula:
[0024] Binarization coefficient of each region = (L / maximum brightness value) × (detail retention degree / maximum detail retention degree);
[0025] In this embodiment, the maximum brightness value is taken as 255 (representing the maximum brightness value in the image), and the maximum detail retention degree is taken as 200; according to the binary coefficient and area ratio of each region, the binary coefficient of the overall image can be calculated; specifically, based on the following formula:
[0026]
[0027] Step A3: Perform comprehensive analysis based on the combination of the overall image grayscale coefficient and the binary coefficient to achieve image standardization; first, based on the weighted calculation formula of the grayscale coefficient and the binary coefficient, obtain the comprehensive standardization coefficient; set the importance weights of the grayscale coefficient and the binary coefficient in image standardization as W and R respectively, and the sum of their weights is 1 (i.e., W + R = 1); through calculation, the grayscale coefficient weight W = 0.6; the binary coefficient weight R = 0.4;
[0028] Specifically, calculate the overall standardization coefficient Z according to the following formula:
[0029] Z = W × S0 + R × overall binary coefficient;
[0030] Multiply the grayscale value of the original image by Z to scale the brightness; in this embodiment, set the grayscale value of the original image as Y; the adjusted new grayscale value is H; according to the formula H = Y × Z, substitute the data into the calculation to obtain the adjusted grayscale value;
[0031] The key to binary processing is to set an appropriate threshold. In this embodiment, the adjustment of the threshold is affected by the image grayscale value and the binary coefficient;
[0032] Online note formula processing includes the following steps:
[0033] Step B1: Perform normalization processing on the obtained note trajectory, and scale the collected note coordinates to a unified range, such as [0, 1];
[0034] Sort the coordinate points of each stroke in ascending order according to the timestamp using merge sort to ensure that the coordinate points of each stroke are arranged in chronological order;
[0035] Step B2: Use the spatial jump segmentation method to segment the strokes of the obtained image data; calculate the distance between the coordinates of each stroke. In this embodiment, set the spatial distance threshold as 50 pixels. When the spatial distance between strokes is greater than 20 pixels, it is determined that these two symbols are independent.
[0036] In a preferred embodiment, for offline symbols: First, pixel normalization and symbol segmentation are performed on the preprocessed image. The convolutional neural network (CNN) convolutional layer is used for the input image data to extract the features of the symbol edges and shapes, and a high-dimensional feature representation of the symbol is output; the extracted features are mapped to symbol categories through a fully connected layer, and each category is labeled with a symbol label respectively;
[0037] For online symbols: The recurrent neural network (RNN) is used to process online symbols. First, the hidden state of the recurrent neural network (RNN) records the dynamic changes of the strokes, and the coordinate data of the input time series is output as the temporal features of the symbol. The extracted features are mapped to symbol categories through a classification layer, and each category is labeled with a symbol label respectively.
[0038] In a preferred embodiment, first, the geometric features of each symbol are extracted, including its central coordinates, bounding box size, and relative position with other symbols, so as to judge the arrangement relationship of the symbols;
[0039] One-hot encoding is used to define the grammatical feature data of each symbol in the mathematical formula 3x + 4 = 7, and the multi-dimensional feature vectors of each symbol are calculated in combination with the arrangement relationship data;
[0040] According to the representative symbol features, a syntax tree of the formula is constructed, and the graph neural network (GNN) is used to further optimize the symbol relationship;
[0041] The recursive transformation method is used to transform each node and subtree into a LaTeX expression.
[0042] The present invention proposes a method for recognizing handwritten mathematical formulas. The handwritten mathematical formulas are obtained in offline and online modes respectively. Image normalization is performed on the two modes from the perspectives of pixel rgb, illumination conditions, spatial resolution, frequency resolution, and handwritten trajectories. Then, the convolutional neural network (CNN) and the recurrent neural network (RNN) are used to perform symbol recognition and model training on offline symbols and online symbols respectively. The GNN is used to further optimize the symbols, and the LaTeX expression is obtained in a recursive manner, improving the recognition accuracy of handwritten mathematical formulas. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 It is a flowchart of a method for recognizing handwritten mathematical formulas according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0044] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.
[0045] Embodiment
[0046] The present invention discloses a method for recognizing handwritten mathematical formulas, including the following steps:
[0047] Step 1: Obtain the data of the handwritten mathematical formula and determine the data reception type.
[0048] The data acquisition is divided into offline mode acquisition and online mode acquisition. Offline mainly provides in the form of images, and online mainly provides in the form of note trajectories.
[0049] Specifically, for offline mode acquisition: First, the user takes a picture of a paper note containing a mathematical formula with a smart phone, provides the image file to the interface and verifies the validity of the file to obtain the image file data. The data includes: image size: 1920×1080 pixels; color mode: RGB; image size: 5 inches × 10 inches; For online mode acquisition: First, the user uses a stylus to input a mathematical formula on a tablet and records the input data of each stroke. The data content includes: stroke coordinate point sequence, timestamp.
[0050] It should be noted that the note input application is a customized formula input mode. In this embodiment, the user's writing trajectory is recorded in real time according to the default formula input mode.
[0051] Step 2: After obtaining the handwritten mathematical formula data, preprocess the data, and perform offline image formula processing and online note formula processing on the data obtained in the offline mode and the online mode respectively.
[0052] The offline image formula processing includes the following steps:
[0053] Perform image standardization on the obtained image data. The specific solution is as follows:
[0054] First, convert the color or RGB image into a grayscale image to reduce the data dimension and remove color interference; further, convert the grayscale image into a black and white image to make the handwriting of the formula clearer and at the same time compress the data volume.
[0055] It should be noted that the image data captured in different environments and the original image resolution will both affect image grayscale conversion. The image data includes the RGB values of the image and the lighting conditions in different shooting environments; the original image resolution includes spatial resolution and frequency resolution.
[0056] In this embodiment, the image is divided into different regions according to different lighting conditions, RGB values, spatial resolutions, and frequency resolutions, and the image grayscale coefficient and binarization coefficient are determined; the image standardization is realized by comprehensively analyzing the image grayscale coefficient and the binarization coefficient.
[0057] The specific solution is as follows:
[0058] Step A1: Divide the image into several regions, and calculate the RGB mean value in each region through the RGB values corresponding to each pixel point in the region to obtain the corresponding RGB mean value data in each region;
[0059]
[0060] It should be noted that the division method and result of the several regions are not unique. For example, the overall image can be evenly divided in a certain direction, such as dividing from top to bottom, from left to right, etc. In this embodiment, the picture is divided into three parts: upper, middle, and lower according to the division method from top to bottom.
[0061] Evaluate the lighting conditions by calculating the brightness value in the region. It should be noted that the brightness L is obtained by weighted averaging the RGB means of each region. The specific reference formula is: L = 0.299×R + 0.587×G + 0.114×B. The following lighting condition statuses of each region are obtained through calculation:
[0062]
[0063]
[0064] Set different grayscale coefficients for each region according to the RGB mean value and brightness value difference of each region.
[0065] Specifically, according to the brightness value of each region, set the grayscale coefficient S1 = 1.5 for the dark region with lower brightness; the grayscale coefficient S2 = 1 for the normal region with moderate brightness; the grayscale coefficient S3 = 0.75 for the bright region with higher brightness; the overall grayscale coefficient is S0; calculate the overall grayscale coefficient of the image based on the grayscale coefficients of each region and the area ratio of the region division. The specific reference formula is: S0 = (S1 + S2 + S3) / 3; substitute the grayscale coefficients of each region to obtain the overall grayscale coefficient S0 = 1.08 of the image.
[0066] Step A2: Calculate the spatial resolution of each region based on the pixel size, physical size, number of pixels in each region, and area ratio of the image. First, determine the height and width of each region. In this embodiment, each region is evenly divided from top to bottom. Therefore, the height of each region is 5 / 3 = 1.67 inches, and the width remains 10 inches unchanged.
[0067] Spatial resolution refers to the number of pixels per inch (PPI). Based on the resolution and size of the image, we can calculate the spatial resolution of the image. Spatial resolution is divided into horizontal spatial resolution and vertical spatial resolution, specifically according to the formula: number of pixels / size. The horizontal spatial resolution of each region is obtained as: 1980 / 10 = 198 PPI; the vertical spatial resolution is (1080 / 3) / 1.67 = 72 PPI.
[0068] Frequency resolution is directly related to spatial resolution. In the frequency domain, frequency resolution represents the minimum frequency difference that can be distinguished. According to the Nyquist sampling theorem, the maximum frequency (highest distinguishable frequency) of the image is half of the spatial resolution. Therefore, for the frequency resolution of the image and each region, specifically according to the formula: horizontal / vertical frequency resolution = horizontal / vertical spatial resolution / 2. The horizontal frequency resolution of each region is obtained as 198 / 2 = 99 cycles / inch; the vertical frequency resolution is 216 / 2 = 108 cycles / inch.
[0069] Calculate the binarization coefficient of each region based on the spatial resolution and frequency resolution of each region.
[0070] It should be noted that the detail retention degree of each region can be calculated based on the spatial resolution and frequency resolution of each region. It is closely related to the binarization coefficient. The higher the value of the detail retention degree, the richer the details in the region. Binarization requires a higher threshold to avoid detail loss.
[0071] Specifically according to the following formula:
[0072] Detail retention degree = (spatial resolution + frequency resolution) / 2;
[0073] The detail retention degree of each region is obtained as (198 + 99) / 2 = 148.5;
[0074] Next, calculate the binarization coefficient of each region by combining the brightness value and detail retention degree of each region. Specifically according to the following formula:
[0075] Binarization coefficient of each region = (L / maximum brightness value) × (detail retention degree / maximum detail retention degree);
[0076] In this embodiment, the maximum brightness value is taken as 255 (representing the maximum brightness value in the image), and the maximum detail retention degree is taken as 200. The binarization coefficient data of each region is calculated as follows:
[0077] Region binarization coefficient
[0078] Upper 0.104
[0079] Middle 0.12
[0080] Lower 0.15
[0081] Based on the binarization coefficients and area ratios of each region, the binarization coefficient of the overall image can be calculated. Specifically, it is based on the following formula:
[0082]
[0083] Substituting the binarization coefficient data of each region, the binarization coefficient of the overall image is obtained as 0.1247.
[0084] Step A3: Based on the combination of the overall image grayscale coefficient and the binarization coefficient, comprehensive analysis is carried out to achieve image standardization. It should be noted that the grayscale coefficient and the binarization coefficient affect the brightness adjustment and detail retention of the image respectively. The grayscale coefficient controls the overall brightness adjustment of the image, while the binarization coefficient is used to judge the degree of detail retention to avoid detail loss.
[0085] First, based on the weighted calculation formula of the grayscale coefficient and the binarization coefficient, a comprehensive standardization coefficient can be obtained. Assume that the importance weights of the grayscale coefficient and the binarization coefficient in image standardization are W and R respectively, and the sum of their weights is 1 (i.e., W + R = 1). Through calculation, the grayscale coefficient weight W = 0.6; the binarization coefficient weight R = 0.4.
[0086] Specifically, the overall standardization coefficient Z is calculated according to the following formula:
[0087] Z = W × S0 + R × overall binarization coefficient;
[0088] Substituting the data, the overall standardization coefficient Z = 0.69788 can be obtained.
[0089] The overall normalization coefficient Z mainly affects the overall brightness adjustment of the image and ensures appropriate brightness by adjusting the grayscale values of the image. Therefore, Z is applied to adjust the grayscale value at each pixel of the image. Specifically, we multiply the grayscale value of the original image by Z to scale the brightness. In this embodiment, the grayscale value of the original image is set as Y; the new adjusted grayscale value is H; according to the formula H = Y × Z, substituting the data for calculation to obtain the adjusted grayscale value. Specifically, when the original grayscale value Y of a pixel in the image is 180, the adjusted grayscale value calculated according to the formula is H = 180 × 0.69788 = 125.64.
[0090] By adjusting the overall brightness of the image, it can still maintain good visibility under different lighting conditions. For darker or overly bright areas, coefficient adjustment is used to avoid loss of details caused by too low or too high brightness.
[0091] The key to binarization processing is to set an appropriate threshold. In this embodiment, the adjustment of the threshold is affected by the grayscale value of the image and the binarization coefficient. Specifically, in this embodiment, the globally default binarization threshold is 128, and the binarization threshold is adjusted in combination with the adjusted grayscale value H and the binarization coefficient.
[0092] Specifically, according to the following formula:
[0093]
[0094] Among them, k is an adjustment constant (usually set to 0.5, which specifically changes according to different specific environments). The binarization coefficient reflects the degree of detail retention. The larger the value, the higher the threshold required, and the smaller the value, the lower the threshold required.
[0095] After obtaining the adjusted threshold data, the image is binarized. Specifically, according to the following formula:
[0096]
[0097] Among them, here, Y is the original grayscale value of the image, and T is the adjusted binarization threshold. Each pixel will be converted to black (0) or white (255) according to this threshold.
[0098] Online note formula processing includes the following steps:
[0099] Step B1: Normalize the obtained note trajectory, and scale the collected note coordinates to a unified range, such as [0, 1].
[0100] Specifically, the original data obtained from the handwriting device is as shown in the following example:
[0101] The writer wrote the mathematical expression "3 + 4", and the original stroke coordinates and timestamps are as follows:
[0102] Stroke 1: [(100, 200), (150, 250), (200, 300)]
[0103] Timestamps: [t1, t2, t3]
[0104] Stroke 2: [(220, 320), (250, 360), (300, 410)]
[0105] Timestamps: [t4, t5, t6]
[0106] Stroke 3: [(50, 80), (70, 110), (90, 130)]
[0107] Timestamps: [t7, t8, t9]
[0108] Normalize the coordinates of each stroke using the formula: In the formula, Normalized_Score is the value after normalization.
[0109] The normalized coordinates are obtained through calculation:
[0110] Stroke 1: [(0.2, 0.4), (0.4, 0.6), (0.6, 0.8)]
[0111] Timestamps: [t1, t2, t3]
[0112] Stroke 2: [(0.68, 0.8), (0.72, 0.9), (0.8, 1)]
[0113] Timestamps: [t4, t5, t6]
[0114] Stroke 3: [(0, 0), (0.08, 0.1), (0.16, 0.2)]
[0115] Timestamps: [t7, t8, t9]
[0116] The order of the stylus trajectory is very important. Only by arranging the strokes in the correct order can the symbol segmentation and recognition in subsequent steps be accurate. Therefore, the coordinates of each stroke need to be normalized not only according to the X-axis and Y-axis but also sorted according to the writing time sequence.
[0117] Each stroke coordinate point has a timestamp. The coordinate points of each stroke are sorted using merge sort according to the timestamps to ensure that the coordinate points of each stroke are arranged in time sequence. It should be noted that merge sort belongs to a conventional technical means and will not be elaborated here.
[0118] Step B2: Use the space jump segmentation method to segment the acquired image data. Utilize the spatial distance formula: Calculate the distances between the coordinates of each stroke. In this embodiment, the spatial distance threshold is set to 50 pixels. When the spatial distance between strokes is greater than 20 pixels, it is determined that these two symbols are independent.
[0119] Specifically, in the calculation example, the last coordinate of stroke 1 is (160, 210), and the last coordinate of stroke 2 is (200, 250). Substituting the data into the formula, the spatial distance between the two coordinates is 56.57. It is determined that segmentation is required between stroke 1 and stroke 2, and "3" and "+" are two independent symbols. It should be noted that substituting data into the calculation to determine whether symbols are independent belongs to conventional technical means and will not be elaborated here.
[0120] Step 3: Use a convolutional neural network (CNN) and a recurrent neural network (RNN) to perform symbol recognition and model training on offline symbols and online symbols respectively, predict their corresponding mathematical symbol categories, and assign corresponding symbol labels.
[0121] It should be noted that the input for offline symbol recognition is a static image. The convolutional neural network (CNN) is used to process offline symbols, and spatial features of the image are learned through convolution and pooling operations to identify the shape and structure of the symbols.
[0122] Specifically, first perform pixel normalization and symbol segmentation on the preprocessed image. The following is an example of the segmentation result:
[0123] Symbol "3": Corresponding pixel matrix [[0,0,1],[0,1,1],...]
[0124] Symbol "x": Corresponding pixel matrix [[0,1,0],[1,0,1],...];
[0125] For the input image data, use the convolutional layer of the convolutional neural network (CNN) to extract the features of the symbol edges and shapes, and output the high-dimensional feature representation of the symbol. Map the extracted features to symbol categories through a fully connected layer and assign symbol labels to each category respectively.
[0126] Specifically, the results of the convolutional neural network (CNN) for recognizing the handwritten mathematical formula contained in the image are as follows:
[0127] Symbol "3": Classified as "3"; Labeled as "3".
[0128] Symbol "x": Classified as "x"; Labeled as "x".
[0129] Symbol "+": Classified as "+"; Labeled as "+".
[0130] The input for online symbol recognition is the handwritten trajectory. A recurrent neural network (RNN) is used to process the online symbols. First, the hidden state of the recurrent neural network (RNN) records the dynamic changes of the strokes, such as the temporal characteristics of the start, turning, and end of the strokes. The coordinate data of the input time series is output as the temporal features of the symbol, and the extracted features are mapped to symbol categories through a classification layer and symbol labels are assigned to each category respectively.
[0131] Specifically, the results of the recurrent neural network (RNN) for handwritten trajectory recognition are as follows:
[0132] Symbol "3": Classified as "3"; Label is "3".
[0133] Symbol "x": Classified as "x"; Label is "x".
[0134] Symbol "+": Classified as "+"; Label is "+".
[0135] Combining the recognition of offline and online symbols gives the following examples:
[0136] Offline symbol recognition result (CNN):
[0137] Symbol sequence ["3", "x", "+", "4", "=", "7"]
[0138] Online symbol recognition result (RNN):
[0139] Symbol sequence ["3", "x", "+", "4", "=", "7"]
[0140] It should be noted that using a convolutional neural network (CNN) and a recurrent neural network (RNN) model for symbol recognition and model training are conventional technical means and will not be described in detail here.
[0141] In this embodiment, the offline symbols and online symbols will be uniformly classified into symbol data after symbol recognition.
[0142] Step 4: After completing symbol recognition, perform symbol parsing on the symbols in the formula and then combine the symbols into an expression that conforms to mathematical logic.
[0143] Step C1: First, extract the geometric features of each symbol, including its center coordinates, bounding box size, and relative position with other symbols, and use this to judge the arrangement relationship of the symbols.
[0144] Specifically, after obtaining the symbol recognition classification in step 3, the following symbol data is obtained:
[0145] Symbol 3: A constant, located within the bounding box [10, 10, 20, 20], with a width of 10 and a height of 10.
[0146] Symbol x: A variable, located within the bounding box [25, 10, 35, 20], with a width of 10 and a height of 10.
[0147] And so on for other symbols.
[0148] Using the center coordinate formula Calculate the center point of each symbol. The following example data is obtained:
[0149] Symbol 3: Center coordinates (15, 15)(15, 15)(15, 15).
[0150] Symbol x: Center coordinates (30, 15)(30, 15)(30, 15).
[0151] And so on for other symbols.
[0152] By calculating the width w = x2 - x1 and height h = y2 - y1, the following example data is obtained:
[0153] Symbol 3 has a width of 10 and a height of 10.
[0154] Symbol x has a width of 10 and a height of 10.
[0155] And so on for other symbols.
[0156] Analyze the horizontal and vertical positions of each symbol relative to adjacent symbols based on the center coordinates of the symbols. The following example data is obtained:
[0157] The symbol closest to the right of Symbol 3 is x, with a horizontal distance of 30 - 20 = 1030 - 20 = 1030 - 20 = 10.
[0158] The symbol closest to the right of Symbol x is +, with a horizontal distance of 45 - 35 = 1045 - 35 = 1045 - 35 = 10.
[0159] The symbol closest to the right of Symbol + is 4, with a horizontal distance of 60 - 50 = 1060 - 50 = 1060 - 50 = 10.
[0160] And so on for other symbols.
[0161] Based on the geometric characteristics of each symbol and the symbol classification information, further analyze the mathematical arrangement relationship of the symbols: Symbol 3 and x are adjacent, x is a variable, and the system determines that 3x represents a multiplication relationship; Symbol + is located between 3x and 4, indicating an addition operation between them; Symbol = connects the left - hand expression 3x + 4 and the right - hand value 7, indicating an assignment relationship.
[0162] The mathematical functions of different symbols determine their roles in the syntax. For example, symbols such as operators, function symbols, variables, constants, and parentheses participate in the syntax in different ways. In this embodiment, representative symbol features are obtained by combining the syntax of the symbols and their arrangement relationships.
[0163] Based on the syntax feature data and arrangement relationship data of the obtained mathematical formula, a multi-dimensional feature vector of the mathematical formula is comprehensively analyzed, that is, the representative symbol feature.
[0164] Specifically, one-hot encoding is used to define the syntax feature data of each symbol in the mathematical formula 3x + 4 = 7 and calculate the multi-dimensional feature vector of each symbol in combination with the arrangement relationship data.
[0165] The following example data is obtained:
[0166] Symbol 3x: Comprehensive feature vector: [1, 0, 0, 0, 0, 0.1, 0].
[0167] Symbol +: Comprehensive feature vector: [0, 1, 0, 0, 0, 0.2, 0.1].
[0168] And so on for other symbols.
[0169] It should be noted that the technologies used above are all conventional technologies and will not be elaborated here.
[0170] Step C2: Construct a syntax tree of the formula according to the representative symbol features, and use a graph neural network (GNN) to further optimize the symbol relationships.
[0171] Specifically, based on the obtained multi-dimensional feature data, the following preliminary syntax tree can be constructed:
[0172] Root node =.
[0173] Left subtree +.
[0174] Sub-node of the left subtree 3x.
[0175] Sub-node of the right subtree 4.
[0176] Right subtree 7.
[0177] The graph neural network propagates and aggregates information through the connection relationships between nodes, optimizing the node features, so that the representation of each node better conforms to the global structure and syntax relationships of the graph. In the graph neural network (GNN), the features of each node affect the neighboring nodes connected to it, and the features of the neighboring nodes also affect the current node.
[0178] Specifically, first, each symbol in the formula is transformed into a node in the graph, and the mathematical relationships between the symbols are transformed into edges in the graph, obtaining the following data:
[0179] Nodes:
[0180] 3x, +, 4, =, 7;
[0181] Edges:
[0182] There is an edge between 3x and +, indicating that there is a multiplication operation relationship between 3x and +;
[0183] There is an edge between + and 4, indicating that + is the addition operator connecting 3x and 4;
[0184] There is an edge between 4 and =, indicating that both sides of the equation are connected;
[0185] There is an edge between = and 7, indicating that the equal sign connects 4 and 7.
[0186] The working principle of the Graph Neural Network (GNN) is to gradually optimize the node features through multiple rounds of iteration. After each round of iteration, the features of the nodes will more accurately reflect the relationships between the symbols. For the symbol nodes in the formula, each node will pass its syntactic category and position features to its neighbor nodes (i.e., other nodes connected to it by edges). The features received by the neighbor nodes will be aggregated. The features after message aggregation will be further processed through update functions (such as fully connected layers, non-linear activation functions) to update the features of the nodes. This process enables each node to not only retain its own information but also incorporate information from neighbor nodes.
[0187] Specifically, in the first round:
[0188] 3x receives information from + and updates its features, indicating its relationship with the operator.
[0189] + receives information from 3x and 4 and updates its features, indicating that it is an addition symbol.
[0190] 4 receives information from + and updates its features, indicating that it is the other operand of the addition operation.
[0191] = receives information from + and 7 and updates its features, indicating that it is the equality symbol connecting two expressions.
[0192] In the second round:
[0193] 3x receives more information from + and 4 and updates its features, strengthening its syntactic representation of the addition relationship.
[0194] + receives more precise information from 3x and 4 and strengthens its role as an addition operator.
[0195] 4 receives more information, further clarifying its role as the constant term in the addition operation.
[0196] = receives more information, optimizing its feature of connecting the expressions on both sides as an equality sign.
[0197] After several rounds of information passing and node feature updates, the features of the symbol nodes will contain more accurate information:
[0198] The node feature of 3x will contain more information about it being a variable term and the left operand of the addition.
[0199] The node feature of + will contain that it is an addition symbol and connects the two operands 3x and 4.
[0200] The node feature of 4 will contain that it is a constant term and the right operand of the addition.
[0201] The node feature of = will contain that it is an equality sign and connects the addition expression and the constant 7.
[0202] The node feature of 7 will contain that it is a constant and the right side of the equation.
[0203] The symbol relationships optimized by the graph neural network (GNN) will make the construction of the syntax tree clearer and more precise. The specific example is as follows:
[0204]
[0205] Among them, the root node: =; the left subtree: represents the addition operation 3x + 4; the right subtree: represents the constant 7
[0206] Step C3: Use the method of recursive transformation to transform each node and subtree into a LaTeX expression.
[0207] It should be noted that the core idea of recursive transformation is: starting from the bottom layer (leaf nodes) of the tree, gradually transform the content of the nodes into LaTeX code until reaching the root node of the tree. The content of each node (such as operator or constant) will construct a LaTeX expression according to its syntax role (such as operator, variable, constant, etc.) and its relationship with adjacent nodes (such as operation order).
[0208] Specifically, first, a mathematical formula syntax tree is constructed based on the symbol relationships optimized by a graph neural network (GNN), and it is processed step by step from the leaf nodes to the root node: 3x is a leaf node, and its LaTeX representation is: 3x; 4 is a leaf node, and its LaTeX representation is: 4; 7 is a leaf node, and its LaTeX representation is: 7. + is an operator node, and the operator node needs to recursively call the LaTeX expressions of its child nodes and merge them: its LaTeX representation is: 3x + 4; = is an operator node, and the root node = is the top-level node of the entire syntax tree, and its LaTeX expression is obtained by recursively merging the results of the left and right subtrees. Its LaTeX representation is: 3x + 4 = 7.
[0209] It should be noted that the processing of each node and the recursive call are conventional technical means and will not be elaborated here.
[0210] The above formulas are all dimensionless and take their numerical calculations. The formula is a formula obtained by collecting a large amount of data for software simulation to approximate the real situation as closely as possible. The preset parameters in the formula are set by those skilled in the art according to the actual situation.
[0211] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product.
[0212] Those of ordinary skill in the art can realize that the modules and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application of the technical solution and the inventive constraints. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.
[0213] The above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application, and all should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.
[0214] Finally: The above is only the preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for recognizing handwritten mathematical formulas. It is characterized by; including: Step 1: Obtain the data of the handwritten mathematical formula and determine the data receiving type; Step 2: After obtaining the handwritten mathematical formula data, preprocess the data, and perform offline image formula processing and online note formula processing on the data obtained in the offline mode and the online mode respectively; Step 3: Use convolutional neural network (CNN) and recurrent neural network (RNN) to perform symbol recognition and model training on offline symbols and online symbols respectively, predict their corresponding mathematical symbol categories and label them with corresponding symbols; Step 4: After completing symbol recognition, perform symbol analysis on the symbols in the formula and then combine the symbols into expressions that conform to mathematical logic.
2. A handwritten mathematical formula recognition method according to claim 1, characterized in that: Data acquisition is divided into offline mode acquisition and online mode acquisition.
3. A handwritten mathematical formula recognition method according to claim 2, characterized in that: Offline image formula processing includes the following steps: The acquired image data is normalized. The specific scheme is as follows: Step A1: Divide the image into several regions, and calculate the RGB mean value in each region through the RGB value corresponding to each pixel point in each region; The brightness L is calculated by weighted average of the RGB mean values of each area, according to the formula: L = 0.299 × R + 0.587 × G + 0.114 × B: The grayscale coefficient of the dark area with low brightness is set to S1=1.5; the grayscale coefficient of the normal area with moderate brightness is set to S2=1; the grayscale coefficient of the bright area with high brightness is set to S3=0.75; the overall grayscale coefficient is S0; the grayscale coefficient of the overall image is calculated based on the grayscale coefficient of each area and the area ratio of the area division, specifically according to the formula: S0=(S1+S2+S3) / 3; substituting the grayscale coefficient of each area into the grayscale coefficient, the grayscale coefficient of the overall image is obtained to be S0=1.08; Step A2: Calculate the spatial resolution of each region based on the pixel size, physical size, number of pixels in each region, and area ratio of the image; Spatial resolution is divided into horizontal spatial resolution and vertical spatial resolution, which is based on the formula: number of pixels / size; according to the Nyquist sampling theorem, the maximum frequency (highest recognizable frequency) of the image is half of the spatial resolution; based on the formula: horizontal / vertical frequency resolution = horizontal / vertical spatial resolution / 2; obtain the horizontal frequency resolution of each area; The detail retention of each area can be calculated based on the spatial resolution and frequency resolution of each area, which is closely related to the binarization coefficient. According to the following formula: Detail retention = (spatial resolution + frequency resolution) / 2; The binarization coefficient of each area is calculated by combining the brightness value of each area and the detail retention degree, according to the following formula: Binarization coefficient of each region = (L / maximum brightness) × (detail retention / maximum detail retention); In this embodiment, the maximum brightness is 255 (representing the maximum brightness in the image), and the maximum detail retention is 200; the binarization coefficient of the entire image can be calculated according to the binarization coefficient and area ratio of each region; specifically according to the following formula: Step A3: Perform comprehensive analysis based on the overall image grayscale coefficient and the binarization coefficient to achieve image standardization; first, based on the weighted calculation formula of the grayscale coefficient and the binarization coefficient, obtain the comprehensive standardization coefficient; set the importance weights of the grayscale coefficient and the binarization coefficient in image standardization to W and R respectively, and the sum of the weights of the two is 1 (i.e., W+R=1); calculate and obtain the grayscale coefficient weight W=0.6; the binarization coefficient weight R=0.4; The overall standardized coefficient Z is calculated according to the following formula: Z = W × S0 + R × overall binarization coefficient; The gray value of the original image is multiplied by Z to scale the brightness; in this embodiment, the gray value of the original image is set to Y; the adjusted new gray value is H; according to the formula H=Y×Z, the data is substituted into the calculation to obtain the adjusted gray value; The key to binarization is to set an appropriate threshold. In this embodiment, the threshold is adjusted by the image gray value and the binarization coefficient. Online note formula processing includes the following steps: Step B1: normalize the acquired note tracks and scale the collected note coordinates to a uniform range, such as [0, 1]; Sort the coordinate points of each stroke by timestamp using merge sort to ensure that the coordinate points of each stroke are arranged in chronological order; Step B2: Use the space jump segmentation method to segment the acquired image data into strokes; calculate the distance between the coordinates of each stroke. In this embodiment, the space distance threshold is set to 50 pixels. When the space distance between the strokes is greater than 20 pixels, it is determined that the two symbols are independent.
4. A method for recognizing handwritten mathematical formulas according to claim 3, characterized in that ; Offline symbol: First, the preprocessed image is pixel normalized and the symbol is segmented. The convolutional neural network (CNN) convolutional layer is used to extract the edge and shape features of the symbol for the input image data, and the high-dimensional feature representation of the symbol is output; The extracted features are mapped to symbolic categories through a fully connected layer and each category is labeled with a symbol; Online symbols: Use a recurrent neural network (RNN) to process online symbols. First, the hidden state of the RNN records the dynamic changes of the strokes, outputs the input time series coordinate data as the temporal features of the symbol, maps the extracted features to symbol categories through the classification layer, and labels each category with symbols.
5. A handwritten mathematical formula recognition method according to claim 4, characterized in that: First, the geometric features of each symbol are extracted, including its center coordinates, bounding box size, and relative position to other symbols, so as to determine the arrangement relationship of the symbols; One-hot encoding is used to define the grammatical feature data of each symbol in the mathematical formula 3x+4=7 and the multi-dimensional feature vector of each symbol is calculated in combination with the arrangement relationship data; The syntax tree of the formula is constructed based on representative symbolic features, and the symbolic relationship is further optimized using graph neural network (GNN); A recursive transformation method is used to convert each node and subtree into a LaTeX expression.
