Formula structural analysis and semantic understanding method and system for PDF document

Abstract

The application discloses a formula structural analysis and semantic understanding method and system for a PDF document, relates to the technical field of intelligent document processing, and comprises the following steps: sequentially performing rendering, normalization and tilt correction processing on a pre-acquired PDF document to obtain a preprocessed document image; based on a context information fusion mechanism and in combination with a pre-set formula region detection model, detecting the preprocessed document image to obtain a formula image; and based on an attention mechanism and by using a pre-set formula analysis model, performing structural analysis on the formula image to obtain two-dimensional structural information of the formula.The application adopts a deep learning model to perform formula detection and analysis, has high precision and high robustness, realizes the structuring of the formula, improves information utilization efficiency, and reduces artificial cost.

Description

Technical Field

[0001] This invention relates to the field of intelligent document processing technology, and more specifically, to a method and system for structured parsing and semantic understanding of formulas for PDF documents. Background Technology

[0002] PDFs typically do not contain semantic information about formulas; they only record the appearance of the formulas, not their meaning. For formula editing, reuse, and retrieval, the formulas in the PDF must be converted back into structured code. However, current methods for structured parsing and semantic understanding of formulas in PDF documents have many critical shortcomings, making it difficult to meet practical usage needs.

[0003] Formula location and extraction are inaccurate. Formulas in PDF documents are usually in the form of images or vector graphics, rather than editable text. Existing OCR technology struggles to accurately identify and locate these formula areas, especially when formulas are mixed with ordinary text, or when formulas span multiple lines or columns. The positioning accuracy is low, and it is easy to miss or falsely detect formulas.

[0004] Complex formula formats are difficult to parse. Formulas in PDF documents have complex and diverse formats, including nested structures such as fractions, subscripts and superscripts, square roots, summation symbols, matrices, and curly braces. Existing formula recognition methods mostly rely on traditional image segmentation and symbol classification, which struggle to accurately handle these complex two-dimensional layout structures, resulting in parsed results that do not conform to the correct grammatical structure of the formula.

[0005] The parsed results are difficult to use in a structured way. Even if the symbols in the formula can be identified, the traditional recognition results are usually simple strings, such as E=mc2, losing two-dimensional structural information (such as c being a superscript). Such unstructured results are difficult to further process by computer programs, such as for symbol editing, typesetting, or reuse in other software.

[0006] Poor robustness to low-quality images and scanned documents. PDF documents come from diverse sources, some being high-quality vector graphics and others low-resolution scans. Existing methods perform poorly in recognizing low-quality images and are easily affected by noise, blur, and other factors.

[0007] No effective solutions have yet been proposed to address the problems in the relevant technologies. Summary of the Invention

[0008] To address the problems in related technologies, this invention proposes a method and system for structured parsing and semantic understanding of formulas in PDF documents, aiming to solve the problems of inaccurate formula location, difficulty in structure parsing, and unstructured parsing results in PDF documents.

[0009] Therefore, the specific technical solution adopted by the present invention is as follows:

[0010] According to one aspect of the present invention, a method for structured parsing and semantic understanding of formulas in PDF documents is provided, the method comprising the following steps:

[0011] S1. The pre-acquired PDF document is rendered, normalized, and skew-corrected sequentially to obtain the pre-processed document image.

[0012] S2. Based on the context information fusion mechanism and combined with the preset formula region detection model, the preprocessed document image is detected to obtain the formula image;

[0013] S3. Based on the attention mechanism and using a preset formula parsing model, the formula image is structured and parsed to obtain the two-dimensional structure information of the formula.

[0014] Furthermore, the pre-acquired PDF document is sequentially rendered, normalized, and skew-corrected to obtain a pre-processed document image, including the following steps:

[0015] S11. Obtain the PDF document and render it page by page to obtain the rendered document image;

[0016] S12. Based on Otsu's algorithm, the rendered image is binarized to obtain a binarized document image;

[0017] S13. Using the Hough transform algorithm, the binarized document image is tilted to obtain the preprocessed document image.

[0018] Furthermore, the Hough transform algorithm is used to correct the tilt of the binarized document image, resulting in a preprocessed document image, which includes the following steps:

[0019] S131. Based on the edge detection algorithm, extract the contour edges of the binarized document image to obtain the edge document image;

[0020] S132. Using the Hough transform algorithm, the pixels in the document edge image are mapped to the Hough parameter space to obtain the mapping curve.

[0021] S133. Based on a two-dimensional accumulator, vote on the mapping curve and determine the peak point according to the voting results.

[0022] S134. Determine the tilt angle of the binarized document image based on the peak points;

[0023] S135. Using a rotation matrix, perform an affine transformation on the binarized document image to obtain the preprocessed document image.

[0024] Furthermore, based on the context information fusion mechanism and combined with the preset formula region detection model, the preprocessed document image is detected to obtain the formula image, including the following steps:

[0025] S21. Using the information divergence regression loss function, optimize the bounding box regression loss of the target detection model to obtain the formula region detection model;

[0026] S22. Using the backbone network and neck network of the formula region detection model, the preprocessed document image is processed to obtain a proposal of candidate formula regions, the proposal including bounding boxes and corresponding visual feature vectors.

[0027] S23. Based on the coordinate transformation algorithm, the bounding box of the candidate formula region is mapped to obtain the corresponding original PDF document coordinate points;

[0028] S24. Based on the coordinate points of the corresponding original PDF document and combined with the context information fusion mechanism, extract the layout features of the document;

[0029] S25. Using the head network of the formula region detection model, the plate features and visual feature vectors are concatenated to obtain the fused feature vector.

[0030] S26. Using the classification and regression layers of the formula region detection model, process the fused feature vector to obtain the formula category and bounding box position.

[0031] Furthermore, by employing the information divergence regression loss function, the bounding box regression loss of the target detection model is optimized, resulting in the formula for the region detection model, which includes:

[0032] S211. Map both the predicted bounding boxes and the ground truth bounding boxes of the object detection model to a two-dimensional Gaussian distribution;

[0033] S212. Calculate the bounding box regression loss of the target detection model using the information divergence regression loss function;

[0034] S213. Based on the bounding box regression loss of the target detection model, optimize the target detection model to obtain the formula region detection model.

[0035] Furthermore, the formula for the information divergence regression loss function is as follows:

[0036] ;

[0037] In the formula, m p Indicates the prediction of Gaussian distribution N p The mean vector; Σ p Indicates the prediction of Gaussian distribution N pThe covariance matrix; m t Indicates the prediction of Gaussian distribution N t The mean vector; Σ t Indicates the prediction of Gaussian distribution N t The covariance matrix; T The transpose operation of a matrix is ​​represented by tr(·); the trace of a matrix is ​​represented by tr(·); and the determinant of a matrix is ​​represented by det(·). k The dimension of the Gaussian distribution is represented by ln(·); ln(·) represents the natural logarithm. L KLD This represents the information divergence regression loss function.

[0038] Furthermore, based on the attention mechanism and using a pre-defined formula parsing model, the formula image is structurally parsed to obtain the two-dimensional structural information of the formula, including the following steps:

[0039] S31. Preprocess the formula image to obtain an image block containing two-dimensional positional encoding;

[0040] S32. Based on the attention mechanism and combined with the encoder of the formula analysis model, the image patch containing two-dimensional position encoding is transformed to obtain the image patch features;

[0041] S33. Based on the attention mechanism and combined with the decoder of the formula parsing model, the image patch features are correlated and calculated to obtain the two-dimensional structure information of the formula.

[0042] Furthermore, preprocessing the formula image to obtain image blocks containing two-dimensional positional codes includes the following steps:

[0043] S311. Divide the formula image into several non-overlapping image blocks of fixed size;

[0044] S312. Using a linear projection layer, flatten and map each image patch to obtain an embedding vector;

[0045] S313. Add two-dimensional positional encoding to each embedding vector to obtain two-dimensional positionally encoded image blocks.

[0046] Furthermore, based on the attention mechanism and combined with the decoder of the formula parsing model, the image patch features are correlated and calculated to obtain the two-dimensional structure information of the formula, including the following steps:

[0047] S331. The decoder of the formula parsing model runs in an autoregressive manner; during the process, self-attention calculation is performed on the currently generated part of the LaTeX labeled sequence to obtain the current sequence state;

[0048] S332. Based on the cross-attention mechanism, the current sequence state is associated with the image patch features output by the encoder to predict the next LaTeX tag;

[0049] S333. Iterate through the above process until a sequence terminator is generated. Integrate all generated LaTeX tags to obtain the two-dimensional structure information of the formula.

[0050] According to another aspect of the present invention, a formula structure parsing and semantic understanding system for PDF documents is provided, the system comprising: a document preprocessing module, a formula region detection module, and a formula information extraction module;

[0051] The document preprocessing module is used to render, normalize, and correct the skewness of the pre-acquired PDF document in sequence to obtain the preprocessed document image.

[0052] The formula region detection module is used to detect formula images in preprocessed document images based on a context information fusion mechanism and a preset formula region detection model.

[0053] The formula information extraction module is used to perform structured analysis on the formula image based on the attention mechanism and using a preset formula parsing model to obtain the two-dimensional structure information of the formula.

[0054] The beneficial effects of this invention are as follows:

[0055] 1. This invention has high precision and high robustness. It uses a deep learning model for formula detection and parsing, which has better robustness and accuracy for complex layouts, handwritten text, and low-quality scanned documents.

[0056] 2. This invention realizes the structuring of formulas. This invention can directly parse formula images into editable LaTeX format, preserving the two-dimensional structural information of the formulas, so that the formulas can be understood and reused by machines.

[0057] 3. This invention improves information utilization efficiency. By structuring formulas and associating them with contextual text, this invention greatly enhances the usability of formulas in PDF documents, facilitating subsequent retrieval, editing, and knowledge management.

[0058] 4. This invention reduces labor costs. It automates the process from formula recognition to structured parsing, reducing manual input and typesetting workload and improving efficiency.

[0059] 5. This invention supports a variety of downstream applications. The parsing results of this invention can be widely used in various scenarios such as intelligent retrieval, formula editor, knowledge graph construction, and document conversion. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart of a method for structured parsing and semantic understanding of formulas for PDF documents according to an embodiment of the present invention.

[0062] Figure 2 This is a principle block diagram of a method and system for formula structured parsing and semantic understanding of PDF documents according to an embodiment of the present invention.

[0063] In the picture:

[0064] 1. Document preprocessing module; 2. Formula area detection module; 3. Formula information extraction module. Detailed Implementation

[0065] To further illustrate the various embodiments, the present invention provides accompanying drawings, which are part of the disclosure of the present invention. These drawings are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementation methods and the advantages of the present invention. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0066] According to embodiments of the present invention, a method and system for structured parsing and semantic understanding of formulas for PDF documents are provided.

[0067] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figure 1 As shown, according to an embodiment of the present invention, the configuration method includes the following steps:

[0068] S1. The pre-acquired PDF document is rendered, normalized, and skew-corrected sequentially to obtain the pre-processed document image.

[0069] It should be noted that PDF documents come from diverse sources, some being high-quality vector graphics and others low-resolution scans. Existing methods are ineffective at recognizing low-quality images and are easily affected by noise, blur, and other factors. Therefore, it is necessary to sequentially render, normalize, and correct skew in the pre-acquired PDF documents to improve their quality.

[0070] In this optional embodiment, the pre-acquired PDF document is sequentially rendered, normalized, and skew-corrected to obtain a pre-processed document image, including the following steps:

[0071] S11. Obtain the PDF document and render it page by page to obtain the rendered document image;

[0072] Specifically, to avoid information loss, this invention first renders the input PDF document page by page. The rendering library used is the mature Python library PyMuPDF. The rendering parameters are set to a target rendering resolution of 300 DPI (Dots Per Inch). This resolution is a recognized standard in the field of Optical Character Recognition (OCR), ensuring that during rasterization, tiny symbols and fine lines in formulas, such as decimal points, superscript commas, and fraction lines, are clearly preserved, providing high-quality visual input for subsequent recognition and analysis. The rendered images are then uniformly converted to 8-bit grayscale images. This reduces the interference of irrelevant color information on the recognition model and normalizes pixel values ​​to an integer range, facilitating calculations in subsequent image processing algorithms.

[0073] S12. Based on Otsu's algorithm, the rendered image is binarized to obtain a binarized document image;

[0074] Specifically, to improve the robustness of the model and address documents with poor scanning quality, noise, or tilt, this invention employs the Otsu method for image binarization, rather than a globally fixed threshold. The Otsu method is an adaptive thresholding technique, particularly suitable for document images with uneven lighting or complex backgrounds.

[0075] Its core principle is to iterate through all possible grayscale thresholds. t The goal is to find an optimal threshold that maximizes the between-class variance of the foreground (such as text and formula symbols) and background pixels segmented by that threshold. This is equivalent to minimizing the intra-class variance.

[0076] The mathematical expression for minimizing the within-class variance is:

[0077] ;

[0078] In the formula, t This indicates the grayscale threshold for the current test; oh 0( t ) indicates that the threshold t The proportion of background pixels to the total number of pixels; Represents the variance of the grayscale values ​​of the background pixels; oh 1( t ) indicates that the threshold t The proportion of foreground pixels (such as text and formulas) to the total number of pixels; Represents the variance of the grayscale values ​​of the foreground pixels; The within-class variance of the image is represented by t. The goal of the algorithm is to find the threshold t that minimizes this value. This invention has uniformly converted the images to 8-bit grayscale. t The value range is the integer interval [0, 255].

[0079] S13. Using the Hough transform algorithm, the binarized document image is tilted to obtain the preprocessed document image.

[0080] Specifically, for page tilt caused during scanning or shooting, this invention employs a tilt correction algorithm based on Hough Transform.

[0081] In this optional embodiment, the Hough transform algorithm is used to perform tilt correction on the binarized document image to obtain the preprocessed document image, including the following steps:

[0082] S131. Based on the edge detection algorithm, extract the contour edges of the binarized document image to obtain the edge document image;

[0083] Specifically, the Canny edge detection operator is applied to the binarized image to highlight the outlines of text lines and formulas.

[0084] S132. Using the Hough transform algorithm, the pixels in the document edge image are mapped to the Hough parameter space to obtain the mapping curve.

[0085] Specifically, each pixel in the edge image ( x , y Mapped to Hough parameter space ( r , i Each point in the parameter space represents a straight line in the original image space. The expression for the Hough transform equation is:

[0086] ;

[0087] In the formula, x Represents the x-coordinate of an edge pixel in the image space; y Represents the ordinate of a pixel at the same edge in the image space; r This represents a parameter in the Hough parameter space, whose physical meaning is the perpendicular distance from the origin of the image coordinate system to a certain straight line; iThis represents another parameter in the Hough parameter space, whose physical meaning is the perpendicular line (i.e., the normal line) from the origin to the line. x The angle between the positive axis and the positive axis.

[0088] Through this relationship, a point in image space ( x , y ) was transformed into parameter space ( r , i A sine curve in the image space. Conversely, all points on a straight line in the image space will have their corresponding curves intersecting at the same point in the parameter space. r , i This allows for the detection of straight lines.

[0089] S133. Based on a two-dimensional accumulator, vote on the mapping curve and determine the peak point according to the voting results.

[0090] Specifically, create a two-dimensional accumulator array and map it to all edge points. r , i The curve is used for voting. The point that receives the most votes in the parameter space corresponds to the straight line with the strongest collinearity of pixels in the image, which is usually the baseline of the text.

[0091] Accumulator voting is the core of the Hough transform algorithm, and its purpose is to find the intersection point between the parameter space and the line in the image space. This process can be broken down into the following steps:

[0092] Parameter space discretization and accumulator creation. First, the Hough parameter space needs to be discretized ( r , i Discretization is performed. (Angle) i The range is typically set to [-90°, 90°] and divided in fixed increments (e.g., 0.5 degrees or 1 degree); distance r The range is determined by the image size, with its maximum value being the length of the image diagonal. Based on the discretized... r and i Range, create a two-dimensional array, i.e., an accumulator. The rows of this array correspond to... r The discrete values, corresponding to the columns i The discrete values ​​are given, and the initial value of all cells is set to 0.

[0093] Traverse the edge points and vote. For each edge pixel in the image output from the previous step (Cann edge detection), x , y ) perform a traversal. For each edge point ( x , y Then iterate through all the discretized angles. i Value. Each ( x ,y , i Substituting the combination of ) into the Hough transform equation, calculate the corresponding r value.

[0094] Calculate ( r , i The count value of the corresponding accumulator array cell for a given point is incremented by 1. This incrementing operation is equivalent to a vote. It represents the number of votes cast for a point in the image space. x , y ) supports the use of ( ) in parameter space r , i The straight line defined by ).

[0095] Find the peak. After all edge points have voted for all angles, the cell with the highest value in the accumulator array is the peak point, that is, the point with the most votes.

[0096] S134. Determine the tilt angle of the binarized document image based on the peak points;

[0097] Specifically, the peak point corresponds to ( r peak， i peak This represents the straight line in the image that contains the most collinear pixels. In the application scenario of this invention, this is usually the baseline of the text. i peak This refers to the overall tilt angle of the document.

[0098] S135. Using a rotation matrix, perform an affine transformation on the binarized document image to obtain the preprocessed document image.

[0099] To clarify, a simplified example is provided to make the process easier to understand. Assume that after edge detection, three collinear edge points are identified on the image: A(20,10), B(40,20), and C(60,30). These three points lie on the same straight line.

[0100] The voting process is as follows:

[0101] For processing point A(20,10), traverse all discrete angles. i (e.g., from -90° to 90°), for each i Calculate one r Values. These ( r , i This will create a sine curve in the accumulator array. Increment the count value of all corresponding cells on this curve by 1.

[0102] Similarly, for point B(40,20), calculate the sine curve of point B in the parameter space and increment the count value of the corresponding cell by 1.

[0103] To process point C(60,30), perform the same operation on point C.

[0104] The results are analyzed as follows:

[0105] Since points A, B, and C are collinear in image space, the three sine curves corresponding to them in parameter space will intersect at the same point. Assume the true parameters of this line are ( r 0, i 0). Then in the accumulator array, the corresponding ( r 0, i Cells with a count of 0 will be voted on three times, resulting in a final count of 3. Other cells not traversed by all three curves will have a count of at most 1 (if only one curve passes through them) or 0 (if no curve passes through them). By scanning the entire accumulator array, the peak point with a count of 3 can be easily found. The angle corresponding to this point... i 0 represents the document tilt angle we are looking for, thus providing a basis for subsequent affine transformation correction.

[0106] The angle is determined as follows:

[0107] Find the angle corresponding to the peak point in the accumulator. i peak This angle is the overall tilt angle of the document image.

[0108] The affine transformation is as follows:

[0109] Calculate a rotation matrix to rotate the original grayscale image by an angle of... i peak The affine transformation is used to correct the tilted page to a horizontal position.

[0110] S2. Based on the context information fusion mechanism and combined with the preset formula region detection model, the preprocessed document image is detected to obtain the formula image.

[0111] It should be explained that the formula region detection module of this invention is the key to achieving high-precision analysis, and its innovation lies in the loss function and information fusion mechanism specifically designed to solve the problem of formula detection.

[0112] In this optional embodiment, based on a context information fusion mechanism and combined with a preset formula region detection model, the preprocessed document image is detected to obtain the formula image, including the following steps:

[0113] S21. Using the information divergence regression loss function, optimize the bounding box regression loss of the target detection model to obtain the formula region detection model;

[0114] Specifically, the L1 / L2 loss used in traditional object detectors is applied to the five parameters of the regressed rotating box ( x , y , w , h , i When performing regression loss functions (LRFS) with large aspect ratios, issues arise regarding boundary discontinuities and poor parameter coupling. For example, a change in angle from 89° to -89° results in a significant numerical change, but the actual change in the frame's pose is minimal. Furthermore, for formulas with large aspect ratios, even small errors in angle can cause a sharp drop in IoU, which L1 loss functions are insensitive to. To address this issue, this invention introduces a regression loss function based on Kullback-Leibler divergence (KLD), which is a core innovation of this invention.

[0115] S22. Using the backbone network and neck network of the formula region detection model, the preprocessed document image is processed to obtain a proposal of candidate formula regions, the proposal including bounding boxes and corresponding visual feature vectors.

[0116] S23. Based on the coordinate transformation algorithm, the bounding box of the candidate formula region is mapped to obtain the corresponding original PDF document coordinate points;

[0117] S24. Based on the coordinate points of the corresponding original PDF document and combined with the context information fusion mechanism, extract the layout features of the document;

[0118] S25. Using the head network of the formula region detection model, the plate features and visual feature vectors are concatenated to obtain the fused feature vector.

[0119] S26. Using the classification and regression layers of the formula region detection model, process the fused feature vector to obtain the formula category and bounding box position.

[0120] It should be explained that this invention proposes a specific late fusion mechanism to enhance the model's ability to distinguish between inline formulas and interline formulas.

[0121] A deep learning-based object detection model (such as YOLOv8) uses a backbone network and a neck network (FPN) to process the input image and generate a series of candidate region proposals, each containing a bounding box (pixel coordinates). B pix and the corresponding visual feature vectors V feat .

[0122] B pix This represents the bounding box. It is used to precisely mark the location and extent of a potential formula region detected by the model on a rendered PDF page image. Its value is defined based on the image's pixel coordinate system and contains the information needed to determine a rectangular region (e.g., center point coordinates, width, height, and possible rotation angles).

[0123] V feat This represents the visual feature vector. It is a high-dimensional numerical vector extracted by the backbone and neck network of a deep learning model after processing the image content within the bounding box. This vector is a summary and digital representation of the content of the formula image within the box, describing the visual features of the formula such as shape, structure, and texture, rather than its positional information. This feature vector is a key input for subsequent structured parsing and semantic understanding of the formula.

[0124] For each candidate bounding box B pix Using a coordinate transformation algorithm, it is accurately mapped back to the point coordinates of the original PDF page. B pdf .

[0125] Using libraries such as pdfminer.six, analyze B pdf Text elements within the inner and immediately adjacent outer regions. Extract. S in , S out , R line , C center and F layout Layout features.

[0126] S in This indicates the average font size of the characters within the box; S out The average font size of the characters immediately preceding or following the box (on the same line or in the preceding and following lines); R line This represents the ratio of the line height of the line containing the box to the average line height of the document. C center This indicates whether the area is horizontally centered within the page or column. F layout This means quantizing and combining the above layout features into a low-dimensional layout feature vector.

[0127] In the head network of the detection model, visual feature vectors are... V feat With layout feature vector F layout Concatenation is performed to form an enhanced fused feature vector. F fused .

[0128] The formula for expressing the fused feature vector is:

[0129] ;

[0130] In the formula, F fused This represents the enhanced fusion feature vector; V feat Represents visual feature vectors; F layout This represents the layout feature vector; Concat represents the concatenation operation.

[0131] Will F fused The input is fed into the final classification and regression layers to output more accurate formula categories (inline / spaced) and bounding box locations. In this way, the model not only sees the shape of the formula but also understands its layout, greatly improving detection accuracy under complex layouts.

[0132] In this optional embodiment, the information divergence regression loss function is used to optimize the bounding box regression loss of the target detection model, resulting in a formula-based region detection model including:

[0133] S211. Map both the predicted bounding boxes and the ground truth bounding boxes of the object detection model to a two-dimensional Gaussian distribution;

[0134] S212. Calculate the bounding box regression loss of the target detection model using the information divergence regression loss function;

[0135] S213. Based on the bounding box regression loss of the target detection model, optimize the target detection model to obtain the formula region detection model.

[0136] It should be noted that the KLD loss-based rotated box regression no longer treats the bounding box as an independent set of parameters, but rather models it as a whole as a two-dimensional Gaussian distribution. .

[0137] The transformation from a bounding box to a Gaussian distribution is to convert a bounding box to a Gaussian distribution based on a center point ( x , y ),width w ,high h and rotation angle iThe defined rotated bounding box is uniquely mapped to a two-dimensional Gaussian distribution, whose mean vector is... m The covariance matrix Σ is defined as follows:

[0138] ;

[0139] ;

[0140] in, R ( i ) is a standard two-dimensional rotation matrix:

[0141] ;

[0142] In the formula, x The x-coordinate of the center point of the bounding box; y This represents the ordinate of the center point of the bounding box; w Indicates the width of the bounding box; h Indicates the height of the bounding box; α The rotation angle of the bounding box is usually defined as the angle traversed by rotating counterclockwise from the positive direction of the horizontal axis to the width direction of the bounding box; m Σ represents the mean vector of the two-dimensional Gaussian distribution, i.e., the center point of the bounding box; Σ represents the covariance matrix of the two-dimensional Gaussian distribution, which is determined by the width, height and rotation angle, and describes the shape and direction of the distribution; R This represents a standard two-dimensional rotation matrix, defined by the rotation angle. α Decide.

[0143] This invention uses the Gaussian distribution corresponding to the prediction box. N p ( m p ,Σ p Gaussian distributions corresponding to the ground truth bounding boxes N t ( m t ,Σ t The regression loss is calculated using the KLD (Knowledge, Dispersion, and Least Accuracy) between the two values. The formula for the information divergence regression loss function is:

[0144] ;

[0145] In the formula, m p Indicates the prediction of Gaussian distribution N p The mean vector; Σ p Indicates the prediction of Gaussian distribution N p The covariance matrix; m t Indicates the prediction of Gaussian distributionN t The mean vector; Σ t Indicates the prediction of Gaussian distribution N t The covariance matrix; T The transpose operation of a matrix is ​​represented; tr(·) represents the trace of a matrix, which is the sum of the elements on the main diagonal of the matrix; det(·) represents the determinant of a matrix. k The dimension representing the Gaussian distribution, in the two-dimensional case of this invention, k =2; ln(·) represents the natural logarithm; L KLD This represents the information divergence regression loss function.

[0146] The KLD loss function couples all five parameters into a unified probabilistic framework. By taking the derivative of this loss function, it can be seen that it adjusts according to the characteristics of the target (such as aspect ratio). h / w Automatically adjusts the gradient weights of each parameter. For slender formulas, the loss modulo angle... i The penalty increases significantly, forcing the model to learn the direction more accurately. Due to the continuity of the Gaussian distribution, this method naturally solves the problem of abrupt loss caused by angular periodicity. The KLD loss is scale invariant, providing a consistent optimization objective for formulas of different sizes.

[0147] S3. Based on the attention mechanism and using a preset formula parsing model, the formula image is structured and parsed to obtain the two-dimensional structure information of the formula.

[0148] It should be explained that this module is the core innovation of this invention, which redefines the formula recognition problem as an image-to-sequence translation task.

[0149] In this optional embodiment, based on an attention mechanism and utilizing a preset formula parsing model, the formula image is structurally parsed to obtain the two-dimensional structural information of the formula, including the following steps:

[0150] S31. Preprocess the formula image to obtain an image block containing two-dimensional positional encoding;

[0151] S32. Based on the attention mechanism and combined with the encoder of the formula analysis model, the image patch containing two-dimensional position encoding is transformed to obtain the image patch features;

[0152] Specifically, the encoder's role is to transform the input formula image I into a series of feature vectors rich in contextual information. This invention uses VisionTransformer (ViT) as the encoder backbone.

[0153] The vector sequence embedded with location information is fed into a multi-layer Transformer encoder module. The core of each module is a multi-head self-attention mechanism. Its calculation formula is as follows:

[0154] ;

[0155] In the formula, Q This represents the query vector for the current block; K Represents the key vector of all blocks, used for... Q match, V This represents the value vector of all blocks, i.e., the content itself; QK T Indicates passage Q and K The dot product is used to calculate the original similarity score between the query block and all other blocks; Denotes the scaling factor, where d k is the dimension of the key vector (Key). Using it for scaling can prevent the dot product result from becoming too large when it is large, causing the softmax function to enter the region of minimum gradient, thus stabilizing the training process. softmax means normalizing the scaled score and converting it into attention weights, with the sum of all weights being 1. Finally, the obtained attention weights are multiplied by the value matrix and weighted summed to obtain a new vector representation that incorporates global context information.

[0156] Q (Query), K (Key), V The (Value) matrix is ​​obtained by performing different linear transformations on the input image patch embedding sequence. For a given image patch (such as the patch representing the integral symbol ∫), its... Q The vector will query all other image patches (including the patches representing the upper and lower limits a, b). K vector.

[0157] QK T The dot product of the values ​​is used to calculate the similarity score, representing the importance of each image patch to the current patch. Softmax normalizes the score into attention weights, with higher weights indicating stronger correlation. Finally, the weights are multiplied by the sum of the values ​​of the two patches. V The weighted summation of vectors (representing the content of each block) yields a new representation of the image block that incorporates global contextual information. This mechanism enables the model to establish dependencies between any two blocks in an image within a single layer, thereby efficiently understanding the global two-dimensional layout of formulas, such as complex structures like fractions, matrices, and square roots.

[0158] S33. Based on the attention mechanism and combined with the decoder of the formula parsing model, the image patch features are correlated and calculated to obtain the two-dimensional structure information of the formula.

[0159] In this optional embodiment, preprocessing the formula image to obtain an image patch containing two-dimensional positional encoding includes the following steps:

[0160] S311. Divide the formula image into several non-overlapping image blocks of fixed size;

[0161] Specifically, the cropped formula image (scaled to a fixed size, 224×224) is divided into N non-overlapping fixed-size image patches, each 16×16 pixels. Each image patch is flattened into a one-dimensional vector.

[0162] S312. Using a linear projection layer, flatten and map each image patch to obtain an embedding vector;

[0163] Specifically, each flattened image patch vector is mapped into a D-dimensional embedding vector through a learnable linear projection layer.

[0164] S313. Add two-dimensional positional encoding to each embedding vector to obtain two-dimensional positionally encoded image blocks.

[0165] Specifically, since the Transformer itself lacks the ability to process sequential information, in order to preserve the spatial relationship between image patches and facilitate understanding the two-dimensional structure of the formula, this invention adds a learnable two-dimensional positional encoding (2DPpositionalEncoding) to each embedding vector. Without positional encoding, the model will be unable to distinguish between a / b and b / a.

[0166] In this optional embodiment, based on the attention mechanism and combined with the decoder of the formula parsing model, the image patch features are correlated and calculated to obtain the two-dimensional structure information of the formula, including the following steps:

[0167] S331. The decoder of the formula parsing model runs in an autoregressive manner; during the process, self-attention calculation is performed on the currently generated part of the LaTeX labeled sequence to obtain the current sequence state;

[0168] S332. Based on the cross-attention mechanism, the current sequence state is associated with the image patch features output by the encoder to predict the next LaTeX tag;

[0169] S333. Iterate through the above process until a sequence terminator is generated. Integrate all generated LaTeX tags to obtain the two-dimensional structure information of the formula.

[0170] It should be explained that the decoder is a standard Transformer decoder, whose task is to generate the final LaTeX string one token at a time in an auto-regressive manner based on the visual features output by the encoder.

[0171] The autoregressive process occurs when the decoder generates a LaTeX token at step t, and then it considers all the tokens generated in the previous t-1 steps (such as...). <start>E=) is used as input. This process refers to generation step by step. When the decoder predicts the next LaTeX character (token), it must rely on all the characters that have been generated before it.

[0172] Cross-attention acts as a bridge connecting the encoder and decoder. In each attention layer of the decoder, in addition to self-attention computation on the generated LaTeX sequence, there is a cross-attention module. In this module, Q The matrix comes from the decoder (representing what information is needed next based on the generated sequence), while K and V The matrix is ​​derived from the encoder's final output (representing all visual features of the entire formula image). This allows the decoder to review and focus on the most relevant regions of the formula image when generating each LaTeX token. For example, when generating {, cross-attention focuses heavily on the molecular portion of the image.

[0173] When the decoder needs to generate LaTeX code for the famous formula E=mc^2, the steps are as follows.

[0174] Step 1: Enter a special start character <start>The model predicts the first character as E.

[0175] Step 2: <start>Using E as the new input, the model predicts the second character =.

[0176] Step 3: <start>E= is the new input, and the model needs to predict the third character m based on this sequence.

[0177] ( <start>E=) is the input content of the decoder when performing the third prediction step. It is not a complete formula in itself, but an incomplete sequence of intermediate steps used to illustrate the "connecting the preceding and following" working mechanism of autoregression.

[0178] According to another embodiment of the invention, such as Figure 2 As shown, a formula structure parsing and semantic understanding system for PDF documents is also provided. This system includes: document preprocessing module 1, formula region detection module 2, and formula information extraction module 3.

[0179] Document preprocessing module 1 is used to render, normalize and skew correct the pre-acquired PDF document in sequence to obtain the pre-processed document image;

[0180] Formula region detection module 2 is used to detect preprocessed document images based on context information fusion mechanism and in combination with preset formula region detection model to obtain formula images;

[0181] Formula information extraction module 3 is used to perform structured analysis on the formula image based on the attention mechanism and using a preset formula parsing model to obtain the two-dimensional structure information of the formula.

[0182] In summary, by utilizing the above-mentioned technical solutions of this invention, the invention achieves high precision and robustness. Employing a deep learning model for formula detection and parsing, it exhibits better robustness and accuracy even with complex layouts, handwritten text, and low-quality scanned documents. This invention achieves formula structuring, directly parsing formula images into editable LaTeX format while preserving the formula's two-dimensional structural information, enabling machine understanding and reuse. This invention improves information utilization efficiency by structuring formulas and associating them with contextual text, significantly enhancing the usability of formulas in PDF documents and facilitating subsequent retrieval, editing, and knowledge management. This invention reduces labor costs by automating the formula recognition and structured parsing process, reducing manual input and typesetting workload, and improving efficiency. This invention supports various downstream applications; its parsing results can be widely applied to intelligent retrieval, formula editors, knowledge graph construction, and document conversion, among other scenarios.

[0183] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.< / start> < / start> < / start> < / start> < / start>

Claims

1. A method for structured parsing and semantic understanding of formulas in PDF documents, characterized in that, The method includes the following steps: S1. The pre-acquired PDF document is rendered, normalized, and skew-corrected sequentially to obtain the pre-processed document image. S2. Based on the context information fusion mechanism and combined with the preset formula region detection model, the preprocessed document image is detected to obtain the formula image; specifically including: S21. Using the information divergence regression loss function, optimize the bounding box regression loss of the target detection model to obtain the formula region detection model; S22. Using the backbone network and neck network of the formula region detection model, the preprocessed document image is processed to obtain a proposal of candidate formula regions, the proposal including bounding boxes and corresponding visual feature vectors. S23. Based on the coordinate transformation algorithm, the bounding box of the candidate formula region is mapped to obtain the corresponding original PDF document coordinate points; S24. Based on the coordinate points of the corresponding original PDF document and combined with a contextual information fusion mechanism, extract the document's layout features; the layout features include... S in , S out , R line , C center and F layout ; S in This indicates the average font size of the characters within the box; S out This indicates the average font size of the characters immediately preceding and following the box; R line This represents the ratio of the line height of the line containing the box to the average line height of the document. C center This indicates whether the area is horizontally centered within the page or column. F layout This means quantizing and combining the above layout features into a low-dimensional layout feature vector; S25. Using the head network of the formula region detection model, the layout features and visual feature vectors are concatenated to obtain a fused feature vector. S26. Using the classification and regression layers of the formula region detection model, process the fused feature vector to obtain the formula category and bounding box position; S3. Based on the attention mechanism and using a preset formula parsing model, the formula image is structured and parsed to obtain the two-dimensional structure information of the formula.

2. The method for structured parsing and semantic understanding of formulas for PDF documents according to claim 1, characterized in that, The process of rendering, normalizing, and skew correction sequentially on the pre-acquired PDF document to obtain the pre-processed document image includes the following steps: S11. Obtain the PDF document and render it page by page to obtain the rendered document image; S12. Based on Otsu's algorithm, the rendered image is binarized to obtain a binarized document image; S13. Using the Hough transform algorithm, the binarized document image is tilted to obtain the preprocessed document image.

3. The method for structured parsing and semantic understanding of formulas in PDF documents according to claim 2, characterized in that, The process of using the Hough transform algorithm to perform tilt correction on the binarized document image to obtain the preprocessed document image includes the following steps: S131. Based on the edge detection algorithm, extract the contour edges of the binarized document image to obtain the edge document image; S132. Using the Hough transform algorithm, the pixels in the document edge image are mapped to the Hough parameter space to obtain the mapping curve. S133. Based on a two-dimensional accumulator, vote on the mapping curve and determine the peak point according to the voting results. S134. Determine the tilt angle of the binarized document image based on the peak points; S135. Using a rotation matrix, perform an affine transformation on the binarized document image to obtain the preprocessed document image.

4. The method for structured parsing and semantic understanding of formulas in PDF documents according to claim 1, characterized in that, The method employs the information divergence regression loss function to optimize the bounding box regression loss of the target detection model, resulting in the formula for the region detection model, which includes: S211. Map both the predicted bounding boxes and the ground truth bounding boxes of the object detection model to a two-dimensional Gaussian distribution; S212. Calculate the bounding box regression loss of the target detection model using the information divergence regression loss function; S213. Based on the bounding box regression loss of the target detection model, optimize the target detection model to obtain the formula region detection model.

5. The method for structured parsing and semantic understanding of formulas in PDF documents according to claim 4, characterized in that, The formula for the information divergence regression loss function is as follows: ； In the formula, μ p Indicates the prediction of Gaussian distribution N p The mean vector; Σ p Indicates the prediction of Gaussian distribution N p The covariance matrix; μ t Indicates the prediction of Gaussian distribution N t The mean vector; Σ t Indicates the prediction of Gaussian distribution N t The covariance matrix; T The transpose operation of a matrix is ​​represented by tr(·); the trace of a matrix is ​​represented by tr(·); and the determinant of a matrix is ​​represented by det(·). k The dimension of the Gaussian distribution is represented by ln(·); ln(·) represents the natural logarithm. L KLD This represents the information divergence regression loss function.

6. The method for structured parsing and semantic understanding of formulas in PDF documents according to claim 1, characterized in that, The process of performing structured analysis on the formula image based on an attention mechanism and utilizing a pre-defined formula analysis model to obtain the two-dimensional structure information of the formula includes the following steps: S31. Preprocess the formula image to obtain an image block containing two-dimensional positional encoding; S32. Based on the attention mechanism and combined with the encoder of the formula analysis model, the image patch containing two-dimensional position encoding is transformed to obtain the image patch features; S33. Based on the attention mechanism and combined with the decoder of the formula parsing model, the image patch features are correlated and calculated to obtain the two-dimensional structure information of the formula.

7. The method for structured parsing and semantic understanding of formulas in PDF documents according to claim 6, characterized in that, The preprocessing of the formula image to obtain an image block containing two-dimensional positional encoding includes the following steps: S311. Divide the formula image into several non-overlapping image blocks of fixed size; S312. Using a linear projection layer, flatten and map each image patch to obtain an embedding vector; S313. Add two-dimensional positional encoding to each embedding vector to obtain two-dimensional positionally encoded image blocks.

8. The method for structured parsing and semantic understanding of formulas in PDF documents according to claim 7, characterized in that, The process of using an attention mechanism and a decoder combined with a formula parsing model to perform correlation calculations on image patch features to obtain the two-dimensional structure information of the formula includes the following steps: S331. The decoder of the formula parsing model runs in an autoregressive manner; during the process, self-attention calculation is performed on the currently generated part of the LaTeX labeled sequence to obtain the current sequence state; S332. Based on the cross-attention mechanism, the current sequence state is associated with the image patch features output by the encoder to predict the next LaTeX tag; S333. Iterate through the above process until a sequence terminator is generated. Integrate all generated LaTeX tags to obtain the two-dimensional structure information of the formula.

9. A system for structured parsing and semantic understanding of formulas for PDF documents, used to implement the method for structured parsing and semantic understanding of formulas for PDF documents as described in any one of claims 1-8, characterized in that, The system includes: a document preprocessing module, a formula area detection module, and a formula information extraction module; The document preprocessing module is used to render, normalize, and correct the skewness of the pre-acquired PDF document in sequence to obtain the preprocessed document image. The formula region detection module is used to detect formula images in preprocessed document images based on a context information fusion mechanism and a preset formula region detection model. The formula information extraction module is used to perform structured analysis on the formula image based on the attention mechanism and using a preset formula parsing model to obtain the two-dimensional structure information of the formula.

Images