Intelligent extraction method and system for input text containing mathematical formula

Abstract

The present invention belongs to the technical field of text processing, and relates to an intelligent extraction method and system for an input text containing a mathematical formula. The method comprises: 1) performing format determination, conversion and preprocessing on an input text; 2) performing angle correction on the preprocessed image-formatted text; 3) performing formula detection; 4) performing layout analysis; 5) for inline formulas, according to a formula detection box, determining whether a corrected OCR detection box contains an inline formula, and splitting the OCR detection box containing the inline formula, so as to obtain a text-only OCR detection box; 6) performing formula recognition to obtain a formula recognition result; 7) performing text recognition to obtain a text recognition result; and 8) on the basis of a layout analysis box and a layout category thereof, performing same-line detection box determination and merging on the formula recognition result and the text recognition result, so as to obtain an extraction result for the input text. The present invention can effectively improve the extraction efficiency and accuracy for input texts containing mathematical formulas.

Description

A method and system for intelligent extraction of input text containing mathematical formulas Technical Field

[0001] The present invention belongs to the technical field of text extraction, and relates to a method and system for extracting input text, and in particular to a method and system for intelligently extracting input text containing mathematical formulas. Background Art

[0002] With the rapid development of information technology, a large amount of text data is stored electronically. The effective extraction and utilization of this data has become increasingly important, especially in the fields of knowledge management and information retrieval.

[0003] Among them, optical character recognition (OCR) technology has received significant attention due to its wide application in document digitization and automatic text analysis. OCR technology mainly uses character template matching and machine learning algorithms to convert characters in images into editable text. In recent years, with the rise of deep learning, OCR systems based on convolutional neural networks (CNN) have begun to significantly improve their performance. For example, open source OCR systems such as Tesseract OCR have successfully improved the accuracy of text recognition by combining traditional algorithms and deep learning technology. In comparison, PaddleOCR, which uses deep learning, performs better in Chinese character recognition. However, when faced with complex document structures such as charts, formulas, and page layouts, existing OCR technology still does not meet ideal standards in terms of accuracy, robustness, and processing speed.

[0004] Meanwhile, formula detection and recognition in text extraction has traditionally relied on pattern-matching methods, which have limitations when dealing with complex and irregular formulas. With the development of machine learning, deep learning-based methods (such as LaTeX-OCR) have made significant progress, improving the ability to recognize complex formulas through automatic feature learning using deep neural networks. However, current models still face challenges when dealing with complex formulas embedded in the text.

[0005] Therefore, in view of the defects existing in the above-mentioned prior art, it is necessary to develop a new intelligent extraction method and system for input text containing mathematical formulas.

[0006] Summary of the Invention

[0007] In order to overcome the shortcomings of the existing technology, the present invention proposes an intelligent extraction method and system for input text containing mathematical formulas, which can effectively improve the efficiency and accuracy of extracting text and formula content in textbooks, books, PDF papers, and pictures, and provide strong support for text information processing in related fields.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A method for intelligently extracting input text containing mathematical formulas, characterized by comprising the following steps:

[0010] 1) Determine the format of the input text. If the format of the input text is not a picture format, convert the input text into a picture format text and pre-process the picture format text;

[0011] 2) performing text detection on the preprocessed image format text to obtain an OCR detection frame, determining whether to perform angle correction on the preprocessed image format text based on the aspect ratio and text direction classifier of the OCR detection frame, and performing angle correction if necessary to obtain the corrected image format text and OCR detection frame;

[0012] 3) performing formula detection on the corrected text in the image format to obtain a formula detection frame and its formula category, wherein the formula category includes an embedded formula and an interline formula;

[0013] 4) performing layout analysis on the corrected text in the image format to obtain a layout analysis frame and its layout categories, wherein the layout categories include title, text, image, figure annotation, table, table annotation, formula, header, footer, and references;

[0014] 5) For the embedded formula, judging whether the corrected OCR detection frame contains the embedded formula based on the formula detection frame and splitting the OCR detection frame containing the embedded formula to obtain the OCR detection frame of the plain text;

[0015] 6) cropping the text in the corrected image format according to the formula detection frame to obtain a formula area, performing formula recognition on the formula area using a formula recognition model, and simplifying the output LaTeX format to obtain a formula recognition result;

[0016] 7) Use the corresponding OCR recognition model for different scenarios to perform text recognition on the OCR detection box of the plain text to obtain the text recognition result;

[0017] 8) Combine the layout analysis frame and its layout category to determine and merge the formula recognition results and text recognition results in the same detection frame to obtain the extraction result of the input text.

[0018] Preferably, the intelligent extraction method of input text containing mathematical formulas further comprises:

[0019] 9) For the interline formula, determine whether the formula detection frame and the corrected OCR detection frame overlap significantly. If so, proceed to step 10);

[0020] 10) cropping the text in the corrected image format according to the formula detection frame to obtain a formula area, performing formula recognition on the formula area using a formula recognition model, and simplifying the output LaTeX format to obtain a formula recognition result;

[0021] 11) Use the corresponding OCR recognition model for different scenarios to perform text recognition on the corrected OCR detection frame to obtain the text recognition result;

[0022] 12) The text recognition result is replaced with the formula recognition result to obtain an extraction result of the input text.

[0023] Preferably, the step 1) specifically includes:

[0024] 1.1) Determine whether the input text is in PDF format or image format. If the input text is in PDF format, use a PDF parser to convert the input text into image format.

[0025] 1.2) Dynamically extend the size of the text in the image format to match the background size, and use a median filter algorithm to denoise the text in the image format.

[0026] Preferably, the step 2) specifically includes:

[0027] 2.1) Input the pre-processed text in the image format into the OCR detection model to perform text detection to obtain the OCR detection box;

[0028] 2.2) Calculate the aspect ratio of the OCR detection frame. If the ratio of OCR detection frames with a height greater than their width exceeds a set threshold, rotate the pre-processed image-format text 90 degrees clockwise to obtain the updated image-format text. Otherwise, directly output the pre-processed image-format text and OCR detection frame.

[0029] 2.3) Inputting the updated text in the image format into the OCR detection model for text detection to obtain an updated OCR detection frame;

[0030] 2.4) Cropping the updated OCR detection frame from the updated image-formatted text to obtain an OCR detection frame area and feeding the OCR detection frame area into a text direction classifier for direction determination. The output of the text direction classifier is either 0 degrees or 180 degrees. If the proportion of the OCR detection frame area outputted as 180 degrees in the total OCR detection frame area is greater than a set threshold, the updated image-formatted text is rotated 180 degrees clockwise to obtain the updated image-formatted text. Otherwise, the updated image-formatted text and the updated OCR detection frame are directly output.

[0031] 2.5) Input the updated text in the image format into the OCR detection model for text detection to obtain an updated OCR detection frame, and output the updated text in the image format and the updated OCR detection frame.

[0032] Preferably, the step 5) specifically includes:

[0033] 5.1) For each OCR detection box, obtain the corresponding linear equations of the upper, lower, left, and right boundaries based on the coordinates of its upper left corner, lower left corner, upper right corner, and lower right corner;

[0034] 5.2) Filtering formula detection frames whose formula category is embedded formula from the formula detection frames, calculating the X-axis coordinate of the center point of the formula detection frame whose formula category is embedded formula and substituting it into the linear equation of the upper and lower boundary lines of the OCR detection frame to determine whether the Y-axis coordinate of the center point of the formula detection frame whose formula category is embedded formula is within the Y value interval of the upper and lower boundary lines of the OCR detection frame, calculating the Y-axis coordinate of the center point of the formula detection frame whose formula category is embedded formula and substituting it into the linear equation of the left and right boundary lines of the OCR detection frame to determine whether the X-axis coordinate of the center point of the formula detection frame whose formula category is embedded formula is within the X value interval of the left and right boundary lines of the OCR detection frame;

[0035] 5.3) If the Y-axis coordinate of the center point of the embedded formula detection box is within the Y-value range of the upper and lower boundaries of the OCR detection box, and the X-axis coordinate of the center point of the embedded formula detection box is within the X-value range of the left and right boundaries of the OCR detection box, calculate the ratio of the overlapping area of ​​the OCR detection box and the embedded formula detection box to the total area of ​​the two, that is, the Intersection over Union (IoU).

[0036] 5.4) If the IoU is greater than 0, add the formula detection box with the formula category as embedded formula to the formula list and split the OCR detection box.

[0037] Preferably, the step 6) is specifically as follows:

[0038] 6.1) Crop the text in the rectified image format according to the coordinates of the formula detection box to obtain the formula area and feed the formula area into the formula recognition model, outputting the result as LaTeX;

[0039] 6.2) The output LaTeX is sent to the rendering engine for rendering judgment. If the rendering fails, the format is repaired. If the rendering still fails after the format repair, an additional error message is output to prevent the output of incorrect formulas;

[0040] 6.3) Simplify LaTeX and remove unnecessary formats. The final output includes the formula detection box, formula category and LaTeX of the formula.

[0041] Preferably, the step 7) is specifically as follows:

[0042] 7.1) Crop the OCR detection frame of the split plain text from the rectified image format text to obtain the OCR detection frame area;

[0043] 7.2) Send the OCR detection frame area to the Chinese OCR recognition model for preliminary text recognition to obtain preliminary text recognition results;

[0044] 7.3) If the preliminary text recognition result contains only English or numbers, the OCR detection box area is sent to the English OCR recognition model for recognition to obtain the final text recognition result and overwrite the preliminary text recognition result. The final output result includes the OCR detection box and the text recognition result.

[0045] Preferably, the step 8) specifically includes:

[0046] 8.1) Add the formula recognition results to the text recognition results;

[0047] 8.2) Preliminarily determine whether the formula detection box and the OCR detection box are in the same row, so as to preliminarily filter out the detection boxes belonging to the same row;

[0048] 8.3) Classifying the preliminarily screened detection frames according to the layout analysis frame and its layout category, determining whether the preliminarily screened detection frames belong to the same layout category, and selecting detection frames belonging to the same layout;

[0049] 8.4) Project the selected detection frames belonging to the same layout on the X-axis to determine the split points. Based on the number of split points, determine whether the detection frame can be split on the X-axis. If it cannot be split, determine that the selected detection frames belonging to the same layout belong to the same line of detection frames. If it can be split, determine that the selected detection frames belonging to the same layout do not belong to the same line of detection frames.

[0050] 8.5) The determined detection boxes in the same row are sorted within the row and then merged. The output extraction result of the input text includes the merged row text detection box and the merged text content.

[0051] Preferably, in step 8.2), when the Y-axis intersection of the formula detection frame and the OCR detection frame is greater than a threshold and the X-axis intersection is less than a threshold, it is preliminarily determined that the formula detection frame and the OCR detection frame are in the same row.

[0052] In addition, the present invention also provides an intelligent extraction system for input text containing mathematical formulas, characterized by comprising:

[0053] An input format determination and image preprocessing module, which is used to determine the format of the input text. If the input text is not in an image format, the input text is converted into an image format and preprocessed.

[0054] An automatic image correction module is used to perform text detection on the pre-processed image format text to obtain an OCR detection frame, determine whether to perform angle correction on the pre-processed image format text based on the aspect ratio and text direction classifier of the OCR detection frame, and perform angle correction if necessary to obtain the corrected image format text and OCR detection frame;

[0055] A formula detection module is used to perform formula detection on the text in the corrected image format to obtain a formula detection frame and its formula category, wherein the formula category includes an embedded formula and an interline formula;

[0056] a layout analysis module for performing layout analysis on the text in the corrected image format to obtain a layout analysis frame and its layout categories, wherein the layout categories include title, text, image, figure annotation, table, table annotation, formula, header, footer, and references;

[0057] An embedded formula text splitting module is used to determine whether the corrected OCR detection frame contains an embedded formula based on the formula detection frame and split the OCR detection frame containing the embedded formula to obtain an OCR detection frame of the plain text;

[0058] A formula recognition module is used to crop the text in the picture format according to the formula detection frame to obtain a formula area, perform formula recognition on the formula area using a formula recognition model, and simplify the format of the output LaTeX to obtain a formula recognition result;

[0059] The text recognition module is used to perform text recognition on the corrected OCR detection frame using the corresponding OCR recognition model for different scenarios to obtain text recognition results;

[0060] The merging processing module is used to combine the layout analysis frame and its layout category to perform peer detection frame judgment and merge the formula recognition result and the text recognition result to obtain the extraction result of the input text.

[0061] Compared with the prior art, the method and system for intelligently extracting input text containing mathematical formulas of the present invention have one or more of the following beneficial technical effects:

[0062] 1. The present invention provides an innovative solution for extracting input text containing mathematical formulas. It can accurately identify and process mathematical formulas in complex documents. It separates the embedded formulas in the text from the ordinary text and uses the best model to identify them separately, and then merges and restores them, solving the problem of nested output of formula text.

[0063] 2. The present invention adopts different text recognition models in a targeted manner, so that the recognition is accurate and can efficiently process texts mixed with Chinese and English, and can solve the problem of English and numbers sticking together without spaces.

[0064] 3. The present invention realizes the intelligent generation of coherent text lines by identifying whether the recognized text is in the same line, and through internal sorting and merging, and takes into account the layout of the lines in the page layout, so that the text is organized together accurately and orderly, thereby enhancing the coherence and readability of the text processing results. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] FIG1 is a flow chart of the method for intelligently extracting input text containing mathematical formulas according to the present invention.

[0066] FIG2 is a flow chart of the automatic image correction method of the present invention.

[0067] FIG3 is a flow chart of the embedded formula text splitting of the present invention.

[0068] FIG4 is a flow chart of the process of combining the formula recognition result and the text recognition result according to the present invention.

[0069] FIG5 is a schematic diagram showing the structure of the intelligent extraction system for input text containing mathematical formulas according to the present invention. DETAILED DESCRIPTION

[0070] Before describing in detail any embodiment of the present invention, it should be understood that the present invention is not limited in its application to the construction and arrangement details of the components set forth in the following description or illustrated in the following figures. The present invention is capable of other embodiments and can be practiced or carried out in various ways. In addition, it should be understood that the words and terms used herein are for descriptive purposes and should not be considered restrictive. The use of "including" or "having" and their variations herein is intended to cover the items and their equivalents set forth below and additional items. Unless otherwise specified or limited, the terms "mounted", "connected", "supported" and "coupled" and their variations are used broadly and cover direct mounting and indirect mounting, connection, support and coupling. In addition, "connected" and "coupled" are not limited to physical or mechanical connections or couplings.

[0071] Furthermore, on the first hand, in the disclosure of the present invention, the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like to indicate orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore the above terms cannot be understood as limitations on the present invention; on the second hand, the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" cannot be understood as a limitation on the quantity.

[0072] FIG1 shows a flow chart of the method for intelligently extracting input text containing mathematical formulas of the present invention. As shown in FIG1 , the method for intelligently extracting input text containing mathematical formulas of the present invention comprises the following steps:

[0073] 1. Input format judgment and image preprocessing.

[0074] For the input text to be recognized, it is necessary to determine the format of the input text. If the format of the input text is not an image format, the input text is converted into a text in an image format and the text in the image format is preprocessed.

[0075] Specifically, first, it is determined whether the input text is in a picture format. If the input text is not in a picture format, for example, in a PDF format, a PDF parser is used to convert the input text into a picture format.

[0076] Next, the text in the image format is dynamically resized to match the background size, and a median filter algorithm is used to denoise the text in the image format. This dynamic resizing of the background size ensures that the optimal input resolution requirements of multiple subsequent detection and recognition models are met. Denoising can also be used to process the image background and improve image quality. The specific techniques for dynamically resizing the background size and denoising are existing technologies and will not be described in detail here for simplicity.

[0077] 2. Automatic image correction.

[0078] Considering that some PDFs or images of books may have pages rotated 90 degrees, automatic image correction is required.

[0079] When performing automatic image correction, text detection is performed on the text in the preprocessed image format to obtain an OCR detection frame. The aspect ratio of the OCR detection frame and the text direction classifier are used to determine whether to perform angle correction on the text in the preprocessed image format. If angle correction is required, angle correction is performed to obtain the corrected text in the image format and the OCR detection frame.

[0080] Specifically, as shown in FIG2 , the automatic image correction includes:

[0081] 1. Input the preprocessed text in image format into the OCR detection model for text detection to obtain the OCR detection box.

[0082] 2. Calculate the aspect ratio of the OCR detection frame. If the ratio of OCR detection frames whose height is greater than their width exceeds the set threshold, the pre-processed image format text is rotated 90 degrees clockwise to obtain the updated image format text. Otherwise, the pre-processed image format text and OCR detection frame are directly output.

[0083] The threshold in this step can be set to 0.8.

[0084] 3. Input the updated text in the image format into the OCR detection model for text detection to obtain the updated OCR detection box.

[0085] 4. Based on the updated OCR detection frame, the updated image text is cropped to obtain the OCR detection frame area. The OCR detection frame area is then fed into the text direction classifier for direction determination. The text direction classifier outputs either 0 degrees or 180 degrees. If the proportion of the OCR detection frame area outputted as 180 degrees in the total OCR detection frame area is greater than the set threshold, the updated image text is rotated 180 degrees clockwise to obtain the updated image text. Otherwise, the updated image text and the updated OCR detection frame are directly output.

[0086] The text direction classifier can be PaddleOCR's text direction classifier. Furthermore, when cropping the text in the updated image format based on the updated OCR detection frame, the updated image format text can be cropped based on the coordinates of the updated OCR detection frame to obtain multiple OCR detection frame areas, which are then batch-fed into the text direction classifier for direction determination. Furthermore, the threshold in this step can also be set to 0.8.

[0087] 5. Input the updated text in the image format into the OCR detection model for text detection to obtain an updated OCR detection frame, and output the updated text in the image format and the updated OCR detection frame.

[0088] The OCR detection model may adopt the PP-OCRv4 detection model of PaddleOCR.

[0089] Through the above steps, you can obtain text and OCR detection frames in the image format with the correct orientation.

[0090] 3. Perform formula detection on the text in the corrected image format to obtain a formula detection frame and its formula category.

[0091] The formula categories include embedded formulas and interline formulas. In addition, when performing formula detection, the corrected text in image format can be fed into a formula detection model to obtain a formula detection frame and its formula category.

[0092] 4. Perform layout analysis on the text in the corrected image format to obtain a layout analysis frame and its layout category.

[0093] The layout categories include title, text, pictures, figure annotations, tables, table annotations, formulas, headers, footers and references.

[0094] Furthermore, when performing layout analysis, the text in the corrected image format may be fed into a layout analysis model, for example, a PP-PicoDet model, to perform layout analysis to obtain a layout analysis frame and its layout category.

[0095] At the same time, preferably, after obtaining the layout analysis frame, overlap judgment can be performed based on the coordinates of the layout analysis frame. If the layout analysis frame overlaps significantly, for example, the overlap rate exceeds 30%, deduplication processing is performed.

[0096] In the present invention, different processing methods are used for embedded formulas and in-row formulas. For embedded formulas, the following steps 5 to 8 are performed, and for in-row formulas, the following steps 9 to 12 are performed.

[0097] 5. Embedded formula text splitting.

[0098] For OCR detection boxes containing embedded formulas, the formula and text are split to obtain formula detection results, thereby improving recognition accuracy.

[0099] Among them, the formula detection frame with the formula category of embedded formula can be used to determine whether the OCR detection frame with the correct direction obtained in step 2 contains the embedded formula and split the OCR detection frame containing the embedded formula to obtain the OCR detection frame of the plain text.

[0100] In the present invention, the embedded formula text splitting includes matching the embedded formula with the text and segmenting the embedded formula with the text. By matching the embedded formula with the text, it is possible to determine whether an OCR detection frame contains an embedded formula. By segmenting the embedded formula with the text, the OCR detection frame containing the embedded formula can be split into a formula detection frame and an OCR detection frame containing the plain text.

[0101] The embedded formula and text matching is as follows: for each OCR detection box, find the embedded formula in it. The matching conditions are: (1) the center coordinates of the formula detection box with the formula category of embedded formula are within the range of the OCR detection box; (2) the formula detection box with the formula category of embedded formula overlaps with the OCR detection box.

[0102] The specific algorithm for matching condition (1) is:

[0103] 1. Calculate the coordinates of the OCR detection frame and generate the upper and lower boundary lines and left and right boundary line equations of the OCR detection frame;

[0104] 2. Calculate the X coordinate of the center point of the formula detection box whose formula category is embedded formula and substitute it into the upper and lower boundary line equations of the OCR detection box to determine whether the Y coordinate of the center point of the formula detection box whose formula category is embedded formula is within the Y value range of the upper and lower boundary lines;

[0105] 3. Calculate the Y coordinate of the center point of the formula detection box whose formula category is embedded formula and substitute it into the left and right boundary line equations of the OCR detection box to determine whether the X coordinate of the center point of the formula detection box whose formula category is embedded formula is within the X value range of the left and right boundary lines.

[0106] The specific algorithm for matching condition (2) is:

[0107] 1. Calculate the coordinates of the formula detection frame and the OCR detection frame based on the formula category of the embedded formula to determine whether the formula detection frame and the OCR detection frame have an intersection.

[0108] 2. When matching condition (1) is met, the IOU (Intersection over Union, which represents the ratio of the area of ​​the overlapping portion of the formula detection box to the total area) of the formula detection box with the formula category of embedded formula and the OCR detection box is calculated. If the formula detection box with the formula category of embedded formula is within the range of the OCR detection box and there is a certain overlap between the two, the formula detection box with the formula category of embedded formula is added to the corresponding formula list, and the OCR detection box is determined to be the OCR detection box containing the embedded formula.

[0109] The specific steps of embedding formula and text segmentation are: segment the OCR detection frame according to the coordinates of the formula detection frame whose formula category is embedded formula, add the segmented formula detection frame to the corresponding list, and return the result list. The specific algorithm for segmenting the OCR detection frame is:

[0110] 1. If the OCR detection box does not have an embedded formula, return to the original OCR detection box directly.

[0111] 2. If there is an embedded formula in the OCR detection frame, sort the formula detection frames with the formula category of embedded formula in the list obtained above according to the X-axis, process each formula detection frame with the formula category of embedded formula in turn, and split it according to the position of its left boundary and right boundary.

[0112] 3. Ensure that the OCR detection box output after segmentation is a rectangle. If it is not a rectangle, the excess part is also segmented. Finally, return a list of all segmented OCR detection boxes.

[0113] Based on the above principles, as shown in FIG3 , the embedded formula text splitting of the present invention specifically includes:

[0114] 1. For each OCR detection box, obtain the corresponding linear equations of the upper, lower, left and right boundary lines according to the coordinates of its upper left corner, lower left corner, upper right corner and lower right corner.

[0115] 2. Filtering a formula detection frame whose formula category is an embedded formula from the formula detection frame, calculating the X-axis coordinate of the center point of the formula detection frame whose formula category is an embedded formula and substituting it into a linear equation of the upper and lower boundary lines of the OCR detection frame to determine whether the Y-axis coordinate of the center point of the formula detection frame whose formula category is an embedded formula is within the Y value interval of the upper and lower boundary lines of the OCR detection frame, calculating the Y-axis coordinate of the center point of the formula detection frame whose formula category is an embedded formula and substituting it into a linear equation of the left and right boundary lines of the OCR detection frame to determine whether the X-axis coordinate of the center point of the formula detection frame whose formula category is an embedded formula is within the X value interval of the left and right boundary lines of the OCR detection frame;

[0116] 3. If the Y-axis coordinate of the center point of the formula detection box with the formula category of embedded formula is within the Y value range of the upper and lower boundary lines of the OCR detection box, and the X-axis of the center point of the formula detection box with the formula category of embedded formula is within the X value range of the left and right boundary lines of the OCR detection box, then calculate the ratio of the overlapping area of ​​the OCR detection box and the formula category of embedded formula detection box to the total area of ​​the two, that is, IoU;

[0117] 4. Determine whether the IoU is greater than 0. If so, add the formula detection box with the formula category as the embedded formula to the formula list, determine that the OCR detection box is an OCR detection box containing a mathematical formula, and split the OCR detection box.

[0118] If it is determined that the OCR detection frame does not contain an embedded formula, the OCR detection frame is directly returned. If it is determined that the OCR detection frame contains an embedded formula, it is determined whether the OCR detection frame contains one or more embedded formulas. If there are multiple embedded formulas, the multiple formula detection frames with the formula category of embedded formula are sorted on the X-axis, and the OCR detection frames are cut according to the left and right boundary positions of the formula detection frames with the formula category of embedded formula.

[0119] Furthermore, if there is a non-rectangular OCR detection frame, during the cutting process, the redundant non-rectangular portion of the OCR detection frame is cut and discarded, and finally the multiple split OCR detection frames are returned.

[0120] 6. Formula recognition.

[0121] The text in the corrected picture format is cropped according to the formula detection frame to obtain a formula area, and the formula recognition model is used to perform formula recognition on the formula area and simplify the format of the output LaTeX to obtain a formula recognition result.

[0122] Specifically, first, the coordinates of the formula detection frame are cropped from the text in the rectified image format to obtain the formula area. To avoid garbled characters in small image formula recognition, the formula area is dynamically extended with background processing.

[0123] The processed formula area is fed into a formula recognition model, and the output result is in LaTeX format, that is, LaTeX. The formula recognition model can adopt an existing handwritten formula recognition model.

[0124] Then, the output LaTeX is sent to the rendering engine for rendering judgment. If the rendering fails (rendering failure is mainly due to the omission of some paired tokens), the format is repaired. If the rendering still fails after the format repair, an additional error message is output to prevent the output of incorrect formulas. Among them, the rendering engine can use the KaTex rendering engine.

[0125] Finally, the LaTeX is simplified and unnecessary formatting is removed to provide a clearer and more concise formula expression. The final output includes a formula detection box, a formula category (embedded formula), and the LaTeX of the formula.

[0126] 7. Text recognition.

[0127] Use the corresponding OCR recognition model for different scenarios to perform text recognition on the OCR detection box of the plain text to obtain the text recognition result.

[0128] Specifically, first, the plain text OCR detection frame split according to step 5 is cropped in the text in the rectified image format to obtain the OCR detection frame area.

[0129] Then, the OCR detection frame area is fed into the Chinese OCR recognition model for preliminary text recognition to obtain preliminary text recognition results. The Chinese OCR recognition model can adopt the PP-OCRv4 Chinese recognition model of PaddleOCR.

[0130] In order to solve the problem of overlapping spaces between English and numbers and improve the recognition accuracy of English and numerical content, in the present invention, if the preliminary text recognition result only contains English or numbers, the OCR detection frame area is sent to the English OCR recognition model for recognition to obtain the final text recognition result and overwrite the preliminary text recognition result. The English OCR recognition model can adopt PaddleOCR's PP-OCRv4 English recognition model. Therefore, by using different text recognition models in a targeted manner, the recognition is accurate and can efficiently process texts mixed in Chinese and English.

[0131] In the present invention, the final output result of text recognition includes an OCR detection box and a final text recognition result.

[0132] 8. Merger processing.

[0133] Combined with the layout analysis frame and its layout category, the formula recognition result of step six and the text recognition result of step seven are judged and merged in the same detection frame to obtain the extraction result of the input text.

[0134] In the present invention, after the formula recognition result is added to the text recognition result, the merging process continuously evaluates whether the formula recognition result and the text recognition result are located in the same row, and the formula recognition result and the text recognition result confirmed to be in the same row are merged.

[0135] The specific steps of the merge row algorithm are:

[0136] 1. Determine whether the formula detection box and the OCR detection box for the embedded formula are on the same line:

[0137] (1) Initial filtering: When the Y-axis intersection between the formula detection frame and the OCR detection frame of the formula category is embedded formula is greater than the threshold of 0.5, and the X-axis intersection is less than the threshold of 0.5 (screening out the formula detection frame and OCR detection frame that exist in different rows but have overlapping axes), the formula detection frame and OCR detection frame of the formula category are initially filtered out;

[0138] (2) Further confirmation: after the initial filtering is satisfied, the layout analysis frame and its layout category are combined to determine whether the formula detection frame with the formula category of embedded formula and the OCR detection frame are in the same layout;

[0139] (3) After the above two conditions are met, the X_cut algorithm is used to determine whether the formula detection box and the OCR detection box are segmented on the X-axis. If they cannot be segmented, they are determined to be in the same row, otherwise they do not belong to the same row.

[0140] In the present invention, the X_cut algorithm is specifically:

[0141] (1) Detection frame sorting: Sort the formula detection frames and OCR detection frames according to the X-axis coordinates of the formula category embedded formula;

[0142] (2) X-axis projection: Perform X-axis projection based on the sorted formula detection boxes and OCR detection boxes whose formula categories are embedded formulas, and generate a projection array that reflects the distribution of formula detection boxes and OCR detection boxes whose formula categories are embedded formulas;

[0143] (3) Split point confirmation: find the point with a low projection height (e.g., 0) according to the projection array and regard it as a potential split point;

[0144] (4) Segmentation decision: If the potential segmentation point is 1, it indicates that the formula detection frame and OCR detection frame of the formula category is embedded formula cannot be segmented on the X-axis. If there are multiple segmentation points, the formula detection frame and OCR detection frame of the formula category is embedded formula are further segmented according to each segmentation point.

[0145] 2. Internal sorting within a row: Perform internal sorting on formula detection boxes and OCR detection boxes that are confirmed to be embedded formulas in the same row.

[0146] The specific in-row sorting algorithm is as follows: a starting box is specified in the row, and its maximum allowed intersection range on the X-axis is calculated to filter and sort the adjacent formula detection boxes and OCR detection boxes on the left and right sides whose formula categories are embedded formulas. These boxes are added one by one to the result list and used as new starting boxes in turn. This cycle continues until no new formula detection boxes and OCR detection boxes that meet the conditions and whose formula categories are embedded formulas can be found.

[0147] 3. Merge detection frames: Merge the formula detection frames and OCR detection frames with the embedded formula category that are in the same row and have been correctly sorted. The merging methods include: merging text content in order, merging into one detection frame, etc.

[0148] Based on the above principles, as shown in FIG4 , the merging process of the present invention specifically includes:

[0149] 1. Add the formula recognition results to the text recognition results.

[0150] 2. Preliminarily determine whether the formula detection frame and the OCR detection frame for the embedded formula category are in the same row, thereby preliminarily screening out detection frames belonging to the same row. Specifically, if the Y-axis intersection of the formula detection frame and the OCR detection frame is greater than the threshold of 0.5 and the X-axis intersection is less than the threshold of 0.5, the formula detection frame and the OCR detection frame are preliminarily determined to be in the same row.

[0151] 3. Classify the preliminarily screened detection frames into layout categories based on the layout analysis frame and its layout category, and determine whether the preliminarily screened detection frames belong to the same layout category, so as to screen out the detection frames belonging to the same layout.

[0152] 4. Project the selected detection frames belonging to the same layout on the X-axis to determine the segmentation points. Based on the number of segmentation points, determine whether the detection frames can be segmented on the X-axis. If they cannot be segmented, determine that the selected detection frames belonging to the same layout belong to the same industry detection frames. If they can be segmented, determine that the selected detection frames belonging to the same layout do not belong to the same industry detection frames.

[0153] 5. The determined detection boxes of the same line are sorted within the line and then merged. The output extraction result of the input text includes the merged line text detection box and the merged text content.

[0154] 9. For interline formulas, determine whether the formula detection frame significantly overlaps with the corrected OCR detection frame.

[0155] In the present invention, the coordinates of the formula detection frame (e.g., an interline formula) and the coordinates of the corrected OCR detection frame can be used to determine whether they significantly overlap, for example, whether the overlap rate exceeds 90%. If there is significant overlap, it indicates that the corrected OCR detection frame is a pure interline formula OCR detection frame. If there is significant overlap, it indicates that the corrected OCR detection frame is not a pure interline formula OCR detection frame, but rather a normal text OCR detection frame, and text recognition can be performed on it using the text recognition model.

[0156] 10. Identify inter-line formulas.

[0157] In this step, the same method as in step six is ​​used to perform formula recognition on the formula detection box whose formula category is an inter-row formula to obtain a formula recognition result.

[0158] 11. Text recognition.

[0159] In this step, the same method as in step seven is used to perform text recognition on the corrected OCR detection frame that substantially overlaps with the formula detection frame whose formula category is an interline formula, so as to obtain a text recognition result.

[0160] 12. Replacement.

[0161] The text recognition result obtained in step 11 is replaced with the formula recognition result obtained in step 10 to obtain an extraction result of the input text, where the extraction result of the input text is a text extraction result of the inter-line formula.

[0162] In addition, the present invention also relates to an intelligent extraction system for input text containing mathematical formulas. As shown in Figure 5, the intelligent extraction system for input text containing mathematical formulas of the present invention includes:

[0163] An input format determination and image preprocessing module, which is used to determine the format of the input text. If the input text is not in an image format, the input text is converted into an image format and preprocessed.

[0164] An automatic image correction module is used to perform text detection on the pre-processed image format text to obtain an OCR detection frame, determine whether to perform angle correction on the pre-processed image format text based on the aspect ratio and text direction classifier of the OCR detection frame, and perform angle correction if necessary to obtain the corrected image format text and OCR detection frame;

[0165] A formula detection module is used to perform formula detection on the text in the corrected image format to obtain a formula detection frame and its formula category, wherein the formula category includes an embedded formula and an interline formula;

[0166] a layout analysis module for performing layout analysis on the text in the corrected image format to obtain a layout analysis frame and its layout categories, wherein the layout categories include title, text, image, figure annotation, table, table annotation, formula, header, footer, and references;

[0167] An embedded formula text splitting module is used to determine whether the corrected OCR detection frame contains an embedded formula based on the formula detection frame and split the OCR detection frame containing the embedded formula to obtain an OCR detection frame of the plain text;

[0168] A formula recognition module is used to crop the text in the picture format according to the formula detection frame to obtain a formula area, perform formula recognition on the formula area using a formula recognition model, and simplify the format of the output LaTeX to obtain a formula recognition result;

[0169] The text recognition module is used to perform text recognition on the corrected OCR detection frame using the corresponding OCR recognition model for different scenarios to obtain text recognition results;

[0170] The merging processing module is used to combine the layout analysis frame and its layout category to perform peer detection frame judgment and merge the formula recognition result and the text recognition result to obtain the extraction result of the input text.

[0171] This invention provides an innovative solution for extracting input text containing mathematical formulas, enabling accurate recognition and processing of mathematical formulas in complex documents. It separates nested formulas from regular text and identifies each using the optimal model, before merging and recovering them, resolving the problem of nested output of formulaic text. Furthermore, by identifying whether the recognized text is on the same line, the system intelligently generates coherent text lines through internal sorting and merging. Furthermore, the system considers the layout of the lines within the layout, ensuring accurate and orderly organization of the text, enhancing the coherence and readability of the text processing results.

[0172] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art may, based on the principles of the present invention, modify or replace the technical solutions of the present invention with equivalents without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. An intelligent extraction method for input text containing mathematical formulas, characterized in that: The following steps are involved: 1) Determine the format of the input text. If the format of the input text is not a picture format, convert the input text into a picture format text and pre-process the picture format text; 2) performing text detection on the preprocessed image format text to obtain an OCR detection frame, determining whether to perform angle correction on the preprocessed image format text based on the aspect ratio and text direction classifier of the OCR detection frame, and performing angle correction if necessary to obtain the corrected image format text and OCR detection frame; 3) performing formula detection on the corrected text in the image format to obtain a formula detection frame and its formula category, wherein the formula category includes an embedded formula and an interline formula; 4) performing layout analysis on the corrected text in the image format to obtain a layout analysis frame and its layout categories, wherein the layout categories include title, text, image, figure annotation, table, table annotation, formula, header, footer, and references; 5) For the embedded formula, judging whether the corrected OCR detection frame contains the embedded formula based on the formula detection frame and splitting the OCR detection frame containing the embedded formula to obtain the OCR detection frame of the plain text; 6) cropping the text in the corrected image format according to the formula detection frame to obtain a formula area, performing formula recognition on the formula area using a formula recognition model, and simplifying the output LaTeX format to obtain a formula recognition result; 7) Use the corresponding OCR recognition model for different scenarios to perform text recognition on the OCR detection box of the plain text to obtain the text recognition result; 8) Combine the layout analysis frame and its layout category to determine and merge the formula recognition results and text recognition results in the same detection frame to obtain the extraction result of the input text.

2. The intelligent extraction method of input text containing mathematical formulas according to claim 1, characterized in that: Further including: 9) For the interline formula, determine whether the formula detection frame is consistent with the corrected OCR detection frame. If the overlap is large, then go to step 10); 10) cropping the text in the corrected image format according to the formula detection frame to obtain a formula area, performing formula recognition on the formula area using a formula recognition model, and simplifying the output LaTeX format to obtain a formula recognition result; 11) Use the corresponding OCR recognition model for different scenarios to perform text recognition on the corrected OCR detection frame to obtain the text recognition result; 12) The text recognition result is replaced with the formula recognition result to obtain an extraction result of the input text.

3. The intelligent extraction method of input text containing mathematical formulas according to claim 2, characterized in that: The step 1) specifically includes: 1.1) Determine whether the input text is in PDF format or image format. If the input text is in PDF format, use a PDF parser to convert the input text into image format. 1.2) Dynamically extend the size of the text in the image format to match the background size, and use a median filter algorithm to denoise the text in the image format.

4. The intelligent extraction method of input text containing mathematical formulas according to claim 3, characterized in that: The step 2) specifically includes: 2.1) Input the pre-processed text in the image format into the OCR detection model to perform text detection to obtain the OCR detection box; 2.2) Calculate the aspect ratio of the OCR detection frame. If the ratio of OCR detection frames with a height greater than their width exceeds a set threshold, rotate the pre-processed image-format text 90 degrees clockwise to obtain the updated image-format text. Otherwise, directly output the pre-processed image-format text and OCR detection frame. 2.3) Inputting the updated text in the image format into the OCR detection model for text detection to obtain an updated OCR detection frame; 2.4) According to the updated OCR detection frame, the text in the updated image format is cropped to obtain the OCR detection frame area and the OCR detection frame area is sent to the text direction classifier for direction Determine whether the output of the text direction classifier is 0 degrees or 180 degrees. If the proportion of the OCR detection frame area with the output of 180 degrees in the total OCR detection frame area is greater than the set threshold, then the updated image format text is rotated 180 degrees clockwise to obtain the updated image format text again. Otherwise, the updated image format text and the updated OCR detection frame are directly output; 2.5) Input the updated text in the image format into the OCR detection model for text detection to obtain an updated OCR detection frame, and output the updated text in the image format and the updated OCR detection frame.

5. The intelligent extraction method of input text containing mathematical formulas according to claim 4, characterized in that: The step 5) specifically includes: 5.1) For each OCR detection box, obtain the corresponding linear equations of the upper, lower, left, and right boundaries based on the coordinates of its upper left corner, lower left corner, upper right corner, and lower right corner; 5.2) Filtering formula detection frames whose formula category is embedded formula from the formula detection frames, calculating the X-axis coordinate of the center point of the formula detection frame whose formula category is embedded formula and substituting it into the linear equation of the upper and lower boundary lines of the OCR detection frame to determine whether the Y-axis coordinate of the center point of the formula detection frame whose formula category is embedded formula is within the Y value interval of the upper and lower boundary lines of the OCR detection frame, calculating the Y-axis coordinate of the center point of the formula detection frame whose formula category is embedded formula and substituting it into the linear equation of the left and right boundary lines of the OCR detection frame to determine whether the X-axis coordinate of the center point of the formula detection frame whose formula category is embedded formula is within the X value interval of the left and right boundary lines of the OCR detection frame; 5.3) If the Y-axis coordinate of the center point of the embedded formula detection box is within the Y-value range of the upper and lower boundaries of the OCR detection box, and the X-axis coordinate of the center point of the embedded formula detection box is within the X-value range of the left and right boundaries of the OCR detection box, calculate the ratio of the overlapping area of ​​the OCR detection box and the embedded formula detection box to the total area of ​​the two, that is, the Intersection over Union (IoU). 5.4) If the IoU is greater than 0, add the formula detection box with the formula category as embedded formula to the formula list and split the OCR detection box.

6. The intelligent extraction method of input text containing mathematical formulas according to claim 5, characterized in that: The step 6) is specifically as follows: 6.1) Crop the text in the rectified image format according to the coordinates of the formula detection box to obtain the formula area and feed the formula area into the formula recognition model, outputting the result as LaTeX; 6.2) Send the output LaTeX to the rendering engine for rendering judgment. If the rendering fails, the format is repaired. If the rendering still fails after the format repair, an additional error message is output to prevent the output of incorrect formulas; 6.3) Simplify LaTeX and remove unnecessary formats. The final output includes the formula detection box, formula category and LaTeX of the formula.

7. The intelligent extraction method of input text containing mathematical formulas according to claim 6, characterized in that: The step 7) is specifically as follows: 7.1) Crop the OCR detection frame of the split plain text from the rectified image format text to obtain the OCR detection frame area; 7.2) Send the OCR detection frame area to the Chinese OCR recognition model for preliminary text recognition to obtain preliminary text recognition results; 7.3) If the preliminary text recognition result contains only English or numbers, the OCR detection box area is sent to the English OCR recognition model for recognition to obtain the final text recognition result and overwrite the preliminary text recognition result. The final output result includes the OCR detection box and the text recognition result.

8. The intelligent extraction method of input text containing mathematical formulas according to claim 7, characterized in that: The step 8) specifically includes: 8.1) Add the formula recognition results to the text recognition results; 8.2) Preliminarily determine whether the formula detection box and the OCR detection box are in the same row, so as to preliminarily filter out the detection boxes belonging to the same row; 8.3) Classifying the preliminarily screened detection frames according to the layout analysis frame and its layout category, determining whether the preliminarily screened detection frames belong to the same layout category, and selecting detection frames belonging to the same layout; 8.4) Project the selected detection frames belonging to the same layout on the X-axis to determine the split points. Based on the number of split points, determine whether the detection frame can be split on the X-axis. If it cannot be split, determine that the selected detection frames belonging to the same layout belong to the same line of detection frames. If it can be split, determine that the selected detection frames belonging to the same layout do not belong to the same line of detection frames. 8.5) The determined detection boxes in the same row are sorted within the row and then merged. The output extraction result of the input text includes the merged row text detection box and the merged text content.

9. The intelligent extraction method of input text according to claim 8, characterized in that: In step 8.2), when the Y-axis intersection of the formula detection frame and the OCR detection frame is greater than the threshold and the X-axis intersection is less than the threshold, it is preliminarily determined that the formula detection frame and the OCR detection frame are in the same row.

10. An intelligent extraction system for input text containing mathematical formulas, characterized in that: include: An input format determination and image preprocessing module, which is used to determine the format of the input text. If the input text is not in an image format, the input text is converted into an image format and preprocessed. An automatic image correction module is used to perform text detection on the pre-processed image format text to obtain an OCR detection frame, determine whether to perform angle correction on the pre-processed image format text based on the aspect ratio and text direction classifier of the OCR detection frame, and perform angle correction if necessary to obtain the corrected image format text and OCR detection frame; A formula detection module is used to perform formula detection on the text in the corrected image format to obtain a formula detection frame and its formula category, wherein the formula category includes an embedded formula and an interline formula; a layout analysis module for performing layout analysis on the text in the corrected image format to obtain a layout analysis frame and its layout categories, wherein the layout categories include title, text, image, figure annotation, table, table annotation, formula, header, footer, and references; An embedded formula text splitting module is used to determine whether the corrected OCR detection frame contains an embedded formula based on the formula detection frame and split the OCR detection frame containing the embedded formula to obtain an OCR detection frame of the plain text; The formula recognition module is used to detect the formula in the text of the picture format. Cropping to obtain a formula area, performing formula recognition on the formula area using a formula recognition model and simplifying the format of the output LaTeX to obtain a formula recognition result; The text recognition module is used to perform text recognition on the corrected OCR detection frame using the corresponding OCR recognition model for different scenarios to obtain text recognition results; The merging processing module is used to combine the layout analysis frame and its layout category to perform peer detection frame judgment and merge the formula recognition result and the text recognition result to obtain the extraction result of the input text.

Images