Mathematical formula identification method and device, electronic equipment and storage medium

Abstract

The invention provides a mathematical formula identification method and device, electronic equipment and a storage medium. The method comprises the following steps: carrying out hierarchical identification on a mathematical formula, directly identifying an atomic formula to avoid redundancy processing, splitting a composite formula, only processing atomic units with short sequences and shallow structures, reducing long sequence accumulative errors and structural illusion, and adapting to complex scenes such as multiple nesting and special symbols, so as to improve the accuracy of the complex scenes. And the recognition precision of the composite formula is improved.

Description

Mathematical formula recognition methods, devices, electronic equipment and storage media Technical Field

[0001] This application relates to the field of document processing technology, and in particular to a mathematical formula recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] In document processing, mathematical formula recognition is crucial. Existing formula recognition methods mainly employ multimodal models: that is, based on the adaptation of a general vision-language large model, formula recognition is directly processed by relying on pre-trained cross-modal capabilities, without the need to design a separate dedicated module.

[0003] However, complex two-dimensional nested structures of formulas (such as fractional matrices) require precise parsing of hierarchical relationships, while mainstream methods mostly process them according to linear logic, which makes it difficult to deal with multiple nestings, easily leading to mismatched brackets and hierarchical confusion, resulting in inaccurate recognition of mathematical formulas. Summary of the Invention

[0004] Therefore, it is necessary to provide a mathematical formula recognition method, device, electronic device, and storage medium to address the aforementioned technical problems and improve the accuracy of mathematical formula recognition.

[0005] In a first aspect, embodiments of this application provide a mathematical formula recognition method, comprising: acquiring a mathematical formula region image and basic layout information of the mathematical formula region image; identifying the formula type of the mathematical formula region image based on the basic layout information; if the formula type of the mathematical formula region image is an atomic formula, then inputting the atomic formula into a formula recognition model to obtain a first LaTeX code; if the identified formula type of the mathematical formula region image is a compound formula, then dividing the compound formula into multiple atomic formulas; obtaining multiple LaTeX fragments through the formula recognition model, and recombining the multiple LaTeX fragments to obtain a second LaTeX code, thereby completing formula recognition.

[0006] In one embodiment, acquiring the image of the formula region to be identified includes: acquiring the document to be identified; extracting the visual features of the document to be identified; identifying the position information of the formula region image in the document to be identified based on the visual features of the document to be identified; identifying the visual features that indicate the overall layout of the image corresponding to the document to be identified; and cropping the mathematical formula region image from the document to be identified based on the position information while maintaining the original resolution.

[0007] In one embodiment, extracting visual features of the document to be identified and identifying the position information of the mathematical formula region image in the document to be identified based on the visual features of the document to be identified includes: preprocessing the input first resolution document image to generate a second resolution thumbnail; extracting global image features of the second resolution thumbnail through a visual encoder, identifying the mathematical formula region image in the document to be identified, and recording the position information of the mathematical formula region image in the original first resolution document; wherein, the second resolution is greater than the first resolution.

[0008] In one embodiment, identifying the formula type of the mathematical formula region image based on the basic layout information includes: if the basic layout information contains only a single row and there is no cross-row coupling structure, then the mathematical formula region image is determined to be an atomic formula; if the basic layout information contains multi-row derivations, segmented structures, matrix structures, or deep bracket nesting, then the mathematical formula region image is determined to be a composite formula.

[0009] In one embodiment, the composite formula is divided into multiple atomic formulas, including: dividing the composite formula into several sub-rows based on row projection and alignment information to obtain multiple sub-formulas; the type of composite formula includes multi-row derivation, piecewise function, or matrix equation system; each row includes a candidate sub-formula set; in the candidate sub-formula set, a bracket hierarchy tree is constructed based on symbol detection and local connectivity analysis to identify the image regions corresponding to the outer brackets and inner brackets, thus obtaining multiple atomic formulas; in the candidate sub-formula set, for expressions with multiple bracket nesting, fraction nesting, or radical nesting, sub-regions are recursively segmented from the inside to the outside along the bracket hierarchy tree to obtain multiple atomic formulas; wherein, after the segmentation is completed, each atomic formula corresponds to a structural relationship description, which includes the row to which the atomic formula belongs, parent-child nesting relationship, branch order, alignment column information, and position index in the overall formula.

[0010] In one embodiment, sub-region segmentation is performed recursively from the inside to the outside along the bracket hierarchy tree, including: locating the brackets or local structures at the innermost level of the candidate sub-formula image, cropping out the corresponding sub-expression image, and marking it as a candidate atomic formula; recording the position occupied by the candidate atomic formula in the upper-level structure as a placeholder node, and continuing to iterate outward until the entire row is decomposed into a set of atomic formulas.

[0011] In one embodiment, multiple LaTeX fragments are recombined to obtain a second LaTeX code to complete formula recognition, including: for multi-bracket formulas: first, the innermost atomic formula is embedded into the corresponding bracket environment, and then the second LaTeX code is synthesized by nesting it layer by layer outward according to the parent-child nesting relationship of the structural relationship diagram; for multi-line derivation or aligned formulas: an alignment environment is selected, and the LaTeX fragments of each atomic formula are integrated with equal signs or arrows as column alignment anchors to obtain the second LaTeX code; for piecewise functions: a piecewise function environment is used, and the LaTeX fragments corresponding to the expressions of each branch and the LaTeX fragments corresponding to the conditions are sequentially concatenated according to the branch order described by the structural relationship to obtain the second LaTeX code; for parallel formulas: the LaTeX fragments of each atomic formula are directly connected in logical order, or the LaTeX fragments of each atomic formula are concatenated using semicolons or commas to obtain the second LaTeX code.

[0012] Secondly, embodiments of this application provide a mathematical formula recognition device, comprising: an acquisition module for acquiring a mathematical formula region image and basic layout information of the mathematical formula region image; a formula type recognition module for recognizing the formula type of the mathematical formula region image based on the basic layout information; an atomic formula module for inputting the atomic formula into a formula recognition model to obtain a first LaTeX code if the formula type of the mathematical formula region image is an atomic formula; a composite formula module for dividing the composite formula into multiple atomic formulas if the recognized formula type of the mathematical formula region image is a composite formula; obtaining multiple LaTeX fragments through the formula recognition model, and recombining the multiple LaTeX fragments to obtain a second LaTeX code, thereby completing formula recognition.

[0013] In one embodiment, the acquisition module is used to acquire the document to be identified; extract the visual features of the document to be identified; identify the position information of the formula region image in the document to be identified based on the visual features of the document to be identified; crop the mathematical formula region image from the document to be identified based on the position information, while maintaining the original resolution; and the visual features characterize the visual characteristics of the overall layout of the image corresponding to the document to be identified.

[0014] In one embodiment, the acquisition module is used to preprocess the input first-resolution document image to generate a second-resolution thumbnail; extract global image features of the second-resolution thumbnail through a visual encoder, identify the mathematical formula region image in the document to be identified, and record the position information of the mathematical formula region image in the original first-resolution document.

[0015] In one embodiment, the formula type recognition module is used to determine that the mathematical formula region image is an atomic formula if the basic layout information contains only a single row and there is no cross-row coupling structure; and to determine that the mathematical formula region image is a compound formula if the basic layout information contains multi-row derivations, segmented structures, matrix structures or deep bracket nesting.

[0016] In one embodiment, the composite formula module is used to divide the composite formula into several sub-rows based on row projection and alignment information to obtain multiple sub-formulas. The types of composite formulas include multi-row derivations, piecewise functions, or matrix equation systems. Each row includes a set of candidate sub-formulas. In the set of candidate sub-formulas, a bracket hierarchy tree is constructed based on symbol detection and local connectivity analysis to identify the image regions corresponding to the outer brackets and inner brackets, resulting in multiple atomic formulas. In the set of candidate sub-formulas, for expressions with multiple nested brackets, nested fractions, or nested radicals, sub-regions are recursively segmented from the inside out along the bracket hierarchy tree to obtain multiple atomic formulas. After segmentation, each atomic formula has a corresponding structural relationship description, which includes the row to which the atomic formula belongs, the parent-child nesting relationship, the branching order, the alignment column information, and the position index in the overall formula.

[0017] In one embodiment, the composite formula module is used to locate the innermost level of brackets or local structures in the candidate sub-formula image, crop out the corresponding sub-expression image, and mark it as a candidate atomic formula; record the position occupied by the candidate atomic formula in the upper level structure as a placeholder node, and continue to iterate outward until the entire row is decomposed into a set of atomic formulas.

[0018] In one embodiment, the compound formula module is used for: for multi-bracket formulas: first, embedding the innermost atomic formula into the corresponding bracket environment, and then, according to the parent-child nesting relationship of the structural relationship diagram, nesting outward layer by layer to synthesize the second LaTeX code; for multi-line derivation or aligned formulas: selecting an alignment environment, using equal signs or arrows as column alignment anchors, integrating the LaTeX fragments of each atomic formula to obtain the second LaTeX code; for piecewise functions: using a piecewise function environment, according to the branch order described by the structural relationship, sequentially concatenating the LaTeX fragments corresponding to the expressions of each branch and the LaTeX fragments corresponding to the conditions to obtain the second LaTeX code; for parallel formulas: directly connecting the LaTeX fragments of each atomic formula in logical order, or using semicolons or commas to concatenate the LaTeX fragments of each atomic formula to obtain the second LaTeX code.

[0019] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the first aspect and any possible implementation method.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method of the first aspect and any possible implementation.

[0021] The beneficial effects of this application are: by performing hierarchical recognition of mathematical formulas, atomic formulas can be directly recognized to avoid redundant processing, and after decomposing compound formulas, only short sequences and shallow-structured atomic units need to be processed, reducing the cumulative error and structural illusion of long sequences, adapting to complex scenarios such as multiple nesting and special symbols, and solving the problem of inaccurate recognition of high-difficulty formulas by traditional methods. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0023] Figure 1 is an application environment diagram of the mathematical formula recognition method in one embodiment; Figure 2 is a flowchart of the mathematical formula recognition method in one embodiment; Figure 3 is a structural block diagram of the mathematical formula recognition device in one embodiment; Figure 4 is an internal structure diagram of the computer device in one embodiment. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0025] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more, and the embodiments of this application do not impose limitations.

[0026] To facilitate understanding of the technical solutions provided in the embodiments of this application, the design concept of the embodiments of this application is first introduced below: The development of mathematical formula recognition technology has gone through three main stages. The initial stage mainly adopted rule-based and template matching methods, primarily recognizing printed formulas, with limited accuracy and applicability. The traditional machine learning stage combined feature extraction and classifiers, beginning to handle simple handwritten formulas, improving the recognition rate, but still posing difficulties for complex formulas. The deep learning revolution stage, based on deep neural network architecture, achieved end-to-end training, eliminating the need for manual feature engineering, significantly improving accuracy, and enabling the processing of complex handwritten formulas.

[0027] However, the complex structure of mathematical formulas (e.g., multi-line derivations, piecewise functions, or systems of matrix equations) presents a core challenge for recognition. These structures are not simple linear arrangements but contain multiple levels of nested relationships, such as fractions nested within matrices, radicals nested within complex bracket expressions, and matrix elements containing subscripts, superscripts, and fractions. Most mainstream recognition methods follow the linear processing logic used for ordinary text, lacking the ability to accurately model the two-dimensional hierarchy of formulas. This makes it difficult to effectively decompose the structural boundaries of different levels and accurately capture the hierarchical relationships between components. This easily leads to problems such as mismatched brackets (e.g., missing inner brackets, incorrect closure of outer brackets), confused hierarchical relationships (e.g., misclassifying expressions within the numerator as the denominator, incorrect determination of the radicand range of nested radicals), and incorrect assignment of subscripts and superscripts. Ultimately, the recognition results of complex formulas deviate significantly from the original formulas, greatly reducing recognition accuracy.

[0028] Therefore, this application proposes a mathematical formula recognition method, device, electronic device, and storage medium. By performing hierarchical recognition of mathematical formulas, atomic formulas can be directly recognized to avoid redundant processing. After decomposing compound formulas, only short sequences and shallow-structured atomic units need to be processed, reducing the cumulative error and structural illusion of long sequences. It is suitable for complex scenarios such as multiple nesting and special symbols, and solves the problem of inaccurate recognition of high-difficulty formulas by traditional methods.

[0029] The mathematical formula recognition method provided in this application can be applied to the application environment shown in Figure 1. In this method, terminal 102 communicates with server 104 via a network. Terminal 102 can acquire an image of the mathematical formula region and its basic layout information, and then send the data to be processed to server 104 via the network, enabling server 104 to perform formula recognition using the image of the mathematical formula region and its basic layout information.

[0030] The terminal 102 can be of different types of data acquisition devices. The terminal 102 can be, but is not limited to, a vehicle-mounted camera, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0031] The following is a detailed description of a mathematical formula recognition method proposed in this application: In one embodiment, as shown in Figure 2, a mathematical formula recognition method is provided. Taking the application of this method to the server in Figure 1 as an example, the method includes the following steps: Step S201, obtaining the mathematical formula region image and the basic layout information of the mathematical formula region image.

[0032] In some possible embodiments, acquiring the image of the formula region to be identified includes: acquiring the document to be identified; extracting the visual features of the document to be identified, and identifying the position information of the formula region image in the document to be identified based on the visual features of the document to be identified; cropping the mathematical formula region image from the document to be identified based on the position information while maintaining the original resolution; and identifying the visual features of the overall layout of the image corresponding to the document to be identified.

[0033] In some possible embodiments, extracting visual features of the document to be identified and identifying the position information of the mathematical formula region image in the document to be identified based on the visual features of the document to be identified includes: preprocessing the input first resolution document image to generate a second resolution thumbnail; extracting global image features of the second resolution thumbnail through a visual encoder, identifying the mathematical formula region image in the document to be identified, and recording the position information of the mathematical formula region image in the original first resolution document; wherein, the second resolution is greater than the first resolution.

[0034] As an example, acquiring the document to be identified includes a global layout analysis and a local content recognition stage. First, the input first-resolution document image is scaled (e.g., a high-resolution image of 5000×7000 pixels), generating a second-resolution thumbnail (e.g., a low-resolution thumbnail reduced to 1036×1036 pixels). Then, the pre-processed document undergoes layout structure detection. This involves inputting the thumbnail into a visual encoder to extract global image features, and then identifying various layout elements in the document, including paragraph text blocks, mathematical formula areas, tables, images, illustrations, titles, headers, and footers. For each detected layout element, its location information (boundary box coordinates) and category label are output. For detected formula areas, the corresponding position of each layout element in the original high-resolution document is recorded. By performing global analysis on a low-resolution image, the number of visual image blocks that need to be processed is significantly reduced, significantly lowering computational overhead. Simultaneously, it avoids intensive computation on a full-resolution image, enabling the model to quickly and roughly obtain the document's structural relationships from a global perspective.

[0035] For the formula regions to be detected, image fragments of each formula region's layout elements are cropped from the original first-resolution document image to obtain the mathematical formula region image. This allows for the identification of atomic formulas at the original resolution, preserving all detailed information and preventing the loss of small characters or complex symbols due to downsampling.

[0036] Next, the mathematical formula region image needs to be identified: Step S202, identify the formula type of the mathematical formula region image based on the basic layout information.

[0037] In some possible embodiments, identifying the formula type of the mathematical formula region image based on the basic layout information includes: if the basic layout information contains only a single row and there is no cross-row coupling structure, then the mathematical formula region image is determined to be an atomic formula; if the basic layout information contains multi-row derivations, segmented structures, matrix structures, or deep nested brackets, then the mathematical formula region image is determined to be a composite formula.

[0038] Before introducing formula recognition, let's first define formula hierarchy: An atomic formula is the smallest indivisible mathematical expression unit, structurally exhibiting a unidirectional and shallowly nested structure. Further splitting it would destroy its semantics or integrity. For example, an atomic formula includes: a single subscript / superscript expression: , etc.; a single-line expression within a single fraction, such as In and The innermost expression in a multi-bracket structure: such as In Single radical: inner expression in .

[0039] Complex formulas are composed of multiple atomic formulas combined through structures such as rows, columns, alignment points, nested brackets, and nested fractions. For example, complex formulas include multi-line derivations; piecewise functions; multiple radicals and multiple fractions; and deeply nested bracket expressions.

[0040] Step S203: If the formula type of the mathematical formula region image is an atomic formula, then input the atomic formula into the formula recognition model to obtain the first LaTeX code.

[0041] Step S204: If the formula type of the mathematical formula region image is a compound formula, then the compound formula is divided into multiple atomic formulas; multiple LaTeX fragments are obtained through the formula recognition model, and the multiple LaTeX fragments are recombined to obtain the second LaTeX code, thus completing the formula recognition.

[0042] In some possible embodiments, the composite formula is divided into multiple atomic formulas, including: dividing the composite formula into several sub-rows based on row projection and alignment information to obtain multiple sub-formulas; the type of the composite formula includes multi-row derivation, piecewise function, or matrix equation system; each row includes a candidate sub-formula set; in the candidate sub-formula set, a bracket hierarchy tree is constructed based on symbol detection and local connectivity analysis to identify the image regions corresponding to the outer brackets and inner brackets, thus obtaining multiple atomic formulas; in the candidate sub-formula set, for expressions with multiple bracket nesting, fraction nesting, or radical nesting, sub-regions are recursively segmented from the inside to the outside along the bracket hierarchy tree to obtain multiple atomic formulas; wherein, after the segmentation is completed, each atomic formula corresponds to a structural relationship description, which includes the row to which the atomic formula belongs, parent-child nesting relationship, branch order, alignment column information, and position index in the overall formula.

[0043] In some possible embodiments, sub-region segmentation is performed recursively from the inside to the outside along the bracket hierarchy tree, including: locating the brackets or local structures at the innermost level of the candidate sub-formula image, cropping out the corresponding sub-expression image, and marking it as a candidate atomic formula; recording the position occupied by the candidate atomic formula in the upper-level structure as a placeholder node, and continuing to iterate outward until the entire row is decomposed into a set of atomic formulas.

[0044] Traditional approaches treat the entire formula as a single sequence, which can easily lead to structural illusions. This application, through hierarchical modeling, decomposes complex problems into simpler subproblems, fundamentally avoiding the risk of generating long sequences.

[0045] As an example, the following describes the atomic decomposition and recombination formula recognition of this application: First, the mathematical formula region image is classified according to basic layout information, which includes the number of rows, alignment, bracket and fraction distribution, etc. For each mathematical formula region image, its features such as the number of rows, alignment mode and bracket nesting depth are analyzed. If the region contains only a single row, has a shallow nesting level and no cross-row coupling structure, it is determined to be an atomic formula region. If the region contains multi-line derivation, segmented structure, matrix structure or deep bracket nesting, it is determined to be a complex formula region.

[0046] For the atomic formula region: input the atomic formula image into the formula recognition model and output the corresponding first LaTeX code.

[0047] The formula recognition model is trained based on the Transformer encoder-decoder structure. The encoder extracts the spatial visual features of atomic formulas, while the decoder generates the corresponding LaTeX sequences autoregressively under an attention mechanism. Since atomic formulas are structurally one-dimensional and finite in length, the model only needs to process short sequences and shallow-structure formulas at this level, significantly reducing the decoding difficulty and the risk of structural illusion. During the training phase, specialized transcription tasks are designed for common atomic types (sub-expressions within parentheses, single-row numerators or denominators, single rows of matrices, etc.), enabling the model to obtain sufficient prior knowledge at the atomic granularity.

[0048] For complex formula regions: record their structural features and layout information, and perform atomic-level structural decomposition: perform hierarchical decomposition on the composite formula region image: based on row projection and alignment information, divide multi-row derivations, piecewise functions, or matrix equation systems into several sub-rows; each row is regarded as a candidate atomic formula set after segmentation.

[0049] Then, the set of candidate atomic formulas for each row is decomposed into bracket hierarchy and local structure: within each row, a bracket hierarchy tree is constructed based on symbol detection and local connectivity analysis to identify the image regions corresponding to the outer brackets and inner brackets; for expressions with multiple bracket nesting, fraction nesting, or radical nesting, sub-region segmentation is recursively performed from the inside to the outside along the bracket hierarchy tree: first, the brackets or local structures at the innermost level are located, and the corresponding sub-expression images are cropped, and the position occupied by the sub-expression in the upper-level structure is recorded as a placeholder node; the iteration continues to the outer layers until the entire row is decomposed into a set of atomic formulas and their nesting relationships.

[0050] Through the above segmentation, the composite formula image region is divided into a group of atomic formula images (including atomic rows obtained from multi-row segmentation, and atomic sub-expressions at the innermost level of brackets, etc.; structural relationship descriptions corresponding to each atomic formula (row to which it belongs, parent-child nesting relationship, alignment column information, position index in the overall formula, etc.).

[0051] It can be seen that any complex formula is equivalently transformed into several atomic formulas and a structural relationship diagram, laying the foundation for subsequent atomic-level high-precision recognition.

[0052] The atomic formula image is input into the formula recognition model, which outputs the corresponding LaTeX code segments, along with the structural identifier of the atomic formula. Based on the structural relationship diagram, several atomic formulas are recombined to obtain the second LaTeX code.

[0053] Furthermore, for formula recognition models, difficulty example mining and iterative optimization can be performed based on metrics. For instance, the recognition results can be evaluated using the Character Distance Metric (CDM), with the character distance metric used in the project serving as the evaluation indicator. Difficult example mining and iterative optimization of the formula recognition model can then be based on this indicator. The CDM is defined as: calculating the edit distance between multiple output results when recognizing the same formula multiple times. Its formula is as follows:

[0054] in, The number of times the same sample is repeatedly identified (usually taken as...) ), and The first Second and third The result string of the second recognition, The Levenshtein edit distance (CDM) is the distance between the two outputs. CDM is a classic metric for measuring the similarity between two strings. Its core function is to calculate the minimum number of character edits required to transform one string into another; in other words, it's the minimum number of character edits needed to change one output into another. A smaller CDM value indicates a more stable model recognition of that sample; a larger CDM value indicates uncertainty in the model regarding that sample, making it more prone to errors.

[0055] Then, an iterative mining strategy is implemented based on the evaluation metric CDM: First, hard example mining is carried out by performing multiple inferences (usually 5 times) on the samples in the training set and calculating the CDM value of each sample; a threshold is set, and samples with CDM values ​​higher than the threshold are marked as hard examples; the characteristics of hard examples are analyzed (such as formula length, symbol complexity, nesting level, etc.).

[0056] Secondly, there is sample-weighted training. In the next round of training, the sampling frequency of difficult examples is increased; a larger loss weight is used for difficult examples to strengthen the model's learning of such samples; and sampling parameters are dynamically adjusted to avoid overfitting.

[0057] Finally, iterative optimization is performed, repeating the above mining and training process for 3-5 rounds; after each round, the model's performance on the validation set is evaluated; iteration stops when the performance improvement approaches saturation. Through this strategy, the model can automatically identify and strengthen weak points, significantly improving the accuracy in recognizing difficult examples such as complex formulas, multi-line derivations, and formulas containing Chinese variable names. Experiments show that after 3 rounds of optimization, the model's accuracy on long formula recognition tasks improves by 15-20%.

[0058] As can be seen, by segmenting complex formulas as described above, the problem of identifying a large number of independent atomic formulas is transformed, which weakens the impact of long sequences and deep nesting on the recognition accuracy.

[0059] In some possible embodiments, multiple LaTeX fragments are recombined to obtain a second LaTeX code to complete formula recognition, including: for multi-bracket formulas: first, the innermost atomic formula is embedded into the corresponding bracket environment, and then the second LaTeX code is synthesized by nesting it layer by layer outward according to the parent-child nesting relationship of the structural relationship diagram; for multi-line derivation or aligned formulas: an alignment environment is selected, and the LaTeX fragments of each atomic formula are integrated with equal signs or arrows as column alignment anchors to obtain the second LaTeX code; for piecewise functions: a piecewise function environment is used, and the LaTeX fragments corresponding to the expressions of each branch and the LaTeX fragments corresponding to the conditions are sequentially concatenated according to the branch order described by the structural relationship to obtain the second LaTeX code; for parallel formulas: the LaTeX fragments of each atomic formula are directly connected in logical order, or the LaTeX fragments of each atomic formula are concatenated using semicolons or commas to obtain the second LaTeX code.

[0060] As can be seen, by replacing placeholders from the inside out according to the structural relationship diagram, nested structures such as multiple parentheses, fractions, and radicals are specifically restored, avoiding hierarchical confusion and solving the problem of inaccurate parsing of two-dimensional nested formulas by traditional methods. Based on the alignment information, it adapts to the corresponding LaTeX environment, ensuring the alignment of multi-line formulas, the standardization of piecewise function format, and the logical coherence of parallel formulas. It restores the visual and logical layout of the original formula. Based on the reorganization of structured rules rather than the free generation of models, the recognition error of a single atomic formula will not spread to the whole, ensuring the stability and reliability of the overall recognition of complex formulas.

[0061] By decomposing complex formulas into multiple atomic formulas, the difficulty of identifying each atomic formula is significantly reduced, and the accuracy of identifying a single atomic formula is greatly improved. By using structural recombination rather than end-to-end generation, the cumulative error and structural illusion problems inherent in long sequence generation are avoided. It should be understood that although the steps in the flowchart of Figure 2 are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order constraint on the execution of these steps; they can be executed in other orders. Furthermore, at least some steps in Figure 2 may include multiple sub-steps or stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is also not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0062] Based on the same inventive concept, this application provides a mathematical formula recognition device, including: an acquisition module 301, a formula type recognition module 302, an atomic formula module 303, and a composite formula module 304, wherein: the acquisition module is used to acquire a mathematical formula region image and basic layout information of the mathematical formula region image.

[0063] The acquisition module is used to acquire the mathematical formula region image and its basic layout information.

[0064] The formula type recognition module is used to identify the formula type of the mathematical formula area image based on the basic layout information.

[0065] The atomic formula module, if the formula type of the mathematical formula region image is atomic formula, is used to input the atomic formula into the formula recognition model to obtain the first LaTeX code.

[0066] The composite formula module, if the formula type of the mathematical formula region image is a composite formula, is used to divide the composite formula into multiple atomic formulas; multiple LaTeX fragments are obtained through the formula recognition model, and the multiple LaTeX fragments are recombined to obtain the second LaTeX code, thus completing the formula recognition.

[0067] In one embodiment, the acquisition module is used to acquire the document to be identified; extract the visual features of the document to be identified; identify the position information of the formula region image in the document to be identified based on the visual features of the document to be identified; crop the mathematical formula region image from the document to be identified based on the position information, while maintaining the original resolution; and the visual features characterize the visual characteristics of the overall layout of the image corresponding to the document to be identified.

[0068] In one embodiment, the acquisition module is used to preprocess the input first-resolution document image to generate a second-resolution thumbnail; extract global image features of the second-resolution thumbnail through a visual encoder, identify the mathematical formula region image in the document to be identified, and record the position information of the mathematical formula region image in the original first-resolution document; wherein, the second resolution is greater than the second resolution.

[0069] In one embodiment, the formula type recognition module is used to determine that the mathematical formula region image is an atomic formula if the basic layout information contains only a single row and there is no cross-row coupling structure; and to determine that the mathematical formula region image is a compound formula if the basic layout information contains multi-row derivations, segmented structures, matrix structures or deep bracket nesting.

[0070] In one embodiment, the composite formula module is used to divide the composite formula into several sub-rows based on row projection and alignment information to obtain multiple sub-formulas. The types of composite formulas include multi-row derivations, piecewise functions, or matrix equation systems. Each row includes a set of candidate sub-formulas. In the set of candidate sub-formulas, a bracket hierarchy tree is constructed based on symbol detection and local connectivity analysis to identify the image regions corresponding to the outer brackets and inner brackets, resulting in multiple atomic formulas. In the set of candidate sub-formulas, for expressions with multiple nested brackets, nested fractions, or nested radicals, sub-regions are recursively segmented from the inside out along the bracket hierarchy tree to obtain multiple atomic formulas. After segmentation, each atomic formula has a corresponding structural relationship description, which includes the row to which the atomic formula belongs, the parent-child nesting relationship, the branching order, the alignment column information, and the position index in the overall formula.

[0071] In one embodiment, the composite formula module is used to locate the innermost level of brackets or local structures in the candidate sub-formula image, crop out the corresponding sub-expression image, and mark it as a candidate atomic formula; record the position occupied by the candidate atomic formula in the upper level structure as a placeholder node, and continue to iterate outward until the entire row is decomposed into a set of atomic formulas.

[0072] In one embodiment, the compound formula module is used for: for multi-bracket formulas: first, embedding the innermost atomic formula into the corresponding bracket environment, and then, according to the parent-child nesting relationship of the structural relationship diagram, nesting outward layer by layer to synthesize the second LaTeX code; for multi-line derivation or aligned formulas: selecting an alignment environment, using equal signs or arrows as column alignment anchors, integrating the LaTeX fragments of each atomic formula to obtain the second LaTeX code; for piecewise functions: using a piecewise function environment, according to the branch order described by the structural relationship, sequentially concatenating the LaTeX fragments corresponding to the expressions of each branch and the LaTeX fragments corresponding to the conditions to obtain the second LaTeX code; for parallel formulas: directly connecting the LaTeX fragments of each atomic formula in logical order, or using semicolons or commas to concatenate the LaTeX fragments of each atomic formula to obtain the second LaTeX code.

[0073] Specific limitations regarding the mathematical formula recognition device can be found in the limitations of the speech recognition method above, and will not be repeated here. Each module in the aforementioned mathematical formula recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independently of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0074] Based on the same inventive concept, this application provides a computer device, which can be a server, and its internal structure is shown in Figure 4. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a mathematical formula recognition method.

[0075] Those skilled in the art will understand that the structure shown in Figure 4 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0076] Based on the same inventive concept, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: acquiring a mathematical formula region image and basic layout information of the mathematical formula region image; identifying the formula type of the mathematical formula region image based on the basic layout information; if the formula type of the mathematical formula region image is an atomic formula, inputting the atomic formula into a formula recognition model to obtain first LaTeX code; if the identified formula type of the mathematical formula region image is a compound formula, dividing the compound formula into multiple atomic formulas; obtaining multiple LaTeX fragments through the formula recognition model, recombining the multiple LaTeX fragments to obtain second LaTeX code, thus completing formula recognition.

[0077] In one embodiment, when the processor executes the computer program, it further performs the following steps: acquiring a document to be identified; extracting visual features of the document to be identified, and identifying the position information of the formula region image in the document to be identified based on the visual features of the document to be identified; cropping the mathematical formula region image from the document to be identified based on the position information, while maintaining the original resolution; and identifying the visual features of the overall layout of the image corresponding to the document to be identified.

[0078] In one embodiment, when the processor executes the computer program, it further performs the following steps: preprocessing the input first-resolution document image to generate a second-resolution thumbnail; extracting global image features of the second-resolution thumbnail through a visual encoder, identifying the mathematical formula region image in the document to be identified, and recording the position information of the mathematical formula region image in the original first-resolution document; wherein, the second resolution is greater than the second resolution.

[0079] In one embodiment, when the processor executes the computer program, it further implements the following steps: if the basic layout information contains only a single row and there is no cross-row coupling structure, then the mathematical formula region image is determined to be an atomic formula; if the basic layout information contains multi-row derivations, segmented structures, matrix structures, or deep bracket nesting, then the mathematical formula region image is determined to be a composite formula.

[0080] In one embodiment, when the processor executes the computer program, it further performs the following steps: dividing the composite formula into several sub-rows based on row projection and alignment information to obtain multiple sub-formulas; the type of the composite formula includes multi-row derivation, piecewise function, or matrix equation system; each row includes a candidate sub-formula set; in the candidate sub-formula set, constructing a bracket hierarchy tree based on symbol detection and local connectivity analysis to identify the image regions corresponding to the outer brackets and inner brackets, obtaining multiple atomic formulas; in the candidate sub-formula set, for expressions with multiple bracket nesting, fraction nesting, or radical nesting, recursively dividing the sub-regions from the inside to the outside along the bracket hierarchy tree to obtain multiple atomic formulas; wherein, after the division is completed, each atomic formula corresponds to a structural relationship description, which includes the row to which the atomic formula belongs, parent-child nesting relationship, branch order, alignment column information, and position index in the overall formula.

[0081] In one embodiment, when the processor executes the computer program, it also performs the following steps: locating the innermost level of brackets or local structures in the candidate sub-formula image, cropping out the corresponding sub-expression image, and marking it as a candidate atomic formula; recording the position occupied by the candidate atomic formula in the upper-level structure as a placeholder node, and continuing to iterate outwards until the entire row is decomposed into a set of atomic formulas.

[0082] In one embodiment, when the processor executes the computer program, it also performs the following steps: For multi-bracket formulas: first, the innermost atomic formula is embedded into the corresponding bracket environment, and then the second LaTeX code is synthesized by nesting the parent-child nesting relationship of the structural relationship diagram layer by layer outward; For multi-line derivation or aligned formulas: an alignment environment is selected, and the LaTeX fragments of each atomic formula are integrated with equal signs or arrows as column alignment anchors to obtain the second LaTeX code; For piecewise functions: a piecewise function environment is used, and the LaTeX fragments corresponding to the expressions of each branch and the LaTeX fragments corresponding to the conditions are sequentially concatenated according to the branch order described by the structural relationship to obtain the second LaTeX code; For parallel formulas: the LaTeX fragments of each atomic formula are directly connected in logical order, or the LaTeX fragments of each atomic formula are concatenated using semicolons or commas to obtain the second LaTeX code.

[0083] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it performs the following steps: acquiring a mathematical formula region image and basic layout information of the mathematical formula region image; identifying the formula type of the mathematical formula region image based on the basic layout information; if the formula type of the mathematical formula region image is an atomic formula, inputting the atomic formula into a formula recognition model to obtain first LaTeX code; if the identified formula type of the mathematical formula region image is a compound formula, dividing the compound formula into multiple atomic formulas; obtaining multiple LaTeX fragments through the formula recognition model, recombining the multiple LaTeX fragments to obtain second LaTeX code, thus completing formula recognition.

[0084] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: acquiring a document to be identified; extracting visual features of the document to be identified, and identifying the position information of the formula region image in the document to be identified based on the visual features of the document to be identified; cropping the mathematical formula region image from the document to be identified based on the position information, while maintaining the original resolution; and identifying the visual features of the overall layout of the image corresponding to the document to be identified.

[0085] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: In one embodiment, when the computer program is executed by the processor, it further performs the following steps: preprocessing the input first resolution document image to generate a second resolution thumbnail; extracting global image features of the second resolution thumbnail through a visual encoder, identifying the mathematical formula region image in the document to be identified, and recording the position information of the mathematical formula region image in the original first resolution document; wherein, the second resolution is greater than the second resolution.

[0086] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: if the basic layout information contains only a single row and there is no cross-row coupling structure, then the mathematical formula region image is determined to be an atomic formula; if the basic layout information contains multi-row derivations, segmented structures, matrix structures, or deep bracket nesting, then the mathematical formula region image is determined to be a composite formula.

[0087] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: dividing the composite formula into several sub-rows based on row projection and alignment information to obtain multiple sub-formulas; the type of the composite formula includes multi-row derivation, piecewise function, or matrix equation system; each row includes a candidate sub-formula set; in the candidate sub-formula set, constructing a bracket hierarchy tree based on symbol detection and local connectivity analysis to identify the image regions corresponding to the outer brackets and inner brackets, obtaining multiple atomic formulas; in the candidate sub-formula set, for expressions with multiple bracket nesting, fraction nesting, or radical nesting, recursively dividing the sub-regions from the inside to the outside along the bracket hierarchy tree to obtain multiple atomic formulas; wherein, after the division is completed, each atomic formula corresponds to a structural relationship description, which includes the row to which the atomic formula belongs, parent-child nesting relationship, branch order, alignment column information, and position index in the overall formula.

[0088] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: locating the brackets or local structures at the innermost level of the candidate sub-formula image, cropping out the corresponding sub-expression image, and marking it as a candidate atomic formula; recording the position occupied by the candidate atomic formula in the upper-level structure as a placeholder node, and continuing to iterate outwards until the entire row is decomposed into a set of atomic formulas.

[0089] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: For multi-bracket formulas: first, the innermost atomic formula is embedded into the corresponding bracket environment, and then the second LaTeX code is synthesized by nesting the parent-child nesting relationship of the structural relationship diagram layer by layer outward; For multi-line derivation or aligned formulas: an alignment environment is selected, and the LaTeX fragments of each atomic formula are integrated with equal signs or arrows as column alignment anchors to obtain the second LaTeX code; For piecewise functions: a piecewise function environment is used, and the LaTeX fragments corresponding to the expressions of each branch and the LaTeX fragments corresponding to the conditions are sequentially concatenated according to the branch order described by the structural relationship to obtain the second LaTeX code; For parallel formulas: the LaTeX fragments of each atomic formula are directly connected in logical order, or the LaTeX fragments of each atomic formula are concatenated using semicolons or commas to obtain the second LaTeX code.

[0090] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0091] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for recognizing mathematical formulas, characterized in that, include: Obtain the mathematical formula region image and the basic layout information of the mathematical formula region image; The formula type of the mathematical formula region image is identified based on the basic layout information; If the formula type of the mathematical formula region image is an atomic formula, then the atomic formula is input into the formula recognition model to obtain the first LaTeX code; if the formula type of the mathematical formula region image is a compound formula, then the compound formula is divided into multiple atomic formulas; multiple LaTeX fragments are obtained through the formula recognition model, and the multiple LaTeX fragments are recombined to obtain the second LaTeX code, thus completing the formula recognition.

2. The method according to claim 1, characterized in that, Acquiring an image of a formula region to be identified includes: acquiring a document to be identified; extracting visual features from the document to be identified; identifying the position information of the formula region image in the document to be identified based on the visual features; wherein the visual features characterize the overall visual layout of the image corresponding to the document to be identified; and cropping the mathematical formula region image from the document to be identified based on the position information while maintaining the original resolution.

3. The method according to claim 2, characterized in that, Extracting visual features from the document to be identified, and identifying the position information of the mathematical formula region image in the document to be identified based on the visual features of the document to be identified, includes: preprocessing the input first-resolution document image to generate a second-resolution thumbnail; extracting global image features from the second-resolution thumbnail through a visual encoder, identifying the mathematical formula region image in the document to be identified, and recording the position information of the mathematical formula region image in the original first-resolution document; wherein, the second resolution is greater than the first resolution.

4. The method according to claim 1, characterized in that, Identifying the formula type of the mathematical formula region image based on the basic layout information includes: if the basic layout information contains only a single row and there is no cross-row coupling structure, then the mathematical formula region image is determined to be an atomic formula; if the basic layout information contains multi-row derivations, segmented structures, matrix structures, or deep nested brackets, then the mathematical formula region image is determined to be a compound formula.

5. The method according to claim 4, characterized in that, The composite formula is divided into multiple atomic formulas, including: dividing the composite formula into several sub-rows based on row projection and alignment information to obtain multiple sub-formulas; the type of the composite formula includes multi-row derivation, piecewise function, or matrix equation system; each row includes a candidate sub-formula set; in the candidate sub-formula set, a bracket hierarchy tree is constructed based on symbol detection and local connectivity analysis to identify the image regions corresponding to the outer brackets and inner brackets, thus obtaining multiple atomic formulas; in the candidate sub-formula set, for expressions with multiple bracket nesting, fraction nesting, or radical nesting, sub-region segmentation is recursively performed from the inside to the outside along the bracket hierarchy tree to obtain multiple atomic formulas; wherein, after the segmentation is completed, each atomic formula corresponds to a structural relationship description, the structural relationship description including the row to which the atomic formula belongs, parent-child nesting relationship, branch order, alignment column information, and position index in the overall formula.

6. The method according to claim 5, characterized in that, The sub-regions are recursively segmented from the inside out along the bracket hierarchy tree, including: locating the brackets or local structures at the innermost level of the candidate sub-formulas, cropping out their corresponding sub-expression images, and marking them as candidate atomic formulas; recording the position of the candidate atomic formula in the upper-level structure as a placeholder node, and continuing to iterate outwards until the entire row is decomposed into a set of atomic formulas.

7. The method according to claim 5 or 6, characterized in that, The process of recombining multiple LaTeX fragments to obtain a second LaTeX code includes: For multi-bracket formulas: first, embed the innermost atomic formula into the corresponding bracket environment, and then, according to the parent-child nesting relationship of the structural relationship diagram, nest outwards layer by layer to synthesize the second LaTeX code; For multi-line derivation or aligned formulas: select an alignment environment, use equal signs or arrows as column alignment anchors, and integrate the LaTeX fragments of each atomic formula to obtain the second LaTeX code; For piecewise functions: use a piecewise function environment, and according to the branch order described by the structural relationship, sequentially concatenate the LaTeX fragments corresponding to the expressions of each branch and the LaTeX fragments corresponding to the conditions to obtain the second LaTeX code; For parallel formulas: directly connect the LaTeX fragments of each atomic formula in logical order, or use semicolons or commas to concatenate the LaTeX fragments of each atomic formula to obtain the second LaTeX code.

8. A mathematical formula recognition device, characterized in that, include: The acquisition module is used to acquire the mathematical formula region image and the basic layout information of the mathematical formula region image; The formula type recognition module is used to identify the formula type of the mathematical formula region image based on the basic layout information; The atomic formula module is used to input the atomic formula into the formula recognition model to obtain the first LaTeX code if the formula type of the mathematical formula region image is an atomic formula. The composite formula module is used to divide the composite formula into multiple atomic formulas if the formula type of the mathematical formula region image is identified as a composite formula; to obtain multiple LaTeX fragments through the atomic formula recognition model; and to reassemble the multiple LaTeX fragments to obtain a second LaTeX code.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the mathematical formula recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor and implementing the mathematical formula recognition method as described in any one of claims 1 to 7.

Images