Text image and formula image unified identification method, system, storage medium and device

Abstract

The application provides a unified recognition method, system, storage medium and equipment for text images and formula images, and belongs to the technical field of computer vision and natural language processing. The application explicitly encodes the document hierarchy by inserting line end markers and paragraph end markers into the training label sequence; and constructs a semantic decoupling segmenter to assign independent identifiers to the same symbols in the text and the formula, thereby solving the semantic confusion problem. After extracting image features based on a visual encoder, autoregressive training is performed in combination with a decoder to achieve high-precision unified recognition of text, formula and mixed images. The application achieves recognition accuracy comparable to large-scale models at only 0.1B parameter scale, significantly improves the inference speed, and can be widely used in document digitization, academic literature processing and other scenes.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and natural language processing, and relates to a unified recognition method, system, storage medium and equipment for text images and formula images. BACKGROUND

[0002] Document parsing aims to automatically identify the layout structure and reading order of a document, and parse and extract different modal contents such as text, tables, formulas, seals, etc. Under the current wave of artificial intelligence large models, the importance of document parsing is increasingly prominent: it is not only a key component in applications such as intelligent education and information retrieval, but also an important producer of large model corpus. At a time when open-domain large model training corpus is about to be exhausted, using cutting-edge document parsing technology to accurately extract high-density knowledge from complex layout files, financial documents, handwritten documents and other documents that are not easy to parse can provide key data support for the iteration of high-quality large models.

[0003] Text and formula are the main content components of a document. For example, more than 97% of the layout areas on the authoritative document parsing evaluation set OmniDocBench mentioned in the paper “Ouyang L, Qu Y, Zhou H, et al. Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive annotations [C] / / Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 24838-24848.” are text, formula, and mixed areas of text and formula. When using mainstream tools such as MinerU2.5 for parsing, the parsing time of the above areas consumes more than 80% of the total reasoning time. Therefore, accurate and efficient recognition of text and formula is the key to developing a high-performance document parsing system.

[0004] Traditionally, text recognition and formula recognition are treated as two independent tasks and each has undergone a large amount of research. The main achievements include:

[0005] (1) Specialized text recognition models, usually adopting the connectionist temporal classification (CTC)-based method in the paper Yongkun Du, Zhineng Chen, Caiyan Jia, Xiaoting Yin, Tianlun Zheng, Chenxia Li, Yuning Du, and Yu-Gang Jiang. SVTR: Scene text recognition with a single visual model. In IJCAI, pages 884-890, 2022. or the encoder-decoder architecture in the paper Bautista D, Atienza R. Scene text recognition with permuted autoregressive sequence models [C] / / European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 178-196. CTC-based models emphasize fast inference, while encoder-decoder models usually have higher accuracy but correspondingly increased computational cost.

[0006] (2) Specialized formula recognition models, most methods adopt the encoder-decoder framework as in the paper Deng Y, Kanervisto A, Ling J, et al. Image-to-markup generation with coarse-to-fine attention [C] / / International Conference on Machine Learning. PMLR, 2017: 980-989., improving syntactic parsing and contextual understanding ability through adversarial learning, structure perception models or Transformer architecture.

[0007] Although these special methods have made significant progress in the past, they are mainly limited to processing word or line-level input, and for paragraph or document-level data, they usually combine separate detection models to process after obtaining line-level detection results.

[0008] In recent years, with the development of visual-language models, such as the paper Blecher L, Cucurull G, Scialom T, et al. Nougat: Neural optical understanding for academic documents[J]. arXiv preprint arXiv:2308.13418, 2023. The unified text analysis method gradually emerged. This method usually uses general or customized visual-language large models for document analysis tasks. Through the unified "next token prediction" method, it can identify text, formulas, tables and other multi-modal data in a single framework. This method realizes multi-modal unification at the architecture and data levels, and shows impressive end-to-end analysis performance. However, they usually rely on large-scale models with billions of parameters, resulting in high computational cost and long inference delay, limiting their deployment in many practical applications. At the same time, a large number of practices have shown that due to the heterogeneity of different analysis tasks, the unified "next token prediction" is not the optimal analysis method, resulting in the fact that multi-modal large models are usually not as accurate as specialized recognition models in complex document fine-grained content analysis.

[0009] The current methods based on specialized recognition models and visual-language large models have their own advantages and disadvantages. We notice that text, formulas, tables and other elements have different complexities, but current methods based on visual-language large models all use large models for analysis without considering the use of specialized small models for high-frequency and well-studied text and formulas within the large model framework, and using large models for the rest, thus a more efficient and low-resource consumption solution. At the same time, there are still few technical solutions for unified text and formula recognition under the paradigm of specialized recognition models. Since formulas and text in documents exist in multiple lines of expression, in addition to the hierarchical structure of paragraphs, even under the small model framework, building a unified recognition method for text and formulas also faces the following technical challenges:

[0010] (1) Missing structure change record problem. The current text and formula annotation sequence only records its content information and does not mark its presentation structure in the document image, which cannot reflect the structural diversity of these two document elements at different levels such as characters, words, lines, paragraphs, and multiple paragraphs. This makes it difficult for the model to build appropriate multi-level content representation, especially in the case of limited parameters of specialized small models, the model capacity is not enough to drive effective modeling of inter-level structural dependencies.

[0011] (2) Text and formula semantic entanglement problem. Existing methods usually use a coupled tokenizer to process text and formula modalities simultaneously. The same text token (e.g., sum) represents a different semantic meaning in the text context and the formula context, but the corresponding tokenization results are the same, causing semantic confusion and entanglement. Although large language models can partially eliminate this semantic ambiguity based on context modeling through their large model capacity, special recognition models are sensitive to this and are difficult to effectively distinguish, resulting in a significant decline in recognition performance.

[0012] Therefore, when building a unified recognition method for text and formula, the above two problems need to be explicitly represented so that the model can clearly perceive the multi-level organizational structure of the document text and formula and clearly distinguish the same identifier in the formula and the text, effectively solving the hierarchical structure change perception and modality semantic disentanglement problem, thereby realizing the unified recognition of the two types of content in the same framework. This is also the core content of the present application. SUMMARY

[0013] The present application is made to solve the above problems, and aims to provide a unified recognition method, system, storage medium and device for text images and formula images.

[0014] The present application provides a unified recognition method for text images and formula images, which has the following characteristics, comprising the following steps: S10, obtaining training samples, the training samples comprising training images and corresponding first label sequences, the training images comprising training text images, training formula images and training text formula mixed images, then inserting line end markers and paragraph end markers in the first label sequences to obtain second label sequences Label; S20, using a visual encoder to extract multi-scale features F map of the training images, then flattening the spatial dimensions to one dimension to obtain a visual feature sequence F; S30, using pure text corpus and mathematical formula corpus to independently train a text tokenizer and a formula tokenizer, then integrating the formula tokenizer into the text tokenizer and adding identifiers to the formula tokenizer markers appearing in the vocabulary of the text tokenizer to form a unified vocabulary with clear semantics, and finally obtaining a semantic decoupling tokenizer SDT, wherein the identifier is used to clearly identify the formula tokenizer markers appearing in the vocabulary of the text tokenizer, thereby avoiding semantic confusion caused by the mixed use of formula tokenizer markers in the vocabularies of the text tokenizer and the formula tokenizer; S40, using the semantic decoupling tokenizer SDT to tokenize the second label sequence Label, then mapping to obtain a continuous embedding vector T through a text embedding layer; S50, using a decoder based on a cross-attention mechanism to model the visual feature sequence F and the continuous embedding vector T to obtain a predicted tokenization sequence ​and the cross-entropy loss of Y is supervised to train the autoregressive decoding to obtain a unified recognition model of text images and formula images; S60, inputting the to-be-recognized image into the unified recognition model to output a recognition sequence result and correspondingly converting the line end markers and paragraph end markers in the recognition sequence result into line breaks and paragraph end markers, so as to finally realize unified recognition of text and formula, and the to-be-recognized image includes a to-be-recognized text image, a to-be-recognized formula image and a to-be-recognized text and formula mixed image.

[0015] In the unified recognition method of text images and formula images provided by the application, the line end marker is used to indicate the line break within a paragraph in the first label sequence, and the paragraph end marker is used to indicate the end of the paragraph in the first label sequence.

[0016] In the unified recognition method of text images and formula images provided by the application, the step S20 can further include the following sub-steps: S21, defining the training image as , and preprocessing it using the native resolution strategy, wherein Img represents the defined training image, represents the real set, H represents the image height, W represents the image width, and 3 represents the three color channels of RGB; S22, inputting the preprocessed training image into the visual encoder Encoder to extract a multi-scale feature F map : , wherein D1 represents the feature dimension; S23, flattening the spatial dimension of the multi-scale feature F map to a one-dimensional visual feature sequence F: and , wherein Flatten represents the flattening operation.

[0017] In the unified recognition method of text images and formula images provided by the application, the step S40 can further include the following sub-steps: S41, using a semantic decoupling tokenizer SDT to perform tokenization processing on the second label sequence Label to obtain a discrete tokenization sequence Y: , wherein <bos>indicates the start of a segmented sequence, <eos>an end of a word segmentation sequence is represented, and a semantic decoupling tokenizer SDT is used to add identifiers to the words in the word segmentation sequence Y that are overlapped by the text tokenizer and the formula tokenizer, a length of the preset word segmentation sequence Y is represented, is the same as the length of Y; S42, the word segmentation sequence Y is mapped into a continuous embedding vector T by a text embedding layer: wherein, E text represents an embedding matrix, |V| represents the size of the unified vocabulary, and D2 represents the embedding dimension, represents a real number set.

[0018] In the unified recognition method for text images and formula images provided by the application, the multi-scale feature F map has the same numerical value as the embedding dimension of the continuous embedding vector T.

[0019] In the unified recognition method for text images and formula images provided by the application, the step S50 can further include the following sub-steps: S51, performing autoregressive decoding using a decoder Decoder based on a cross-attention mechanism to obtain a predicted word segmentation sequence with identifiers : wherein, M causal represents using a causal mask, represents the i-th token in Y, , represents the length of the predicted word segmentation sequence with identifiers , is the same as the length of Y; S52, at each time step t, based on the current hidden state h t of the decoder Decoder and the output projection matrix W o , a probability distribution of the next identifier is predicted: wherein, represents the token predicted at time step t, the hidden state h t represents the hidden state of the decoder at time step , and the Softmax function normalizes the mapping result into a probability distribution to represent the possibility of each identifier in the unified vocabulary as the next output; S53, the loss function is set as the cross entropy with Y, the unified recognition model is obtained by minimizing the loss function: wherein, represents the cross entropy loss, y t represents the t-th token in Y, denotes the probability of the unified recognition model predicting the next identifier at each time step t.

[0020] In the unified recognition method for text images and formula images provided by the application, the decoder Decoder can also have the following features: the decoder Decoder is composed of 6 Transformer layers with cross-attention modules, the hidden layer size of each layer is D3, the number of attention heads is D3 / 64, and D3 is the same as the numerical value of the feature dimension of the multi-scale feature F map and the embedding dimension of the continuous embedding vector T.

[0021] The application also provides a unified recognition system for text images and formula images, which uses any of the above unified recognition methods for text images and formula images, and includes: a sample input and processing module for inputting training samples by a user and inserting line end markers and paragraph end markers in the first label sequence to obtain a second label sequence Label; a visual feature extraction module for extracting multi-scale features F map of the training image using a visual encoder, then flattening the spatial dimension to one dimension to obtain a visual feature sequence F; a semantic decoupling tokenizer construction module for independently training a text tokenizer and a formula tokenizer from a pure text corpus and a mathematical formula corpus respectively, then integrating the formula tokenizer into the text tokenizer and adding identifiers to the formula tokenizer markers appearing in the vocabulary of the text tokenizer to form a unified vocabulary with clear semantics, and finally obtaining a semantic decoupling tokenizer SDT; a text embedding module for performing tokenization processing on the second label sequence Label by the semantic decoupling tokenizer SDT, then mapping to obtain a continuous embedding vector T through a text embedding layer; a decoder training module for modeling the visual feature sequence F and the continuous embedding vector T through a decoder based on a cross-attention mechanism to obtain a predicted tokenization sequence , then performing autoregressive decoding training using the cross-entropy loss of X and Y as supervision to obtain a unified recognition model for text images and formula images; and an identification output module for inputting an image to be identified by a user into the unified recognition model to output an identification sequence result and correspondingly convert the line end markers and paragraph end markers in the result into line breaks and paragraph end markers, thereby realizing unified recognition of text and formulas.

[0022] The application also provides a storage medium having a computer program / instruction stored thereon, wherein the computer program / instruction is executed by a processor to implement the steps of the unified recognition method for text images and formula images according to any of the above.

[0023] ​​The application further provides a device for unified recognition of text images and formula images, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the unified recognition method of any one of the preceding embodiments.

[0024] The application has the following beneficial effects:

[0025] (1) The application realizes unified recognition of text images and formula images with a 0.1B parameter scale for the first time. Traditional methods either use independent expert models to process text and formula respectively or rely on large visual-language models with tens of billions of parameters or even larger to recognize various modal document objects. The application breaks through the limitation and realizes unified recognition of text images and formula images with only a 0.1B parameter scale. Through ingenious architecture design and training strategy, the application achieves recognition accuracy comparable to the most advanced 3B parameter scale document parsing model in a small model capacity, opening up a new technical path for lightweight document parsing.

[0026] (2) The application innovatively proposes a semantic decoupling tokenizer mechanism. An independent text and formula tokenization system is designed to fundamentally solve the semantic entanglement problem in traditional unified tokenizers. By assigning independent token ID and embedding vectors to the same string in different modalities, such as "sum" representing an ordinary word in pure text and a summation symbol in formula, the semantic confusion across modalities is eliminated. This mechanism enables small-scale models to accurately distinguish the same symbols in different contexts, and the formula recognition accuracy is improved by 12.1%.

[0027] (3) The application proposes a hierarchical supervision training framework. By explicitly inserting row-level labels (end-of-row labels) and paragraph-level labels (end-of-paragraph labels) into the training target, the hierarchical structure information of the document is directly encoded. Unlike traditional methods that treat content as a flat sequence, the application enables the model to learn content semantics and spatial layout simultaneously, achieving unified supervision at five granularities of characters, words, rows, paragraphs and multi-paragraphs, thereby improving the recognition accuracy at the paragraph level and multi-paragraph level. The application explicitly inserts hierarchical supervision labels that do not exist in the label sequence, such as two Latex label sequences of end-of-row and end-of-paragraph. These labels can guide the model to learn the hierarchical spatial relationship of the document and enhance the perception ability of the layout structure.

[0028] (4) The application constructs a general text and formula unified recognition framework, which can flexibly configure different visual encoder and decoder architectures, and can effectively recognize text and formula in native digital documents, and various types of documents such as scanned pages, document pictures and handwritten content. Experimental results show that the application not only achieves leading accuracy on the OmniDocBench benchmark, but also achieves significantly better accuracy than existing methods on character, word, line and other fine-grained text levels, and handwritten notes, newspapers and other complex documents. When the text and formula unified recognition model of the application is integrated into existing document parsing systems such as MinerU2.5 and PaddleOCR-VL, the overall parsing speed can be improved by 4-7 times while the accuracy is improved, fully verifying its effectiveness in practical applications. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a flow chart of a unified recognition method for text images and formula images of an embodiment of the application.

[0030] Figure 2 is a training image example in an embodiment of the application, wherein part (a) is a training text formula mixed image example, and part (b) is a semantic confusion example of text and formula.

[0031] Figure 3 is Figure 2 is a label sequence example obtained after corresponding insertion line end markers and / or paragraph end markers, wherein part (a) is a Latex label sequence obtained after insertion line end markers and paragraph end markers in the training text formula mixed image as the second label sequence Label, and part (b) is a Latex label sequence obtained after insertion line end markers in the semantic confusion example of text and formula as the second label sequence Label.

[0032] Figure 4 is a comparison diagram of segmentation results of a semantic decoupling segmenter SDT and a segmenter of Dolphin in an embodiment of the application.

[0033] Figure 5 is an architecture diagram of a unified recognition system for text images and formula images of an embodiment of the application.

[0034] Figure 6 is the proportion of different modal data in the OminDocBench evaluation set in a test example of the application.

[0035] Figure 7 is a corresponding Figure 6 inference time consumption and proportion of MinerU2.5.

[0036] Figure 8 is a corresponding Figure 6 and Figure 7 The inference time consumption and proportion after replacing the formula and text recognition module in MinerU2.5 with the unified recognition model of the use embodiment. DETAILED DESCRIPTION

[0037] In order to make the technical means, creative features, purposes and effects of the present application easy to understand, the following embodiments will specifically describe a unified recognition method, system, storage medium and device for text image and formula image in combination with the accompanying drawings.

[0038] EMBODIMENT

[0039] Figure 1 is a flowchart of the unified recognition method for text image and formula image of the embodiment of the present application.

[0040] As Figure 1 shown, the present embodiment provides a unified recognition method for text image and formula image, comprising the following steps S10-S60:

[0041] S10, obtaining training samples and performing label preprocessing to obtain a second label sequence Label, comprising the following sub-steps S11-S12:

[0042] S11, obtaining training samples: the training samples comprise training images and corresponding first label sequences; the training images comprise training text images, training formula images and training text formula mixed images. Wherein, the training text formula mixed image is shown as in (a) part of Figure 2 .

[0043] The document text and formula exhibit a natural hierarchical structure, and are usually organized into lines and paragraphs. Most recognition models treat text and formula content as a flat sequence, ignoring the hierarchical spatial relationship between lines and paragraphs. This simplification limits the model's ability to learn spatial layout representations. In order to explicitly model this hierarchical structure, the present embodiment pre-processes the first label sequence by inserting line-level and paragraph-level supervision markers (subsequent line end markers and paragraph end markers) through the following step S12, so as to explicitly guide the model to learn document structure and improve layout perception ability.

[0044] S12, inserting line end markers and paragraph end markers in the first label sequence to obtain the second label sequence Label:

[0045] (1) The line end marker <|ln|> is inserted to represent the line break within the paragraph in the first label sequence. The line end marker <|ln|> helps the model to understand the inter-line relationship of the document, and plays a key role especially when processing multi-line formulas and cross-line text.

[0046] (2) The paragraph end token <|pn|> is used to indicate the end of a paragraph in the first label sequence. The paragraph end token <|pn|> enables the model to understand the macro structure of the document, which helps to maintain the structural integrity of the output.

[0047] An example of the second label sequence Label is shown in (a) of FIG. 3. Figure 3

[0048] S20, image input and feature extraction, to obtain a visual feature sequence F, including the following sub-steps S21-S23:

[0049] S21, defining the training image as and pre-processing it using the native resolution strategy, keeping the original aspect ratio of the image unchanged, avoiding image distortion and text deformation caused by fixed size scaling, and setting the maximum width limit and maximum height limit of the image.

[0050] wherein Img represents the defined training image, is a real set, H represents the image height, W represents the image width, and 3 represents the three color channels of RGB.

[0051] S22, inputting the pre-processed training image into the visual encoder Encoder to extract multi-scale features F map :

[0052] .

[0053] wherein D1 represents the feature dimension.

[0054] S23, to obtain a sequence representation, facilitating subsequent sequence processing and attention calculation, the spatial dimension of the multi-scale features F map is flattened into a one-dimensional visual feature sequence F:

[0055] and .

[0056] wherein Flatten represents a flattening operation, which combines the spatial dimensions (height and width) of the multi-scale features F map into a single sequence dimension.

[0057] Through the above steps, the conversion of Img into the visual feature sequence F provides rich visual representation for subsequent text and formula recognition. This feature extraction method based on the visual encoder Encoder can effectively process text images in various scenarios and has good generalization ability.

[0058] ​Conventional special recognition model based methods usually only target at a single task in text recognition or formula recognition, and there is no semantic confusion problem. The same text token is not distinguished between formula and text in the existing visual-linguistic model based tokenizer, and the tokenizer is usually trained on a joint corpus mixed with text and formula, which often leads to semantic confusion between text and formula tokens, such as the text token sum representing different semantic concepts in pure text and formula modalities, and Figure 2 (b) in FIG. 6, and Figure 3 (b) in FIG. 7. Although large visual-linguistic models can utilize context clues in training on a large amount of data to eliminate this semantic confusion to some extent, it is much more difficult for special recognition models with limited capacity to eliminate this confusion from a data-driven perspective. To solve this problem, the embodiment constructs a semantic decoupling tokenizer by the following step S30.

[0059] S30, constructing a semantic decoupling tokenizer SDT, including the following steps S31-S32:

[0060] S31, respectively constructing a text tokenizer and a formula tokenizer.

[0061] The text tokenizer is trained using a pure text corpus, focusing on the lexical segmentation of natural language; the formula tokenizer is trained using a mathematical formula corpus, focusing on the segmentation of mathematical formulas. The two tokenizers are independently trained, each learning the optimal lexical representation of the respective modality.

[0062] S32, integrating the formula tokenizer into the text tokenizer to construct a unified semantic decoupling tokenizer SDT. During the integration process, for those formula tokenizer tokens that appear in the vocabulary of the text tokenizer, an identifier is added to form a unified vocabulary with clear semantics, which is then used as its representation in the semantic decoupling tokenizer SDT. Through this measure, it is ensured that the same characters have independent token representations.

[0063] The identifier in this step is used to clarify the formula tokenizer tokens that appear in the vocabulary of the text tokenizer, thereby avoiding semantic confusion caused by the mixed use of formula tokenizer tokens in the vocabularies of the text tokenizer and the formula tokenizer.

[0064] Figure 4 is a comparison diagram of the tokenization results of the semantic decoupling tokenizer SDT of the embodiment of the present application and the tokenizer of Dolphin. Among them, Figure 4 (a) in FIG. 8 is the semantic decoupling tokenizer SDT of the embodiment of the present application, Figure 4 (b) in FIG. 8 is the tokenizer of Dolphin.

[0065] As Figure 4 As shown, it can be seen that sum, infty and the like mixed in the word segmenter of Dolphin are clearly distinguished in the semantic decoupling segmenter SDT of the embodiment of the application.

[0066] S40, after the second label sequence Label is segmented by using the semantic decoupling segmenter SDT, a continuous embedding vector T is obtained by mapping through a text embedding layer, including the following sub-steps S41-S42:

[0067] S41, the second label sequence Label is segmented by using the semantic decoupling segmenter SDT to obtain a discrete segmented sequence Y:

[0068] .

[0069] wherein, <bos>indicates the start of a segmented sequence, <eos>End of token sequence, semantic decoupling tokenizer SDT is used to add identifiers to the tokens in the token sequence Y that are overlapped by the text tokenizer and the formula tokenizer, thereby ensuring semantic uniqueness of each dimension of the tokenizer vocabulary, denotes the length of the preset token sequence Y.

[0070] S42, map the token sequence Y to a continuous embedding vector T by a text embedding layer:

[0071] .

[0072] wherein E text denotes an embedding matrix, , |V| denotes the size of the unified vocabulary, D2 denotes the embedding dimension, and the embedding dimension D2 is consistent with the feature dimension D1 to facilitate subsequent interaction.

[0073] S50, train a unified recognition model for text images and formula images, including the following sub-steps S51-S53:

[0074] S51, use a decoder Decoder based on a cross-attention mechanism to perform autoregressive decoding to obtain a predicted token sequence with identifiers :

[0075] .

[0076] wherein the decoder Decoder is composed of 6 Transformer layers with a cross-attention module in series, the size of the hidden layer of each layer is D3, the number of attention heads is D3 / 64, D3 is consistent with D2 and D1, and M causal denotes the use of a causal mask, denotes the i-th token in , , denotes the length of the predicted token sequence with identifiers .

[0077] S52, at each time step t, based on the current hidden state h t of the decoder Decoder and the output projection matrix W o predict the probability distribution of the next identifier:

[0078] .

[0079] wherein denotes the token predicted at time step t, the hidden state h t denotes the hidden state of the decoder at time step The Softmax function normalizes the mapping result into a probability distribution to represent the likelihood of each identifier in the unified vocabulary as the next output.

[0080] S53, set the loss function as the cross-entropy between the token sequence Y and the second label sequence Label, train the unified recognition model by minimizing the loss function, and obtain the unified recognition model.

[0081] .

[0082] wherein, represents the cross-entropy loss, y t represents the t-th token in the token sequence Y, represents the probability of the unified recognition model predicting the next identifier at each time step t. The loss function measures the difference between the probability distribution predicted by the model and the true label sequence. By minimizing this cross-entropy loss, the model learns the ability to accurately predict the next identifier given the visual features and the history of generated identifiers.

[0083] S60, input the image to be recognized into the unified recognition model to output the recognition sequence result and correspondingly convert the line end marker and paragraph end marker in the recognition sequence result into a line break and a paragraph end symbol, thereby realizing unified recognition of text and formula, and the image to be recognized includes a text image to be recognized, a formula image to be recognized, and a mixed image of text and formula to be recognized.

[0084] Specifically, in the present embodiment, for the output (recognition sequence result) in Latex format, the line end marker is deleted (i.e., converted into a line break), and the paragraph end marker is converted into 2 line breaks (i.e., a paragraph end symbol).

[0085] Figure 5 is the architecture diagram of the unified recognition system of the text image and the formula image according to the embodiment of the present application.

[0086] As Figure 5 shown, the present embodiment also provides a unified recognition system 100 of a text image and a formula image, which uses the unified recognition method of the text image and the formula image according to the embodiment of the present application, and includes a sample input and processing module 10, a visual feature extraction module 20, a semantic decoupling tokenizer construction module 30, a text embedding module 40, a decoder training module 50, and a recognition output module 60.

[0087] The sample input and processing module 10 is configured to input the training sample according to the method of step S10 and insert the line end marker and the paragraph end marker in the first label sequence to obtain the second label sequence Label.

[0088] ​The visual feature extraction module 20 is used to extract multi-scale features F of the training image using a visual encoder according to the method in step S20. map Then, its spatial dimension is flattened to one dimension to obtain the visual feature sequence F.

[0089] The semantic decoupling word segmenter construction module 30 is used to independently train a text word segmenter and a formula word segmenter using plain text corpus and mathematical formula corpus respectively according to the method of step S30. Then, the formula word segmenter is integrated into the text word segmenter, and identifiers are added to the formula word segmentation tags that appear in the vocabulary of the text word segmenter to form a semantically clear unified vocabulary, and finally the semantic decoupling word segmenter SDT is obtained.

[0090] The text embedding module 40 is used to process the second tag sequence using the semantic decoupling tokenizer SDT according to the method in step S40. After word segmentation, a continuous embedding vector T is obtained by mapping through the text embedding layer.

[0091] The decoder training module 50 is used to model the visual feature sequence F and the continuous embedding vector T using a decoder based on a cross-attention mechanism, according to the method in step S50, to obtain the predicted word segmentation sequence. After that, with The cross-entropy loss of Y is used as supervision for autoregressive decoding training to obtain a unified recognition model for text images and formula images.

[0092] The recognition output module 60 is used to input the image to be recognized into the unified recognition model after the user inputs the image according to the method of step S60, thereby outputting the recognition sequence result and converting the line end mark and paragraph end mark in it into a newline character and a paragraph end mark, so as to achieve unified recognition of text and formula.

[0093] This embodiment also provides a storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the unified recognition method for text images and formula images in this embodiment.

[0094] This embodiment also provides a unified recognition device for text images and formula images, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the unified recognition method for text images and formula images in this embodiment.

[0095] Test case

[0096] This test case uses the unified recognition system 100 for text images and formula images in the embodiment to conduct actual testing according to the unified recognition method for text images and formula images in the embodiment.

[0097] First, the unified recognition model of text images and formula images is trained, and the specific parameter selection / setting is as follows:

[0098] In step S11,

[0099] The training sample adopts a large-scale data set, which contains nearly 40 million multi-level training text images, training formula images and training text formula mixed images. The data set covers 30 million English and 10 million Chinese samples, including 19 million training text images, 13 million training formula images and 8 million training text formula mixed images, and the corresponding first label sequence.

[0100] The above training sample sources include three main parts:

[0101] (1) Online TeX source: Collect papers with Latex source code on the scientific paper website ArXiv, use the corresponding relationship between Latex document source code, text and formula to extract training text images, training formula images and training text formula mixed images, and the Latex label sequence corresponding to these image regions as the first label sequence.

[0102] (2) Digital PDF document: Collect publicly available PDF financial research reports, news, etc. on the Internet, and use PyMuPDF tools for analysis, also get training text images, training formula images and training text formula mixed images and their first label sequence.

[0103] (3) Public data set integration: Collect existing public scene / handwritten character recognition data sets, formula recognition data sets, etc.

[0104] In step S12,

[0105] (1) The insertion of line end marker <|ln|> follows the following rules: insert when detecting physical line breaks but not ending the current semantic unit while maintaining semantic continuity; insert at natural breakpoints for long formulas or continuous text that span lines; insert at the boundaries of each element for structured elements such as list items, table cells, etc.

[0106] (2) The insertion of paragraph end marker <|pn|> follows the following rules: insert when detecting obvious paragraph separation (such as blank lines, indentation changes, title transitions); insert when the semantic unit is completely ended and the subsequent content belongs to a new topic; insert at the boundaries of document structure elements.

[0107] In step S21,

[0108] The maximum image width W is limited to 960 pixels, and the maximum image height H is limited to 1408 pixels.

[0109] In step S22, step S42, and step S51:

[0110] D1=D2=D3=768.

[0111] In step S22:

[0112] The visual encoder Encoder selects the image encoder based on FocalNet mentioned in the paper Yang J, Li C, Dai X, et al. Focal modulation networks[J]. Advances in Neural Information Processing Systems, 2022, 35: 4203-4217. FocalNet as Encoder is a visual model based on focal modulation mechanism, which can effectively capture multi-scale features of images.

[0113] In step S41:

[0114] The length of the preset word segmentation sequence Y is 1024, which is sufficient to cover most document scenarios.

[0115] In step S42:

[0116] The size of the unified vocabulary |V| is 56371, including 52661 text tokens (vocabulary tokens of the text segmenter trained in step S31), 3705 formula special tokens (vocabulary tokens of the formula segmenter trained in step S31), and 5 system tokens (,,, and <bos> 、 <eos>, <|ln|> and <|pn|> etc.

[0117] In this test example, the proportion balancing sampling strategy is used during training to ensure the balanced learning of text, formula and mixed samples during the training process. The public OmniDocBench is used for evaluation. The training is optimized using the AdamW optimizer with a weight decay coefficient of 0.01. The global batch size is set to 64, and each sample is processed on 8 GPUs. The training round is set to 10 rounds, and the learning rate is set to 1x10 -4 The One-cycle learning rate scheduler is used, and the linear warmup strategy is used in the first 0.5 rounds to ensure the stability at the beginning of training. The data enhancement strategy includes various image transformation techniques: random rotation (-10° to +10°), perspective distortion to simulate different shooting angles, motion blur processing to improve the robustness to blurred images, and Gaussian noise addition to enhance the adaptability to low-quality images. These enhancement strategies are randomly applied to improve the generalization performance of the model.

[0118] Subsequently, the unified recognition model of text images and formula images is deployed and used, as follows:

[0119] (1) Installation: Deploy using Ubuntu 20.04 operating system with CUDA 11.8 environment. Clone the project code repository using Git, create an independent conda virtual environment openocr, and set the Python version to 3.8. Install PyTorch 2.2.0 and the corresponding torchvision 0.17.0, torchaudio 2.2.0 to ensure compatibility with CUDA 11.8 version. Install the dependency packages specified in requirements.txt through pip.

[0120] (2) Download the corresponding dataset. Provide pre-trained model weight files for direct loading and use.

[0121] (3) Through the infer.py script, specify the configuration file and image path to perform inference.

[0122] Figure 6 is the proportion of different modal data in the OminDocBench evaluation set in the test example of the present application; Figure 7 is the inference time consumption and proportion of using MinerU2.5 corresponding to Figure 6 ; Figure 8 is the inference time consumption and proportion of using the unified recognition model of the embodiments corresponding to Figure 6 and Figure 7 .

[0123] As shown in Figure 6~Figure 8 When only MinerU2.5 is used, it takes 41893 seconds to complete the analysis of the entire OminDocBench evaluation set, and the identification of text, formula, and mixed text-formula regions accounts for 87.9% of the total time, i.e., 36824 seconds. When the unified recognition model of the embodiment is used to replace the formula and text recognition modules in MinerU2.5, the time taken to complete the analysis of the entire OminDocBench evaluation set is greatly reduced to 10594 seconds, which is only one fourth of the original time. This result fully demonstrates the efficiency advantage of the unified recognition model of the embodiment.

[0124] The unified recognition model of the embodiment has the advantages of high analysis accuracy and fast analysis speed, which are specifically manifested as follows:

[0125] (1) In terms of recognition accuracy, in the benchmark test, the unified recognition method of text images and formula images of the embodiment achieves an accuracy of 96.2% in text recognition and 86.6% in formula recognition, which is superior to or comparable to models with 30 times more parameters. In the OmniDocBench full-page analysis evaluation, when combined with MinerU2.5 and PaddleOCR-VL (replace the text and formula recognition modules in these two document analysis models with the unified recognition model of the embodiment), the overall edit distance is reduced by 2.3% and 0.2%, respectively, reaching the current best level.

[0126] (2) In terms of efficiency, the inference speed is improved by 2-9 times compared to models with the same accuracy. The character-level recognition only takes 0.03 seconds per character, the paragraph-level recognition takes 0.54 seconds per paragraph, and the full-page analysis takes an average of 6.2 seconds per page. After replacing the text and formula recognition modules of MinerU2.5 with the unified recognition model of the embodiment, the single-page analysis time is reduced from 42.72 seconds to 6.2 seconds, achieving a dual improvement in accuracy and efficiency. In addition, through semantic decoupling and hierarchical supervision, the model performs well in 9 document subfields (including books, PPTs, research reports, test papers, handwritten notes, etc.). In particular, in complex subfields such as test papers, the performance improvement reaches 2.8%-2.9%. Therefore, the unified recognition method of text images and formula images of the embodiment has broad application prospects in document digitization, academic literature processing, education resource electrification, enterprise document management, etc., and is particularly suitable for scenarios that require efficient and accurate recognition of mixed text-formula content.

[0127] Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles of the present application, and various changes and improvements can be made without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.< / eos> < / bos> ​< / eos> ​< / bos> < / eos> ​< / bos>

Claims

1. A unified recognition method for text images and formula images, characterized in that, Includes the following steps: S10, Obtain training samples, the training samples include training images and their corresponding first label sequences, the training images include training text images, training formula images and training text formula mixed images, and then insert line end marks and paragraph end marks into the first label sequence to obtain the second label sequence Label; S20, use a visual encoder to extract multi-scale features F from the training images. map Then, its spatial dimension is flattened to one dimension to obtain the visual feature sequence F; S30: After independently training a text segmenter and a formula segmenter using plain text corpus and mathematical formula corpus respectively, the formula segmenter is integrated into the text segmenter, and identifiers are added to the formula segmentation tags appearing in the vocabulary of the text segmenter to form a semantically clear unified vocabulary, ultimately obtaining the semantically decoupled segmenter SDT. The identifier is used to identify the formula segmentation markers appearing in the vocabulary of the text segmenter, thereby avoiding semantic confusion caused by the mixing of formula segmentation markers in the respective vocabularies of the text segmenter and the formula segmenter; S40, the semantic decoupling word segmenter SDT is used to segment the second label sequence Label to obtain the segmented word sequence Y, and then the text embedding layer is used to map it to obtain a continuous embedding vector T. S50, a decoder based on a cross-attention mechanism is used to model the visual feature sequence F and the continuous embedding vector T to obtain the predicted word segmentation sequence. After that, with The cross-entropy loss of Y is used as supervision for autoregressive decoding training to obtain a unified recognition model for text images and formula images; S60, the image to be recognized is input into the unified recognition model to output the recognition sequence result and convert the line end mark and the paragraph end mark in it into a newline character and a paragraph end mark, so as to achieve unified recognition of text and formula. The image to be recognized includes the text image to be recognized, the formula image to be recognized, and the mixed image of text and formula to be recognized.

2. The unified recognition method for text images and formula images according to claim 1, characterized in that: In step S10, the line end marker is used to indicate a newline within a paragraph in the first tag sequence. The paragraph end marker is used to indicate the end of a paragraph in the first tag sequence.

3. The unified recognition method for text images and formula images according to claim 1, characterized in that: Step S20 includes the following sub-steps: S21, define the training image as And it is preprocessed using the native resolution strategy. Wherein, Img represents the defined training image. H represents the set of real numbers, W represents the image height, and 3 represents the three color channels of RGB. S22, the preprocessed training image is input into the visual encoder to extract multi-scale features F. map : ， Where D1 represents the feature dimension; S23, the multi-scale feature F map The spatial dimension is flattened into a one-dimensional visual feature sequence F: and , Flatten refers to the flattening operation.

4. The unified recognition method for text images and formula images according to claim 1, characterized in that: Step S40 includes the following sub-steps: S41, the semantic decoupling segmenter SDT is used to segment the second label sequence Label to obtain a discrete segmented sequence Y: ， in, <bos>This indicates the start of the word segmentation sequence. , <eos>This indicates the end of the word segmentation sequence. The semantic decoupling segmenter (SDT) is used to add the identifier to words in the word segmentation sequence Y where the text segmenter and the formula segmenter overlap. This indicates the length of the preset word segmentation sequence Y. Same length as Y;< / eos> < / bos> S42, the word segmentation sequence Y is mapped to a continuous embedding vector T through the text embedding layer: ， Among them, E text Represents the embedding matrix. |V| represents the size of the unified vocabulary, and D2 represents the embedding dimension. It represents the set of real numbers.

5. The unified recognition method for text images and formula images according to claim 1, 3, or 4, characterized in that: in, The multi-scale feature F map The feature dimension and the embedding dimension of the continuous embedding vector T have the same numerical value.

6. The unified recognition method for text images and formula images according to claim 1, characterized in that: Step S50 includes the following sub-steps: S51, Autoregressive decoding is performed using a decoder based on a cross-attention mechanism to obtain the predicted word segmentation sequence with the identifier. : ， Among them, M causal This indicates the use of a causal mask. express The i-th word element in , This represents the predicted word segmentation sequence with the identifier. Length, Same length as Y; S52, at each time step t, based on the current hidden state h of the decoder... t and output projection matrix W o Predict the probability distribution of the next identifier: ， in, The hidden state h represents the word predicted at time step t. t This indicates that the decoder is at time step The hidden state of the Softmax function normalizes the mapping result into a probability distribution to represent the probability of each of the identifiers in the unified vocabulary being the next output; S53, set the loss function to The cross-entropy with Y is used to train a unified recognition model by minimizing the loss function. ， in, Represents the cross-entropy loss, y t This represents the t-th word in Y. This represents the probability that the unified identification model predicts the next identifier at each time step t.

7. The unified recognition method for text images and formula images according to claim 6, characterized in that: in, The decoder consists of six Transformer layers with cross-attention modules. Each layer has a hidden layer size of D3 and a number of attention heads of D3 / 64. D3 and the multi-scale feature F map The feature dimension and the embedding dimension of the continuous embedding vector T have the same numerical value.

8. A unified recognition system for text images and formula images, characterized in that, The unified recognition method for text images and formula images according to any one of claims 1 to 7 includes: The sample input and processing module is used to allow users to input the training samples and insert line end marks and paragraph end marks into the first label sequence to obtain the second label sequence Label; The visual feature extraction module is used to extract multi-scale features F from the training images using a visual encoder. map Then, its spatial dimensions are flattened to one dimension to obtain the visual feature sequence F; The semantic decoupling word segmenter construction module is used to independently train a text word segmenter and a formula word segmenter through plain text corpus and mathematical formula corpus, respectively. Then, the formula word segmenter is integrated into the text word segmenter, and identifiers are added to the formula word segmentation tags that appear in the vocabulary of the text word segmenter to form a semantically clear unified vocabulary, and finally the semantic decoupling word segmenter SDT is obtained. The text embedding module is used to process the second tag sequence through the semantic decoupling segmenter SDT. After word segmentation, a continuous embedding vector T is obtained by mapping through a text embedding layer; The decoder training module is used to model the visual feature sequence F and the continuous embedding vector T using a decoder based on a cross-attention mechanism to obtain the predicted word segmentation sequence. After that, with The cross-entropy loss of Y is used as supervision for autoregressive decoding training to obtain a unified recognition model for text images and formula images; and The recognition output module is used to input the image to be recognized into the unified recognition model after the user inputs it, thereby outputting the recognition sequence result and converting the line end mark and paragraph end mark into line break and paragraph end mark respectively, so as to achieve unified recognition of text and formula.

9. A storage medium having a computer program / instruction stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the unified recognition method for text images and formula images as described in any one of claims 1 to 7.

10. A unified recognition device for text images and formula images, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the unified recognition method for text images and formula images as described in any one of claims 1 to 7.

Images