An OCR error detection method based on multi-modal information fusion
Patent Information
- Application Number
- CN202410139257.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-01
- Publication Date
- 2026-11-03
- Estimated Expiration
- 2044-02-01
AI Technical Summary
Existing OCR error detection methods rely solely on single-modal information from OCR text, which fails to effectively handle the complexity and variability of table images, resulting in insufficient detection accuracy.
A multimodal information fusion method is adopted, which combines image and text information, extracts features through residual network and glyph convolutional network, and uses multilayer perceptron for feature fusion to construct an error detection model based on deep neural network.
It improves the accuracy and robustness of OCR error detection, better handles the complexity and variability of table images, reduces manual proofreading workload, and improves overall efficiency.
Smart Images
Figure CN117953524B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of OCR error detection technology, and in particular to an OCR error detection method based on multimodal information fusion. Background Technology
[0002] In today's digital age, the need for accurately extracting data from tabular images is steadily increasing. Optical Character Recognition (OCR) systems play a crucial role in converting such images into machine-readable formats. Cell images in tables often contain Chinese characters, numbers, and symbols, making accurate recognition a complex task. Cell-based OCR error detection is essential for ensuring the quality of OCR output for tables. The importance of OCR error detection lies in its ability to identify OCR output areas requiring further inspection or correction. By marking potentially erroneous segments, human reviewers or subsequent error-correction algorithms can focus their attention on specific areas, significantly reducing the workload of manual proofreading and improving overall efficiency. Furthermore, error detection plays a vital role in maintaining the reliability and credibility of OCR-generated text, especially in applications where accurate information extraction is critical.
[0003] Currently, OCR error detection methods can be broadly categorized into two types: one is based on pattern matching and language analysis, using predefined patterns and dictionaries to compare the recognized text with expected patterns or language rules to detect inconsistencies or deviations; the other relies on statistical and machine learning techniques to detect errors in OCR output. It utilizes statistical models such as n-gram language models or Hidden Markov Models to capture the statistical properties of correct text, or uses machine learning algorithms such as Support Vector Machines (SVM), Random Forests, and deep learning models to learn feature representations of correct text for error detection, leveraging data-driven methods to learn patterns and features that distinguish correct from incorrect text fragments. Both of these error detection methods lack modeling of the original image, relying solely on the text output of OCR. A single modality cannot contain all the effective information needed to generate accurate predictions. Furthermore, both methods struggle to handle the complexity and variability of cell-based images, failing to effectively address the unique challenges posed by table images, including content diversity, the importance of layout and structure, and the presence of noise and human intervention. Summary of the Invention
[0004] To address the problem of existing technologies that rely solely on single-modal information from OCR text for error detection, this invention aims to provide a multimodal information fusion-based OCR error detection method that combines information from text and images to supplement information, broaden the coverage of information contained in the input data, improve the robustness of the prediction model, and effectively enhance the accuracy of OCR error detection for cell images.
[0005] To achieve the above objectives, the present invention adopts the following technical solution: an OCR error detection method based on multimodal information fusion, the method comprising the following sequential steps:
[0006] (1) Preprocess the table image, the preprocessing including binarization, noise reduction and cropping operations, to obtain the cell image and the corresponding ORC text;
[0007] (2) Label the image and corresponding OCR text of each cell, and the labeled data form an error detection training dataset with detection labels;
[0008] (3) Construct a cell OCR error detection model based on multimodality, wherein the cell OCR error detection model based on multimodality includes an image processing module, a text processing module and a multimodal information fusion and classification module;
[0009] (4) The multimodal cell OCR error detection model is iteratively trained using the error detection training dataset to obtain the trained multimodal cell OCR error detection model.
[0010] (5) Input the cell image to be detected and the corresponding OCR text into the trained multimodal cell OCR error detection model and output the detection results.
[0011] In step (1), binarization is to convert the pixel values in the image to 0 or 1, denoising refers to eliminating noise and burrs in the image, and cutting refers to dividing the image into multiple cells according to the row and column structure of the table, with each cell containing a cell image and an OCR text.
[0012] In step (2), the detection label refers to providing a correct label for the image and text of each cell, indicating whether there is an OCR error. The OCR error refers to the inconsistency between the text in the image and the text in the text.
[0013] In step (3), the input of the image processing module is the cell image and the output is the image features. The image processing module first initializes the features of the cell image through the embedding layer, and then extracts the features of the image through the residual network. The residual network extracts local features of the image by stacking multiple residual blocks, and performs feature dimensionality reduction with pooling operation. Finally, the features output by the residual network contain high-level abstract features in the cell image.
[0014] The text processing module takes OCR text as input and outputs OCR text features as output. The text processing module first preprocesses the OCR text through a glyph processing layer to obtain all the original characters that make up the OCR text and the character image corresponding to each character. Then, the original characters and character images are fed into a glyph convolutional network for further learning.
[0015] In the glyph convolutional network, the original characters are fed into the original character embedding layer to obtain the initial character features. The multi-font fusion embedding layer obtains the character information of different fonts through font mapping of the initial character features, and then fuses all the character information to obtain the high-dimensional features of the original characters. The character image is fed into the character image embedding layer to obtain the initial glyph feature vector, and then the initial glyph feature vector is fed into the glyph convolutional block for character glyph feature learning. In the glyph convolutional block, the character image is first processed by a two-dimensional convolutional layer and a max pooling layer for feature extraction and dimensionality reduction, and then passed through a downsampling convolutional layer and a grouped convolutional layer to obtain the character glyph vector.
[0016] The fusion coding layer of the glyph convolutional network aggregates the original character features and character glyph vectors learned from the above two parts through fusion coding to obtain the abstract feature representation of the OCR text.
[0017] The multimodal information fusion classification module consists of a splicing layer and two fully connected layers. The two fully connected layers form a multilayer perceptron, and the multilayer perceptron and the splicing layer form a feedforward neural network. The multimodal information fusion classification module integrates and filters the information learned by the image processing module and the text processing module. Through the feedforward neural network based on the multilayer perceptron, the multimodal space is mapped to a shared semantic subspace, and the high-order interaction relationship between the features of the two modalities is learned, thereby obtaining the error detection result.
[0018] Step (4) specifically includes the following sequential steps:
[0019] (4a) Based on the cell images in the error detection training dataset, the visual features of the images are obtained through the residual network of the image processing module;
[0020] (4b) Based on the OCR text corresponding to the cell image, the feature representation of the text is obtained through the glyph convolutional network of the text processing module;
[0021] (4c) Input the visual features of the image and the feature representation of the text into the feedforward neural network in the multi-module information fusion classification module for further learning, and finally obtain the error detection result.
[0022] In step (4), the training parameter Epoch is set to 300, the batch size is set to 128, the optimization algorithm is Adam, the initial learning rate is set to 0.001, and the error detection training dataset is divided into training and test sets in a 9:1 ratio. The loss function of the multimodal cell OCR error detection model is:
[0023] L=(1-λ(t))L target +λ(t)L cls
[0024]
[0025] Where λ(·) is a function of the number of training iterations t, λ0∈[0,1] represents the initial value, λ1∈[0,1] represents the decay value, and λ0 and λ1 are both hyperparameters. target It is the cross-entropy loss function, L cls It is the logarithmic loss function;
[0026] The formula for the cross-entropy loss function is:
[0027]
[0028] in, It is the entire training set, containing N training data points, x = [x img ,x text ]belong Let x represent each training data point. img ,x text These represent cell images and OCR text, respectively. The OCR text is x. text =[b1,b2,…,b m ], b i Let represent the i-th character in the OCR text, i∈[1,m], where m represents the number of characters in the OCR text; y∈{0,1}, indicating whether the cell image matches the OCR text; p(x) represents the probability predicted by the model that the cell image matches the OCR text.
[0029] The formula for the logarithmic loss function is:
[0030]
[0031] Among them, c i A glyph vector representing a character.
[0032] Step (4b) specifically refers to:
[0033]
[0034] c i =g(b i )
[0035] Where E represents the feature representation of the output text, m represents the number of characters in the OCR text, σ represents the ReLU activation function, and p i f represents the position vector of characters in the text. c (·) represents the convolution operation in a glyph convolutional network; c i The glyph vector representing a character, c i It is obtained from the character image vector generation function g(·); W is the trainable parameter matrix.
[0036] As can be seen from the above technical solution, the beneficial effects of the present invention are as follows: First, the present invention converts each character in the OCR text into an image, uses computer vision technology to capture fine visual details, thereby improving the ability to recognize correct and incorrect characters, numbers and symbols, and eliminating the problem of inconsistency in the feature dimensional space between OCR text and cell image; Second, the present invention considers both OCR text features and original cell image features, proposes and designs a text processing module based on glyph convolutional networks and an image processing module based on residual networks, and performs dimensional fusion of these two features. The multimodal fusion process combines information from text and images, realizes information supplementation, broadens the coverage of information contained in the input data, and improves the robustness and accuracy of the multimodal cell OCR error detection model; Third, the present invention uses a deep neural network-based method to build the model, which can automatically learn effective text features and image features with the help of deep neural networks, and model the complex interaction relationship between multiple features. It does not rely on costly feature engineering, and has the advantages of saving human resources, reducing costs, and high versatility. Attached Figure Description
[0037] Figure 1 This is a flowchart of the method of the present invention;
[0038] Figure 2 This is a diagram illustrating the overall architecture of the multimodal cell OCR error detection model in this invention.
[0039] Figure 3 for Figure 2 Structure diagram of the intermediate residual network;
[0040] Figure 4 for Figure 2 Structure diagram of the Chinese character-shaped convolutional network;
[0041] Figure 5 for Figure 4 A structural diagram of a Chinese character-shaped convolutional block. Detailed Implementation
[0042] like Figure 1 As shown, an OCR error detection method based on multimodal information fusion includes the following sequential steps:
[0043] (1) Preprocess the table image, the preprocessing including binarization, noise reduction and cropping operations, to obtain the cell image and the corresponding ORC text;
[0044] (2) Label the image and corresponding OCR text of each cell, and the labeled data form an error detection training dataset with detection labels;
[0045] (3) Construct a cell OCR error detection model based on multimodality, wherein the cell OCR error detection model based on multimodality includes an image processing module, a text processing module and a multimodal information fusion and classification module;
[0046] (4) The multimodal cell OCR error detection model is iteratively trained using the error detection training dataset to obtain the trained multimodal cell OCR error detection model.
[0047] (5) Input the cell image to be detected and the corresponding OCR text into the trained multimodal cell OCR error detection model and output the detection results.
[0048] In step (1), binarization converts pixel values in the image to 0 or 1 to reduce the influence of noise and background; denoising refers to eliminating noise and jagged edges in the image to improve image clarity; and segmentation refers to dividing the image into multiple cells according to the row and column structure of the table, with each cell containing a cell image and an OCR text. Preprocessing the table image involves converting it into a format suitable for subsequent processing, that is, separating the image and text content of each cell to facilitate feature extraction and error detection.
[0049] In step (2), the detection label refers to providing a correct label for the image and text of each cell, indicating whether there is an OCR error. The OCR error refers to the inconsistency between the text in the image and the text in the text. For example, the text in the cell image is "123", while the text in the text is "124".
[0050] like Figure 2As shown, in step (3), the input of the image processing module is a cell image, and the output is the image features. The image processing module first initializes the features of the cell image through an embedding layer, and then extracts the features of the image through a residual network, such as... Figure 3 As shown, the residual network extracts local image features by stacking multiple residual blocks, and performs feature dimensionality reduction by pooling operations. Finally, the features output by the residual network contain high-level abstract features in the cell image.
[0051] The text processing module takes OCR text as input and outputs OCR text features as output. The text processing module first preprocesses the OCR text through a glyph processing layer to obtain all the original characters that make up the OCR text and the character image corresponding to each character. Then, the original characters and character images are fed into a glyph convolutional network for further learning.
[0052] like Figure 4 As shown, in the glyph convolutional network, the original character is fed into the original character embedding layer to obtain the initial character features. The multi-font fusion embedding layer obtains the character information of different fonts through font mapping of the initial character features, and then fuses all the character information to obtain the high-dimensional features of the original character. The character image is fed into the character image embedding layer to obtain the initial glyph feature vector, and then the initial glyph feature vector is fed into the glyph convolutional block to learn the character glyph features.
[0053] like Figure 5 As shown, in the character convolution block, the character image is first processed by a two-dimensional convolutional layer and a max pooling layer to extract features and reduce dimensionality, and then passed through a downsampling convolutional layer and a grouped convolutional layer to obtain the character character vector.
[0054] The fusion coding layer of the glyph convolutional network aggregates the original character features and character glyph vectors learned from the above two parts through fusion coding to obtain the abstract feature representation of the OCR text.
[0055] The multimodal information fusion classification module consists of a splicing layer and two fully connected layers. The two fully connected layers form a multilayer perceptron, and the multilayer perceptron and the splicing layer form a feedforward neural network. The multimodal information fusion classification module integrates and filters the information learned by the image processing module and the text processing module. Through the feedforward neural network based on the multilayer perceptron, the multimodal space is mapped to a shared semantic subspace, and the high-order interaction relationship between the features of the two modalities is learned, thereby obtaining the error detection result.
[0056] Step (4) specifically includes the following sequential steps:
[0057] (4a) Based on the cell images in the error detection training dataset, the visual features of the images are obtained through the residual network of the image processing module;
[0058] (4b) Based on the OCR text corresponding to the cell image, the feature representation of the text is obtained through the glyph convolutional network of the text processing module;
[0059] (4c) Input the visual features of the image and the feature representation of the text into the feedforward neural network in the multi-module information fusion classification module for further learning, and finally obtain the error detection result.
[0060] In step (4), the training parameter Epoch is set to 300, the batch size is set to 128, the optimization algorithm is Adam, the initial learning rate is set to 0.001, and the error detection training dataset is divided into training and test sets in a 9:1 ratio. The loss function of the multimodal cell OCR error detection model is:
[0061] L=(1-λ(t))L target +λ(t)L cls
[0062]
[0063] Where λ(·) is a function of the number of training iterations t, λ0∈[0,1] represents the initial value, λ1∈[0,1] represents the decay value, and λ0 and λ1 are both hyperparameters. target It is the cross-entropy loss function, L cls It is the logarithmic loss function;
[0064] The formula for the cross-entropy loss function is:
[0065]
[0066] in, It is the entire training set, containing N training data points, x = [x img ,x text ]belong Let x represent each training data point. img ,x text These represent cell images and OCR text, respectively. The OCR text is x. text =[b1,b2,…,b m ], b i Let represent the i-th character in the OCR text, i∈[1,m], where m represents the number of characters in the OCR text; y∈{0,1}, indicating whether the cell image matches the OCR text; p(x) represents the probability predicted by the model that the cell image matches the OCR text.
[0067] The formula for the logarithmic loss function is:
[0068]
[0069] Among them, c i A glyph vector representing a character.
[0070] Step (4b) specifically refers to:
[0071]
[0072] c i =g(b i )
[0073] Where E represents the feature representation of the output text, m represents the number of characters in the OCR text, σ represents the ReLU activation function, and p i f represents the position vector of characters in the text. c (·) represents the convolution operation in a glyph convolutional network; c i The glyph vector representing a character, c i It is obtained from the character image vector generation function g(·); W is the trainable parameter matrix.
[0074] In summary, this invention converts each character in OCR text into an image, utilizing computer vision technology to capture fine visual details, thereby improving the ability to distinguish between correct and incorrect characters, numbers, and symbols, and eliminating the inconsistency in the dimensionality of features between OCR text and cell images. This invention simultaneously considers OCR text features and original cell image features, proposing and designing a text processing module based on glyph convolutional networks and an image processing module based on residual networks, and dimensionally fusing these two feature sets. This multimodal fusion process combines information from both text and images, achieving information supplementation, broadening the coverage of information contained in the input data, and improving the robustness and accuracy of the multimodal cell OCR error detection model. This invention employs a deep neural network-based model construction method, which can automatically learn effective text and image features with the help of deep neural networks and model the complex interaction relationships between multiple features. It does not rely on costly feature engineering, and has the advantages of saving human resources, reducing costs, and high versatility.
Claims
1. An OCR error detection method based on multimodal information fusion, characterized in that: The method includes the following steps in sequence: (1) Preprocess the table image, the preprocessing including binarization, noise reduction and cropping operations, to obtain the cell image and the corresponding ORC text; (2) Label the image and corresponding OCR text of each cell, and the labeled data form an error detection training dataset with detection labels; (3) Construct a cell OCR error detection model based on multimodality, wherein the cell OCR error detection model based on multimodality includes an image processing module, a text processing module and a multimodal information fusion and classification module; (4) The multimodal cell OCR error detection model is iteratively trained using the error detection training dataset to obtain the trained multimodal cell OCR error detection model; (5) Input the cell image to be detected and the corresponding OCR text into the trained multimodal cell OCR error detection model and output the detection results; In step (3), the input of the image processing module is the cell image and the output is the image features. The image processing module first initializes the features of the cell image through the embedding layer, and then extracts the features of the image through the residual network. The residual network extracts local features of the image by stacking multiple residual blocks, and performs feature dimensionality reduction with pooling operation. Finally, the features output by the residual network contain high-level abstract features in the cell image. The text processing module takes OCR text as input and outputs OCR text features as output. The text processing module first preprocesses the OCR text through a glyph processing layer to obtain all the original characters that make up the OCR text and the character image corresponding to each character. Then, the original characters and character images are fed into a glyph convolutional network for further learning. In the glyph convolutional network, the original characters are fed into the original character embedding layer to obtain the initial character features. The multi-font fusion embedding layer obtains the character information of different fonts through font mapping of the initial character features, and then fuses all the character information to obtain the high-dimensional features of the original characters. The character image is fed into the character image embedding layer to obtain the initial glyph feature vector, and then the initial glyph feature vector is fed into the glyph convolutional block for character glyph feature learning. In the glyph convolutional block, the character image is first processed by a two-dimensional convolutional layer and a max pooling layer for feature extraction and dimensionality reduction, and then passed through a downsampling convolutional layer and a grouped convolutional layer to obtain the character glyph vector. The fusion coding layer of the glyph convolutional network aggregates the original character features and character glyph vectors learned from the above two parts through fusion coding to obtain the abstract feature representation of the OCR text. The multimodal information fusion classification module consists of a splicing layer and two fully connected layers. The two fully connected layers form a multilayer perceptron, and the multilayer perceptron and the splicing layer form a feedforward neural network. The multimodal information fusion classification module integrates and filters the information learned by the image processing module and the text processing module. Through the feedforward neural network based on the multilayer perceptron, the multimodal space is mapped to a shared semantic subspace, and the high-order interaction relationship between the features of the two modalities is learned, thereby obtaining the error detection result.
2. The OCR error detection method based on multimodal information fusion according to claim 1, characterized in that: In step (1), binarization is to convert the pixel values in the image to 0 or 1, denoising is to eliminate noise and burrs in the image, and cutting is to divide the image into multiple cells according to the row and column structure of the table, each cell containing a cell image and an OCR text.
3. The OCR error detection method based on multimodal information fusion according to claim 1, characterized in that: In step (2), the detection label refers to providing a correct label for the image and text of each cell, indicating whether there is an OCR error. The OCR error refers to the inconsistency between the text in the image and the text in the text.
4. The OCR error detection method based on multimodal information fusion according to claim 1, characterized in that: Step (4) specifically includes the following sequential steps: (4a) Based on the cell images in the error detection training dataset, the visual features of the images are obtained through the residual network of the image processing module; (4b) Based on the OCR text corresponding to the cell image, the feature representation of the text is obtained through the glyph convolutional network of the text processing module; (4c) Input the visual features of the image and the feature representation of the text into the feedforward neural network in the multi-module information fusion classification module for further learning, and finally obtain the error detection result.
5. The OCR error detection method based on multimodal information fusion according to claim 1, characterized in that: In step (4), the training parameter Epoch is set to 300, the batch size is set to 128, the optimization algorithm is Adam, the initial learning rate is set to 0.001, and the error detection training dataset is divided into training and test sets in a 9:1 ratio. The loss function of the multimodal cell OCR error detection model is: ， ， in, It is a function of the number of training iterations t. Indicates the initial value. Indicates the attenuation value. These are all hyperparameters. It is the cross-entropy loss function. It is the logarithmic loss function; The formula for the cross-entropy loss function is: ， in, It is the entire training set, which contains training data, belong , representing each training data point, These represent cell images and OCR text, respectively. This represents the i-th character in the OCR text. This indicates whether the cell image matches the OCR text; This represents the probability that the model predicts a match between the cell image and the OCR text. The formula for the logarithmic loss function is: ， in, A glyph vector representing a character.
6. The OCR error detection method based on multimodal information fusion according to claim 4, characterized in that: Step (4b) specifically refers to: ， ， Where E represents the feature representation of the output text. Represents the ReLU activation function. This represents a vector representing the position of a character in the text. This represents the convolution operation in a glyph convolutional network. The glyph vector representing a character. Functions for generating character image vectors We obtain W, which is the trainable parameter matrix.