A method for detecting the performance of OCR curved document correction based on multimodal distance collaborative optimization

Abstract

This invention discloses a method for detecting the performance of OCR curved document correction based on multimodal distance collaborative optimization, which relates to the fields of computer vision and image processing. The method includes the following steps: S1, constructing a cost matrix; S2, calculating the optimal matching; S3, performance evaluation. By fusing a weighted cost matrix of geometric distance and text similarity, and combining it with the Hungarian algorithm, the optimal matching is achieved. It also supports dynamic parameter adjustment and abnormal data processing to accurately quantify the improvement or degradation of the distortion correction algorithm on the OCR recognition effect.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing, specifically to a method for detecting the performance of OCR curved document correction based on multimodal distance collaborative optimization. In particular, for images processed by distortion correction (Dewarp), the method combines a comprehensive evaluation of geometric distance and text similarity to achieve accurate quantitative analysis of the OCR distortion correction algorithm effect. Background Technology

[0002] Currently, OCR technology faces certain challenges when processing distortion-corrected images. In particular, after distortion correction, the image size and position may change, causing the correspondence between the OCR recognition result and the original image to become non-linear. Due to the lack of effective evaluation standards, existing technologies cannot accurately measure the effects of distortion correction, and the following problems remain in their application:

[0003] 1. Single evaluation standard: Traditional methods only evaluate distortion correction effect through pixel comparison or simple error analysis, ignoring the correlation between text semantics and spatial relationships.

[0004] 2. Insufficient matching accuracy: The matching does not effectively combine the geometric position of the text box (bbox) (such as the Euclidean distance of the center point and the intersection-union ratio) with the text content (such as the edit distance).

[0005] 3. Poor dynamic adaptability: The weighting coefficients and evaluation thresholds are fixed, which cannot adapt to the differentiated needs of different scenarios (such as medical document and invoice recognition).

[0006] 4. Sensitive to abnormal data: Missing text boxes or text errors in OCR recognition can lead to deviations in matching results, and there is a lack of robust repair mechanisms.

[0007] Therefore, there is an urgent need for an evaluation method that can integrate spatial and textual information and support dynamic adjustments. Summary of the Invention

[0008] 1. Technical problems to be solved:

[0009] To address the problems existing in the prior art, the present invention aims to provide an OCR curved document correction performance detection method based on multimodal distance collaborative optimization. By fusing a weighted cost matrix of geometric distance and text similarity, and combining it with the Hungarian algorithm to achieve optimal matching, the method supports dynamic parameter adjustment and abnormal data processing, so as to accurately quantify the improvement or degradation of the distortion correction algorithm on the OCR recognition effect.

[0010] 2. Technical Solution:

[0011] To solve the above problems, the present invention adopts the following technical solutions.

[0012] A method for detecting the performance of OCR curved document correction based on multimodal distance collaborative optimization includes the following steps:

[0013] S1. Construct the cost matrix: For the distortion-corrected image, detect the OCR text box set B in the original image and the OCR text box set D in the distortion-corrected image, and calculate the cost matrix b for each original text box. i With distortion-corrected text box d j The multimodal matching cost C between them ij , i∈{1,m},j∈{1,n},m andn are the number of text boxes after distortion correction;

[0014] The multimodal matching cost is composed of a weighted average of geometric distance and text similarity distance, as shown in the formula:

[0015] ;

[0016] Where α and β are weighting coefficients, d geom d represents the geometric distance between text boxes. text Indicates the edit distance of the text content;

[0017] S2. Optimal Matching Calculation: Based on the constructed cost matrix, the Hungarian Algorithm is used to calculate the global optimal match between the original text box set B and the distortion-corrected text box set D, and the corresponding optimal allocation matrix X is obtained.

[0018] S3. Performance Evaluation: Based on the optimal matching results, calculate the matching rate, average geometric error, and average text editing distance, and quantitatively evaluate the performance of the distortion correction algorithm by combining the preset threshold.

[0019] A further improvement lies in the geometric distance d geom Calculated in the following way:

[0020] ;

[0021] Where γ and δ are adjustment parameters, d center is the Euclidean distance between the center points of the text boxes, and IoU is the intersection-union ratio.

[0022] A further improvement lies in the fact that the text similarity distance d text The formula for calculating the normalized edit distance is as follows:

[0023] ;

[0024] EditDistance represents the minimum number of editing operations (insertion, deletion, replacement) between the two strings, and after normalization, it is ensured to be in the range [0,1].

[0025] A further improvement lies in the fact that the weighting coefficients α and β are dynamically adjusted according to the actual application scenario, which is achieved through the following steps:

[0026] (a) Train a regression model based on historical matching data to predict the optimal weight combination;

[0027] (b) Manually adjust the weighting coefficients based on the evaluation requirements input by the user.

[0028] A further improvement is that the matching rate in the performance evaluation is calculated as follows:

[0029] ;

[0030] Among them, X ij These are the matching matrix elements output by the Hungarian algorithm.

[0031] A further improvement is that the method also includes a visualization step, which displays the matching relationship between the OCR text boxes of the original image and the distortion-corrected image in the form of a heatmap or line graph.

[0032] A further improvement is that the preset threshold is set in the following way:

[0033] (a) Based on the target scenario of the distortion correction algorithm (such as medical document and invoice recognition), preset the initial threshold;

[0034] (b) Based on the statistical distribution of real-time matching results, the threshold is dynamically adjusted to optimize the evaluation sensitivity.

[0035] A further improvement is that the method also includes an anomaly data processing step:

[0036] (a) Detect missing text boxes or text errors in the OCR recognition results;

[0037] (b) After repairing the abnormal data through interpolation or semantic completion algorithms, the matching calculation is performed again.

[0038] 3. Beneficial effects:

[0039] Compared with the prior art, the technical solution provided by this invention has the following advantages:

[0040] (1) By comprehensively considering spatial error and text error, this invention overcomes the limitation of traditional evaluation methods that only focus on a single factor, thereby achieving more accurate performance testing.

[0041] (2) This method can effectively quantify the improvement or degradation of the distortion correction model in OCR tasks, providing an objective performance evaluation standard, which helps to optimize and adjust the model.

[0042] (3) The application of the Hungarian algorithm ensures the accuracy of the best match, thereby improving the robustness and accuracy of the detection.

[0043] (4) By adjusting the coefficients α and β, the evaluation results can be flexibly adapted to different application requirements, making the method more versatile and scalable.

[0044] (5) Through experimental verification, the present invention improves the matching rate by 19% (from 72% to 91%), reduces the average geometric error by 55.7% (from 18.5 pixels to 8.2 pixels), and reduces the average text editing distance by 57.1% (from 0.35 to 0.15) in the medical document scenario, which is significantly better than the traditional single-modal evaluation method.

[0045] It should be noted that the structures not described in this invention are not related to the design points and improvement directions of this invention, and are the same as or can be implemented using existing technologies, so they will not be elaborated here. Attached Figure Description

[0046] Figure 1 This is the original image input for this invention;

[0047] Figure 2 It is Figure 1 The text box appears after OCR recognition;

[0048] Figure 3 It is Figure 1 Image after bending correction;

[0049] Figure 4 It is Figure 3 The text box appears after OCR recognition;

[0050] Figure 5 This is the matching result after the above images are respectively subjected to OCR recognition and then subjected to multimodal detection by this invention;

[0051] Figure 6 This is a schematic diagram of the overall process of the present invention. Detailed Implementation

[0052] To facilitate understanding of the present invention, a more complete description of the invention will be given below with reference to the accompanying drawings, which illustrate several embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the invention will be more thorough and complete.

[0053] Please see Figure 1-6 This invention proposes an OCR distortion removal performance detection method based on multimodal distance collaborative optimization, the specific steps of which are as follows:

[0054] 1. Construct the cost matrix: For the distortion-corrected image, calculate the Euclidean distance and edit distance of the center point of each bounding box by detecting each bounding box (bbox) in the image and its corresponding OCR recognition result.

[0055] 2. Cost Calculation: Each element in the cost matrix is ​​determined by two factors: the Euclidean distance to the text box position and the edit distance of the text within the text box. The weighted cost is calculated by setting two weighting coefficients (α and β), i.e.:

[0056] Cost = α * Euclidean distance + β * edit distance

[0057] 3. Optimal Matching: Using the Hungarian algorithm, the optimal text box matching method is calculated through the cost matrix to ensure that the text boxes in the distortion-corrected image match the text boxes in the original image optimally.

[0058] 4. Threshold Judgment: Set a threshold to judge the matching quality. The threshold can be adjusted according to the application scenario, so as to quantitatively evaluate the performance of the distortion correction algorithm.

[0059] Specifically:

[0060] 1. Representation of OCR detection box

[0061] Input image such as Figure 1 The image shown is the original image, and its OCR detection result set is as follows:

[0062] ;

[0063] Among them, each OCR text box b i Represented by its top-left and bottom-right corner coordinates and the detected text:

[0064] ;

[0065] in:

[0066] (x) min y min () represents the coordinates of the top-left corner of the text box;

[0067] (x) max y max () represents the coordinates of the bottom right corner of the text box;

[0068] t i The text content recognized by OCR.

[0069] The result of plotting the coordinate frame on the original image is as follows: Figure 2 As shown.

[0070] Similarly, the image after distortion correction is as follows: Figure 3 As shown, the set of OCR detection results is as follows: ;

[0071] Each distortion-corrected text box d j It is also represented by its coordinates and text content:

[0072] ;

[0073] in:

[0074] (x' min y' min () represents the coordinates of the top-left corner of the text box after distortion correction;

[0075] (x' max y' max () represents the coordinates of the bottom right corner of the text box after distortion correction;

[0076] t' i This is the OCR text content after distortion correction.

[0077] Where m and n are the number of text boxes before and after distortion correction, respectively, and m ≠ n.

[0078] The result of plotting the coordinate frame on the original image is as follows: Figure 4 As shown.

[0079] 2. Construction of the cost matrix

[0080] We need to construct an m*n dimensional cost matrix C, where each element C ij This represents the original OCR text box b. i With distortion-corrected text box d j The matching cost between them. The cost can be determined by several factors, which we define as follows:

[0081] ;

[0082] Where, d geom (b) i d j () represents geometric distance, measuring changes in the position and shape of a text box;

[0083] d text (b) i d j () represents the text similarity cost, which measures the differences in text recognized by OCR;

[0084] α and β are hyperparameters used to balance the weights of geometric distance and text similarity.

[0085] 2.1. Geometric Distance Calculation

[0086] Geometric distance d geom (b) i d j The Euclidean distance between the center points can be calculated using a combination of the IoU (Intersection over Union).

[0087] ;

[0088] in:

[0089] Euclidean distance between the center points:

[0090] ;

[0091] Among them, (x) c ,y c ) and (x c ',y c ') are respectively b i and d j Center point:

[0092] ;

[0093] IoU calculation:

[0094] ;

[0095] Here, γ and δ are adjustment parameters used to balance the effects of both.

[0096] 2.2. Text Edit Distance

[0097] EditDistance, the screenwriting distance, abbreviated as D, is calculated as follows:

[0098] ;

[0099] Text similarity cost d text (b) i d j The above edit distance calculation is used:

[0100] ;

[0101] EditDistance represents the minimum number of editing operations (insertion, deletion, replacement) between the two strings, and after normalization, it is ensured to be in the range [0,1].

[0102] 3. Optimal matching calculation

[0103] Given the cost matrix C, we use the Hungarian algorithm to solve for the optimal matching and obtain the corresponding optimal allocation matrix X:

[0104] ;

[0105] in:

[0106] X ij ∈{0,1}, if b i and d j If they match, then X ij =1, otherwise 0.

[0107] Satisfies the unique matching constraint:

[0108] ;

[0109] The Hungarian algorithm solves this minimum-cost bipartite graph matching problem, guaranteeing a globally optimal solution.

[0110] 4. Results Analysis

[0111] After solving the problem using the Hungarian algorithm, the optimal matching relation (b) is obtained. i d j ), and can calculate the following performance metrics:

[0112] a. Match rate:

[0113] ;

[0114] b. Average edit distance:

[0115] ;

[0116] C. Mean geometric error:

[0117] ;

[0118] The above calculations can be used to quantitatively evaluate the OCR distortion correction effect and can be used to optimize the distortion correction model.

[0119] Visualize the matching between the OCR results of the original image and the OCR results of the image after Dewarp distortion correction, as shown below. Figure 5 As shown.

[0120] The overall process involved in this invention is as follows: Figure 6 As shown.

[0121] Example 1: Application of Medical Document Correction

[0122] This embodiment uses medical medical records as an example to illustrate the specific application of this method in OCR performance testing after bending correction.

[0123] Parameter settings:

[0124] Weighting coefficients: α = 0.6 (geometric distance weight), β = 0.4 (text similarity weight);

[0125] Hungarian algorithm hyperparameters: maximum number of iterations is 1000, convergence threshold is 1e-5;

[0126] The preset matching rate threshold is 85%, the average geometric error threshold is 10 pixels, and the average text editing distance threshold is 0.2.

[0127] Experimental comparison:

[0128] The method was tested on 100 curved medical documents to compare the performance of traditional methods (based solely on geometric distance matching) with the multimodal matching method of this invention:

[0129]

[0130] Effect description:

[0131] By fusing geometric and textual features, this method significantly improves matching accuracy in medical document scenarios while reducing OCR recognition errors, thus verifying the effectiveness of multimodal collaborative optimization.

[0132] The Hungarian Algorithm is a polynomial-time algorithm for solving the minimum weight matching problem in bipartite graphs, with a time complexity of O(n³), where n is the number of text boxes to be matched. The reasons for choosing this algorithm in this invention include:

[0133] Global optimality: It can ensure a globally optimal match between the text boxes of the original image and the corrected image, avoiding local optima;

[0134] Applicability: Suitable for sparse or dense matching scenarios, and robust to noisy data;

[0135] Efficiency: In practical applications (n≤1000), the algorithm's running time can be controlled in milliseconds, meeting real-time requirements.

[0136] The above-described embodiments are merely illustrative of certain implementations of the present invention, and are described in a relatively specific and detailed manner. However, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements are all within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for detecting the performance of OCR curved document correction based on multimodal distance collaborative optimization, characterized in that, Includes the following steps: S1. Construct the cost matrix: For the distortion-corrected image, detect the OCR text box set B in the original image and the OCR text box set D in the distortion-corrected image, and calculate the cost matrix b for each original text box. i With distortion-corrected text box d j The multimodal matching cost C between them ij , i∈{1,m}, j∈{1,n}, where m and n are the number of text boxes before and after distortion correction, respectively; The multimodal matching cost is composed of a weighted average of geometric distance and text similarity distance, as shown in the formula: ； Where α and β are weighting coefficients, and α + β = 1, d geom The geometric distance between text boxes is represented by the text similarity distance d calculated using normalized edit distance. text ; The weighting coefficients α and β are dynamically adjusted according to the actual application scenario, which is achieved through the following steps: (a) Train a regression model based on historical matching data to predict the optimal weight combination; (b) Manually adjust the weighting coefficients based on the user's input evaluation requirements; S2. Optimal Matching Calculation: Based on the constructed cost matrix, the Hungarian Algorithm is used to calculate the global optimal match between the original text box set B and the distortion-corrected text box set D, and the corresponding optimal allocation matrix X is obtained. S3. Performance Evaluation: Based on the optimal matching results, calculate the matching rate, average geometric error, and average text editing distance, and quantitatively evaluate the performance of the distortion correction algorithm by combining the preset threshold.

2. The method for detecting the performance of OCR curved document correction based on multimodal distance collaborative optimization according to claim 1, characterized in that, The geometric distance d geom Calculated in the following way: ； Where γ and δ are adjustment parameters, d center is the Euclidean distance between the center points of the text boxes, and IoU is the intersection-union ratio.

3. The method for detecting the performance of OCR curved document correction based on multimodal distance collaborative optimization according to claim 1, characterized in that, The text similarity distance d text The formula for calculating the normalized edit distance is as follows: ； Here, EditDistance represents the minimum number of editing operations for two strings. The minimum number of editing operations includes insertion, deletion, and replacement, and after normalization, the range is ensured to be [0,1].

4. The method for detecting the performance of OCR curved document correction based on multimodal distance collaborative optimization according to claim 1, characterized in that, The matching rate in the performance evaluation is calculated as follows: ； Among them, X ij These are the matching matrix elements output by the Hungarian algorithm.

5. The method for detecting the performance of OCR curved document correction based on multimodal distance collaborative optimization according to claim 1, characterized in that, The method also includes a visualization step, which displays the OCR text box matching relationship between the original image and the distortion-corrected image in the form of a heatmap or line graph.

6. The method for detecting the performance of OCR curved document correction based on multimodal distance collaborative optimization according to claim 1, characterized in that, The preset threshold is set in the following way: (a) Based on the target scenario of the distortion correction algorithm, the target scenario includes medical document and invoice recognition, with a preset initial threshold; (b) Based on the statistical distribution of real-time matching results, the threshold is dynamically adjusted to optimize the evaluation sensitivity.

7. The method for detecting the performance of OCR curved document correction based on multimodal distance collaborative optimization according to claim 1, characterized in that, It also includes steps for handling abnormal data: (a) Detect missing text boxes or text errors in the OCR recognition results; (b) The coordinates of the missing text box are repaired by linear interpolation algorithm, and the text error is corrected based on semantic completion model.

Images