Document correction evaluation methods, apparatus, computer equipment and storage media

Abstract

This invention relates to the field of document corrector technology, and discloses a document corrector evaluation method, apparatus, computer device, and storage medium. First, an error-level injection operation is performed on the original annotated document to obtain an erroneously annotated document. Then, the document corrector uses the erroneously annotated document for correction processing to obtain an output document. Next, text line-by-line processing is performed on the original annotated document and the output document to obtain line-level matching index data, quantifying the effect of line-level content restoration. Document structure parsing is performed on the original annotated document and the output document to obtain tree structure similarity, reflecting the degree of restoration of the document's hierarchical structure. Finally, based on the line-level matching index data and the tree structure similarity, the correction quality evaluation result of the document corrector is determined, thereby achieving a comprehensive performance evaluation of the document corrector in both line-level content and hierarchical structure dimensions, improving the comprehensiveness and accuracy of document corrector performance evaluation.

Description

Technical Field

[0001] This invention relates to the field of document corrector technology, and in particular to a document corrector evaluation method, apparatus, computer device, and storage medium. Background Technology

[0002] In building AI applications based on Retrieval Enhanced Generation (RAG), structured preprocessing of documents is a crucial step. However, converting raw documents to Markdown format commonly suffers from issues such as missing headings and misaligned paragraph levels, affecting the accuracy of subsequent text slicing and vector indexing. Therefore, a document corrector evaluation method is needed. Summary of the Invention

[0003] The embodiments described in this specification aim to at least partially solve one of the technical problems in the related art. To this end, the embodiments of this specification propose a document corrector evaluation method, apparatus, computer device, and storage medium.

[0004] This specification provides a document corrector evaluation method, the method comprising: Perform error-level injection operations based on the original annotated document to obtain the erroneous annotated document; Based on the document corrector, the error-annotated document is corrected to obtain the output document; Based on the original annotated document and the output document, perform text line splitting operations to obtain line-level matching index data; Based on the original annotated document and the output document, document structure parsing is performed to obtain tree structure similarity; Based on the row-level matching index data and the tree structure similarity, the correction quality evaluation result of the document corrector is determined.

[0005] In one implementation, the error level injection operation includes at least one of level modification, level erasure, and level addition.

[0006] In one implementation, the text line-breaking operation based on the original annotated document and the output document to obtain line-level matching index data includes: Perform text line splitting on the original annotated document to obtain the original text line set; Perform text line splitting on the output document to obtain a set of output text lines; The original set of text lines and the output set of text lines are combined to obtain a new set of text lines. The text line set is matched with the original text line set and the output text line set respectively to obtain line-level matching index data.

[0007] In one implementation, the original text line set includes an original header line set, and the output text line set includes an output header line set. The step of matching the text line set with the original text line set and the output text line set respectively to obtain line-level matching index data includes: The number of true positive lines is determined based on the number of lines in the text line set that belong to both the original title line and the output title line; The number of false positive lines is determined based on the number of lines in the text line set that do not belong to the original title line but belong to the output title line; The number of false negative lines is determined based on the number of lines in the text line set that belong to the original header line but not to the output header line; The number of true negative lines is determined based on the number of lines in the text line set that do not belong to the original header line and do not belong to the output header line; Based on the number of true positive rows, the number of false positive rows, the number of false negative rows, and the number of true negative rows, row-level matching index data is determined.

[0008] In one implementation, determining the row-level matching index data based on the number of true positive rows, the number of false negative rows, and the number of true negative rows includes: The total sample data is obtained by summing the number of true positive rows, the number of false positive rows, the number of false negative rows, and the number of true negative rows. The summation of the number of true positive rows and the number of true negative rows is performed to obtain the correct sample data; The summation of the number of true positive rows and the number of false negative rows yields the total number of true positive examples. The accuracy rate is obtained by dividing the correct sample data by the total sample data. The recall rate is obtained by dividing the number of true positive rows by the total number of true positive examples. A calculation score is obtained based on the precision and recall, and the calculation score is determined as the row-level matching indicator data.

[0009] In one implementation, the step of performing document structure parsing processing based on the original annotated document and the output document to obtain tree structure similarity includes: The original annotated document is parsed to obtain the original title structure tree; The output document is parsed to obtain the output title structure tree; The similarity is calculated based on the original title structure tree and the output title structure tree to obtain the tree structure similarity.

[0010] In one implementation, the step of calculating the similarity between the original title structure tree and the output title structure tree to obtain the tree structure similarity includes: Based on the original title structure tree and the output title structure tree, determine the number of operation steps required to make the two tree structures consistent. The tree structure similarity is obtained based on the number of operation steps, the number of nodes in the original title structure tree, and the number of nodes in the output title structure tree.

[0011] This specification provides a document corrector evaluation apparatus, the apparatus comprising: The error level injection module is used to perform error level injection operations based on the original annotated document to obtain the error annotated document; The document correction processing module is used to perform correction processing on the error-annotated document based on the document corrector to obtain the output document; The text line-level matching module is used to perform text line-breaking operations based on the original annotated document and the output document to obtain line-level matching index data. The tree structure matching module is used to perform document structure parsing processing based on the original annotated document and the output document to obtain tree structure similarity. The quality assessment module is used to determine the quality assessment result of the document corrector based on the row-level matching index data and the tree structure similarity.

[0012] This specification provides a computer device comprising: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions executable by the one or more processors, the instructions being executed by the one or more processors to cause the one or more processors to perform the steps of the method described in any of the above embodiments.

[0013] This specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0014] This specification provides a computer program product that includes instructions that, when executed by a processor of a computer device, enable the computer device to perform the steps of the method described in any of the above embodiments.

[0015] In the above-described implementation method, a dual-index evaluation system based on line-level matching and tree structure hierarchy similarity is constructed for evaluating the document hierarchy calibration of the document corrector. First, an error-level injection operation is performed on the original annotated document to obtain an erroneously annotated document. Then, the document corrector uses the erroneously annotated document for correction processing to obtain the output document. Next, to quantitatively evaluate the document corrector's ability to correct line-level content from a text extraction perspective, a line-splitting operation is performed on the original annotated document and the output document to obtain line-level matching index data, thus quantifying the effect of line-level content restoration. Based on this, to more comprehensively measure the document corrector's performance in correcting the overall document structure, document structure parsing is performed on the original annotated document and the output document to obtain tree structure similarity, which is used to identify the correspondence between heading levels and reflect the degree of document hierarchy structure restoration. Finally, based on the line-level matching index data and tree structure similarity, the correction quality evaluation result of the document corrector is determined, thereby achieving a comprehensive performance evaluation of the document corrector in both line-level content and hierarchy structure dimensions, improving the comprehensiveness and accuracy of the document corrector's performance evaluation. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the document corrector evaluation method provided for embodiments of this specification; Figure 2 A flowchart illustrating the process of obtaining row-level matching index data for the implementation of this specification; Figure 3 A flowchart illustrating the process of obtaining row-level matching index data for the implementation of this specification; Figure 4 A flowchart illustrating the process of determining row-level matching index data for the implementation of this specification; Figure 5 A flowchart illustrating the process of obtaining tree structure similarity for embodiments of this specification; Figure 6 A flowchart illustrating the process of obtaining tree structure similarity for embodiments of this specification; Figure 7 A schematic diagram of the document corrector evaluation apparatus provided for embodiments of this specification; Figure 8 An internal structural diagram of a computer device provided for embodiments of this specification. Detailed Implementation

[0017] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0018] In building AI applications based on Retrieval Enhanced Generation (RAG), structured preprocessing of documents is a crucial step. However, converting raw files to Markdown format often results in issues such as missing headings and misaligned paragraph levels, severely impacting the accuracy of subsequent text slicing and vector indexing.

[0019] In the field of cross-format document conversion, structured preprocessing for Markdown faces the following three technical challenges: 1) Word documents based on XML parsing are prone to losing complex paragraph hierarchy information.

[0020] 2) OCR and layout model-driven PDF conversion often lead to a break in the association between text blocks.

[0021] 3) There are inherent differences between the writing style of the source document itself and the Markdown syntax standard.

[0022] To address the aforementioned issues, a Markdown text corrector is typically needed to preprocess the conversion results. This corrector takes the converted Markdown text from the original file as input and outputs the corrected text after error correction and missing paragraph structure completion. Since the accuracy of text correction directly impacts the knowledge recall rate of the RAG system, an effective method is required to evaluate the performance of the Markdown document corrector.

[0023] In related technologies, commonly used evaluation metrics for text correction (or proofreading / error correction) tasks can be mainly divided into the following categories: 1. Basic Indicators: Precision = Number of correct corrections / Total number of system corrections.

[0024] Recall = Corrected numbers / Total number of actual errors.

[0025] F1 score: the harmonic mean of precision and recall.

[0026] 2. Text similarity metrics: BLEU: Focuses on n-gram matching and is suitable for translation-based correction scenarios.

[0027] METEOR: Introduces synonym and stemming matching, making it more sensitive to semantic changes.

[0028] 3. Semantic evaluation: BERTScore: Calculates contextual semantic similarity based on the BERT model.

[0029] Sentence-BERT: Compares similarity using sentence-level semantic vectors.

[0030] However, none of the above metrics can assess the correctness of paragraph structure in structured text.

[0031] Based on the above analysis, this specification provides a document corrector evaluation method. For the document hierarchy calibration evaluation of the document corrector, a dual-index evaluation system based on line-level matching and tree structure hierarchy similarity is constructed. First, an error-level injection operation is performed on the original annotated document to obtain an erroneously annotated document. Then, the document corrector uses the erroneously annotated document for correction processing to obtain the output document. Next, to quantitatively evaluate the document corrector's ability to correct line-level content from the text extraction level, a text splitting operation is performed on the original annotated document and the output document to obtain line-level matching index data, thereby quantifying the effect of line-level content restoration. Furthermore, to more comprehensively measure the document corrector's performance in correcting the overall document structure, document structure parsing processing is performed on the original annotated document and the output document to obtain tree structure similarity, which is used to identify the correspondence between heading levels and reflect the degree of document hierarchy structure restoration. Finally, based on the line-level matching index data and tree structure similarity, the correction quality evaluation result of the document corrector is determined, thereby achieving a comprehensive performance evaluation of the document corrector in both line-level content and hierarchy structure dimensions, improving the comprehensiveness and accuracy of the document corrector performance evaluation.

[0032] This specification provides a document corrector evaluation method. Please refer to [link / reference]. Figure 1 The document corrector evaluation method may include the following steps: S110. Perform error-level injection based on the original annotated document to obtain the error-annotated document.

[0033] S120. Based on the document corrector, the error-annotated document is corrected to obtain the output document.

[0034] The original annotation document can be a source file that records the data annotation content and its structural information in plain text form. Its format can adopt the lightweight markup language Markdown, that is, to mark semantic units in the text with concise and easy-to-read symbols.

[0035] Specifically, the process begins by obtaining the original annotation document, which contains correct annotation information. Then, by performing an error hierarchy injection operation, various annotation errors are artificially introduced into the original annotation document according to preset error types and hierarchical structures, thereby generating an erroneous annotation document containing this error information. Next, a document corrector is used to correct the resulting erroneous annotation document. The document corrector can identify the error hierarchy in the document and automatically correct it, finally outputting the corrected document.

[0036] For example, an error hierarchy injection operation is performed based on the original annotation document D_base to obtain an erroneous annotation document D_wrong. The Markdown document corrector then uses the erroneous annotation document D_wrong for correction processing to obtain the output document D_output.

[0037] S130. Perform text line splitting operations based on the original annotated document and the output document to obtain line-level matching index data.

[0038] S140. Perform document structure parsing processing based on the original annotated document and the output document to obtain tree structure similarity.

[0039] Specifically, to evaluate the quality of the document corrector, it is first necessary to segment the original annotated document (i.e., the standard reference document) and the output document (i.e., the document to be evaluated generated by the corrector) line by line, compare the text content line by line, and calculate line-level matching metrics such as line-level precision, recall, and F1 score. It should be noted that this stage can only detect the extraction accuracy at the text level, that is, whether the character sequence of each line is consistent with the standard document, but it cannot identify structural errors in the directory hierarchy, such as misalignment between parent and child nodes, or indentation depth discrepancies. Therefore, relying solely on line-level matching metrics is insufficient to fully reflect the document corrector's restoration capability. To address this, further document structure parsing is performed on the same pair of original annotated documents and output documents, parsing both documents into tree-like structures. Each node corresponds to a directory entry or paragraph title, the parent-child relationship of nodes reflects the hierarchical nesting relationship of the document, and the order of nodes reflects the linear arrangement of the directory. Based on this, a metric method can be used to calculate the structural similarity between the two trees, obtaining the tree structure similarity.

[0040] S150. Based on row-level matching index data and tree structure similarity, determine the correction quality evaluation result of the document corrector.

[0041] Specifically, the correction quality assessment result of the document corrector is determined by combining row-level matching index data and tree structure similarity. In some implementations, the row-level matching index data and tree structure similarity are fused according to preset weights to determine the correction quality assessment result of the document corrector. When the correction quality assessment result reaches a preset threshold, the correction quality of the document corrector is considered to meet the requirements, thus obtaining a qualified correction quality assessment result. When the correction quality assessment result does not reach the preset threshold, the correction quality of the document corrector is considered to be unqualified, and its parameters need to be adjusted, optimized, or retrained, resulting in an unqualified correction quality assessment result.

[0042] In other implementations, if both the row-level matching index data and the tree structure similarity meet preset thresholds, the document corrector's correction quality is considered to have met the requirements, thus obtaining a satisfactory correction quality evaluation result. If at least one of them fails to meet the preset threshold, the document corrector's correction quality is considered to have failed, requiring parameter adjustment, optimization, or retraining, resulting in an unsatisfactory correction quality evaluation result.

[0043] In the above implementation, a dual-index evaluation system based on line-level matching and tree structure hierarchy similarity was constructed for evaluating the document hierarchy calibration of the document corrector. First, an error-level injection operation is performed on the original annotated document to obtain an erroneously annotated document. Then, the document corrector uses the erroneously annotated document for correction processing to obtain the output document. Next, to quantitatively evaluate the document corrector's ability to correct line-level content from a text extraction perspective, line-by-line processing is performed on the original annotated document and the output document to obtain line-level matching index data, thus quantifying the effect of line-level content restoration. Based on this, to more comprehensively measure the document corrector's performance in correcting the overall document structure, document structure parsing is performed on the original annotated document and the output document to obtain tree structure similarity, which is used to identify the correspondence between heading levels and reflect the degree of document hierarchy structure restoration. Finally, based on the line-level matching index data and tree structure similarity, the correction quality evaluation result of the document corrector is determined, thereby achieving a comprehensive performance evaluation of the document corrector in both line-level content and hierarchy structure dimensions, improving the comprehensiveness and accuracy of the document corrector performance evaluation.

[0044] In some implementations, the error level injection operation includes at least one of level modification, level erasure, and level addition.

[0045] Specifically, in order to simulate possible errors during the annotation process, it is necessary to perform error level injection operations on the original annotation document. The error level injection operations include at least one of level modification, level erasure, and level addition.

[0046] Hierarchical modification operations can change the hierarchical position of one or more annotation nodes in the original annotation document. For example, it can promote a child node that originally belonged to a parent node to a sibling node of the parent node, or demote a node that originally belonged to a certain level to the next level of its original child nodes, thereby disrupting the original hierarchical or sequential relationship.

[0047] Hierarchical erasure can be a purposeful deletion of one or more hierarchical nodes from the original annotated document. For example, it can be the deletion of a node and all its child nodes (i.e., the removal of a hierarchical branch as a whole), resulting in missing or discontinuous hierarchical structures in the document.

[0048] Adding levels can be done by inserting one or more additional level nodes on top of the original annotation document. For example, it can be done by forcibly adding an intermediate level between any two levels in the original annotation document, or by adding a false child node level to a standard node that should not have children, thereby creating redundant structural levels.

[0049] In the above embodiments, the error level injection operation includes at least one of level modification, level erasure, and level addition, so as to obtain error-annotated documents and provide a data basis for subsequent evaluation of document correctors.

[0050] In some implementations, please refer to Figure 2 The process of performing line-by-line text processing on the original annotated document and the output document to obtain line-level matching index data may include the following steps: S210. Perform text line splitting on the original annotated document to obtain the original text line set.

[0051] Specifically, the original annotated document is first processed by line splitting, using newline characters as delimiters to divide the text content in the document, thus treating each line of text as an independent unit, thereby obtaining the original text line set.

[0052] S220. Perform text line splitting on the output document to obtain a set of output text lines.

[0053] Specifically, the output document is first processed by line splitting, using newline characters as delimiters to divide the text content in the document, thus treating each line of text as an independent unit, thereby obtaining a set of output text lines.

[0054] S230. Perform a union operation on the original set of text lines and the output set of text lines to obtain the text line set.

[0055] S240. Perform matching calculations between the text line set and the original text line set and the output text line set respectively to obtain line-level matching index data.

[0056] Specifically, after obtaining the original text line set and the output text line set, a union operation is performed on the original and output text line sets, merging all text lines in both sets and removing duplicates to obtain a single text line set containing all distinct text lines from both sets. Then, using this text line set as a benchmark, line-by-line matching calculations are performed with both the original and output text line sets to determine whether a line exists in the original and output text line sets. This results in the calculation of line-level matching metrics, which quantify the accuracy of the output document relative to the original annotated document in terms of corrected header lines.

[0057] For example, the original annotated document D_base is processed by text line splitting to obtain the original text line set line_base. The output document D_output is processed by text line splitting to obtain the output text line set line_output. The original text line set line_base and the output text line set line_output are then combined to obtain the text line set lines.

[0058] In the above implementation, the effect of restoring line-level content is quantified by using line-level matching index data, thereby enabling a quantitative evaluation of the document corrector's ability to correct line-level content from the text extraction level.

[0059] In some implementations, please refer to Figure 3 The original text line set includes the original header line set, and the output text line set includes the output header line set. Matching the text line sets with the original text line set and the output text line set respectively to obtain line-level matching index data can include the following steps: S310. Determine the number of true positive lines based on the number of lines in the text line set that belong to both the original header line and the output header line.

[0060] S320. Determine the number of false positive lines based on the number of lines in the text line set that do not belong to the original header lines but belong to the output header lines.

[0061] S330. Determine the number of false negative rows based on the number of rows in the text line set that belong to the original header row but not to the output header row.

[0062] S340. Determine the number of true negative rows based on the number of rows in the text line set that do not belong to the original header row and do not belong to the output header row.

[0063] S350. Based on the number of true positive rows, false positive rows, false negative rows, and true negative rows, determine the row-level matching indicator data.

[0064] Specifically, for each text line in the set, it is determined whether it belongs to the original header line (i.e., the header line in the actual annotation) and whether it belongs to the output header line (i.e., the header line output after correction by the document corrector). Based on this, the following statistics are calculated: the number of text lines that belong to both the original and output header lines is determined as the number of true positive lines; the number of text lines that belong to the output header line but not the original header line is determined as the number of false positive lines; the number of text lines that belong to the original header line but not the output header line is determined as the number of false negative lines; and the number of text lines that belong to neither the original nor the output header line is determined as the number of true negative lines. Then, based on the number of true positive lines, false positive lines, false negative lines, and true negative lines, line-level matching index data is calculated to evaluate the accuracy of header line correction.

[0065] For example, the title line in the original text line set line_base is the original title line line_base_title, and the title line in the output text line set line_output is the output title line line_output_title.

[0066] The number of true positive lines tp = the number of lines in the text line set lines that are in the original title line line_base_title and the output title line line_output_title.

[0067] The number of false positive lines fp = the number of lines in the text line set lines that are not in the original title line line_base_title but are in the output title line line_output_title.

[0068] The number of false negative lines fn = the number of lines in the text line set lines that are in the original title line line_base_title and not in the output title line line_output_title.

[0069] The number of true negative lines tn = the number of lines in the text line set lines that are neither in the original title line line_base_title nor in the output title line line_output_title.

[0070] In the above implementation, the effect of restoring line-level content is quantified by using line-level matching index data, thereby enabling a quantitative evaluation of the document corrector's ability to correct line-level content from the text extraction level.

[0071] In some implementations, please refer to Figure 4 Determining row-level matching index data based on the number of true positive rows, false positive rows, false negative rows, and true negative rows may include the following steps: S410. Sum the number of true positive rows, false negative rows, false positive rows, and true negative rows to obtain the total sample data.

[0072] S420. Sum the number of true positive rows and the number of true negative rows to obtain the correct sample data.

[0073] S430. Sum the number of true positive rows and false negative rows to obtain the total number of true positive examples.

[0074] S440. Divide the correct sample data by the total sample data to obtain the accuracy.

[0075] S450. Divide the number of true positive rows by the total number of true positive examples to obtain the recall rate.

[0076] S460: Calculate the score based on precision and recall, and use the calculated score as the row-level matching indicator data.

[0077] Specifically, the total sample data is obtained by summing the number of true positive rows, false negative rows, false positive rows, and true negative rows. The correct sample data is obtained by summing the number of true positive rows and false negative rows. The total number of true positive rows and false negative rows is obtained by summing the number of true positive rows and false negative rows. Based on this, the precision rate is obtained by dividing the correct sample data by the total sample data, which measures the proportion of correct rows corrected by the document corrector. The recall rate is obtained by dividing the number of true positive rows by the total number of true positive rows, which measures the document corrector's ability to correct the actual number of correct rows. Finally, a composite score (e.g., the harmonic mean of precision and recall, i.e., the F1 score) is calculated based on the precision and recall, and this calculated score is used as the row-level matching metric.

[0078] For example, precision = (number of true positive rows tp + number of true negative rows tn) / (number of true positive rows tp + number of false positive rows fp + number of false negative rows fn + number of true negative rows tn).

[0079] Recall = number of true positive rows tp / (number of true positive rows tp + number of false negative rows fn).

[0080] F1 score = 2 (precision recall) / (precision+recall).

[0081] In the above implementation, the effect of restoring line-level content is quantified by using line-level matching index data, thereby enabling a quantitative evaluation of the document corrector's ability to correct line-level content from the text extraction level.

[0082] In some implementations, please refer to Figure 5 Document structure parsing based on the original annotated documents and the output documents to obtain tree structure similarity may include the following steps: S510. Perform document structure parsing on the original annotated document to obtain the original title structure tree.

[0083] Specifically, the original annotated document with various annotations (such as headings and text paragraphs) is used as input. A parsing algorithm or rule engine processes the document structure, automatically identifying and extracting the hierarchical relationships between headings, including subordinate relationships (nested parent and child headings) and parallel relationships (order of multiple headings at the same level), thus constructing a complete heading hierarchy. This parsing process traverses all annotations in the original document, determining the parent and child nodes of each heading based on its level identifier (e.g., first-level heading, second-level heading, third-level heading, etc.) and its order within the document. For example, when a first-level heading is followed by two second-level headings, the first-level heading becomes the parent node of these two second-level headings, while the two second-level headings are arranged from left to right according to their order of appearance in the document, becoming parallel child nodes. After this parsing process, a tree structure that comprehensively reflects the hierarchical relationships of chapters and sub-chapter structures in the original annotated document—the original heading structure tree—is finally output. In this original title structure tree, the root node represents the entire original annotated document, and the content corresponding to each chapter title serves as the content of each node. For example, the original annotated document D_base is parsed to obtain the original title structure tree T_base.

[0084] S520. Perform document structure parsing on the output document to obtain the output title structure tree.

[0085] Specifically, the output document, including various annotation information (such as titles and text paragraphs), is taken as input. A parsing algorithm or rule engine performs document structure parsing on the output document, automatically identifying and extracting the hierarchical logical relationships between the titles in the output document. This includes subordinate relationships (i.e., nested relationships between parent and child titles) and parallel relationships (i.e., the sequential arrangement of multiple titles at the same level), thereby constructing a complete title hierarchy system. This parsing process traverses all annotation content in the output document, determining the parent and child nodes of each title based on its level identifier (e.g., first-level, second-level, third-level, etc.) and its order in the output document. After the above parsing process, a tree structure that comprehensively reflects the hierarchical relationships of each chapter and sub-chapter in the output document is finally output, namely, the output title structure tree. In this output title structure tree, the root node can represent the entire output document, and the content corresponding to each chapter title is the content of each node. For example, the output document D_output is parsed to obtain the output title structure tree T_output.

[0086] S530. Calculate the similarity between the original title structure tree and the output title structure tree to obtain the tree structure similarity.

[0087] Specifically, by traversing the nodes and structural paths of the original title structure tree and the output title structure tree, tree edit distance (TED), subtree matching, or a similarity algorithm based on dynamic programming is used to compare and quantify features such as node labels, node depth, and subtree inclusion relationships layer by layer. Finally, a numerical value that can measure the degree of consistency between the two trees in terms of overall topological shape and hierarchical organization is calculated, namely, tree structure similarity.

[0088] In the above implementation, the original annotated document is parsed to obtain the original title structure tree, and the output document is parsed to obtain the output title structure tree. The similarity between the original title structure tree and the output title structure tree is calculated to obtain the tree structure similarity, which is used to identify the correspondence between the title levels and reflect the degree of restoration of the document hierarchy structure, so as to more comprehensively measure the correction performance of the document corrector on the overall document structure.

[0089] In some implementations, please refer to Figure 6 The similarity calculation based on the original title structure tree and the output title structure tree to obtain the tree structure similarity may include the following steps: S610. Based on the original title structure tree and the output title structure tree, determine the number of operation steps required to make the two tree structures consistent.

[0090] Specifically, using Tree Edit Distance (TED), based on the structural differences between the original and output title structure trees, the algorithm compares the hierarchical relationships, parent-child relationships, and the order of child nodes under the same parent node in each tree. It calculates the minimum number of operations required to transform the output title structure tree into a structure completely identical to the original title structure tree through a series of editing operations. Editing operations include node insertion, deletion, and movement. Insertion operations add nodes that exist in the original title structure tree but are missing in the output title structure tree; deletion operations remove nodes that exist in the output title structure tree but are not needed in the original title structure tree; and movement operations adjust the parent-child relationships or sibling order of nodes in the output title structure tree to match the hierarchy and order in the original title structure tree. For example, if the original title structure tree is T_base and the output title structure tree is T_output, then TED(T_base, T_output) represents the minimum number of operations, or operation steps, required to transform the output title structure tree T_output into a structure completely identical to the original title structure tree T_base through editing operations.

[0091] S620. Based on the number of operation steps, the number of nodes in the original title structure tree, and the number of nodes in the output title structure tree, obtain the tree structure similarity.

[0092] Specifically, the number of nodes reflects the size of the tree. After obtaining the minimum number of operation steps required to transform the output title structure tree into the original title structure tree, the tree structure similarity is obtained by normalizing the number of operation steps to the maximum of the number of nodes in both the original and output title structure trees. It should be noted that the closer the tree structure similarity is to 1, the more similar the structures of the original and output title structure trees are. For example, tree structure similarity = 1. TED(T_base, T_output) / max(size(T_base), size(T_output)), where size(T_base) represents the number of nodes in the original title structure tree, and size(T_output) represents the number of nodes in the output title structure tree.

[0093] In the above implementation, based on the original title structure tree and the output title structure tree, the number of operation steps required to make the two tree structures consistent is determined. Based on the number of operation steps, the number of nodes in the original title structure tree and the number of nodes in the output title structure tree, the tree structure similarity is obtained to identify the correspondence between title levels and reflect the degree of restoration of the document hierarchy structure, so as to more comprehensively measure the document corrector's performance in correcting the overall document structure.

[0094] This specification provides a document corrector evaluation apparatus 700. Please refer to [link to documentation]. Figure 7 The document corrector evaluation device 700 includes: an error level injection module 710, a document correction processing module 720, a text line-level matching module 730, a tree structure matching module 740, and a correction quality evaluation module 750.

[0095] Error level injection module 710 is used to perform error level injection operations based on the original annotation document to obtain the error annotation document; The document correction processing module 720 is used to perform correction processing on the error-annotated document based on the document corrector to obtain an output document; The text line-level matching module 730 is used to perform text line-breaking processing operations based on the original annotated document and the output document to obtain line-level matching index data. The tree structure matching module 740 is used to perform document structure parsing processing based on the original annotated document and the output document to obtain tree structure similarity. The quality assessment module 750 is used to determine the quality assessment result of the document corrector based on the row-level matching index data and the tree structure similarity.

[0096] For a detailed description of the document corrector evaluation apparatus, please refer to the description of the document corrector evaluation method above, which will not be repeated here.

[0097] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a document correction evaluation method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0098] Those skilled in the art will understand that Figure 8 The structures shown are merely block diagrams of some structures related to the solutions disclosed in this specification, and do not constitute a limitation on the computer device to which the solutions disclosed in this specification are applied. Specifically, the computer device may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0099] In some embodiments, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps described above.

[0100] This specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any of the above embodiments.

[0101] One embodiment of this specification provides a computer program product including instructions that, when executed by a processor of a computer device, enable the computer device to perform the steps of the method described in any of the above embodiments.

[0102] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

Claims

1. A document corrector evaluation method, characterized in that, The method includes: Perform error-level injection operations based on the original annotated document to obtain the erroneous annotated document; Based on the document corrector, the error-annotated document is corrected to obtain the output document; Based on the original annotated document and the output document, perform text line splitting operations to obtain line-level matching index data; Based on the original annotated document and the output document, document structure parsing is performed to obtain tree structure similarity; Based on the row-level matching index data and the tree structure similarity, the correction quality evaluation result of the document corrector is determined.

2. The method according to claim 1, characterized in that, Error level injection operations include at least one of level modification, level erasure, and level addition.

3. The method according to claim 1, characterized in that, The text line-breaking operation based on the original annotated document and the output document to obtain line-level matching index data includes: Perform text line splitting on the original annotated document to obtain the original text line set; Perform text line splitting on the output document to obtain a set of output text lines; The original set of text lines and the output set of text lines are combined to obtain a new set of text lines. The text line set is matched with the original text line set and the output text line set respectively to obtain line-level matching index data.

4. The method according to claim 3, characterized in that, The original text line set includes the original title line set, and the output text line set includes the output title line set. The step of matching the text line set with the original text line set and the output text line set respectively to obtain line-level matching index data includes: The number of true positive lines is determined based on the number of lines in the text line set that belong to both the original title line and the output title line; The number of false positive lines is determined based on the number of lines in the text line set that do not belong to the original title line but belong to the output title line; The number of false negative lines is determined based on the number of lines in the text line set that belong to the original header line but not to the output header line; The number of true negative lines is determined based on the number of lines in the text line set that do not belong to the original header line and do not belong to the output header line; Based on the number of true positive rows, the number of false positive rows, the number of false negative rows, and the number of true negative rows, row-level matching index data is determined.

5. The method according to claim 4, characterized in that, The determination of row-level matching index data based on the number of true positive rows, the number of false positive rows, the number of false negative rows, and the number of true negative rows includes: The total sample data is obtained by summing the number of true positive rows, the number of false positive rows, the number of false negative rows, and the number of true negative rows. The summation of the number of true positive rows and the number of true negative rows is performed to obtain the correct sample data; The summation of the number of true positive rows and the number of false negative rows yields the total number of true positive examples. The accuracy rate is obtained by dividing the correct sample data by the total sample data. The recall rate is obtained by dividing the number of true positive rows by the total number of true positive examples. A calculation score is obtained based on the precision and recall, and the calculation score is determined as the row-level matching indicator data.

6. The method according to claim 1, characterized in that, The step of performing document structure parsing processing based on the original annotated document and the output document to obtain tree structure similarity includes: The original annotated document is parsed to obtain the original title structure tree; The output document is parsed to obtain the output title structure tree; The similarity is calculated based on the original title structure tree and the output title structure tree to obtain the tree structure similarity.

7. The method according to claim 6, characterized in that, The step of calculating the similarity between the original title structure tree and the output title structure tree to obtain the tree structure similarity includes: Based on the original title structure tree and the output title structure tree, determine the number of operation steps required to make the two tree structures consistent. The tree structure similarity is obtained based on the number of operation steps, the number of nodes in the original title structure tree, and the number of nodes in the output title structure tree.

8. A document corrector evaluation device, characterized in that, The device includes: The error level injection module is used to perform error level injection operations based on the original annotated document to obtain the error annotated document; The document correction processing module is used to perform correction processing on the error-annotated document based on the document corrector to obtain the output document; The text line-level matching module is used to perform text line-breaking operations based on the original annotated document and the output document to obtain line-level matching index data. The tree structure matching module is used to perform document structure parsing processing based on the original annotated document and the output document to obtain tree structure similarity. The quality assessment module is used to determine the quality assessment result of the document corrector based on the row-level matching index data and the tree structure similarity.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images