Document structured evaluation method and device, equipment and storage medium

Abstract

The invention provides a document structured evaluation method and device, equipment and a storage medium. The method comprises the steps that structured annotation information of the complex layout document and structured detection information output by a pre-trained document structured model are obtained, the structured annotation information comprises multivariate attribute information of a first type of elements, and the structured detection information comprises multivariate attribute information of a second type of elements; according to the multivariate attribute information of the first type of elements and the multivariate attribute information of the second type of elements, determining an element matching pair set between the first type of elements and the second type of elements, a first type of element missing detection set and a second type of element false detection set; and determining a structured deviation score of the complex layout document according to the matching difference degree of each element matching pair in the element matching pair set and the number of elements in the first type of element missing detection set and the second type of element false detection set. According to the method and the device, the overall structured evaluation of the complex layout document is realized, and the document structured effect is ensured to be comprehensively and intuitively embodied.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a document structure evaluation method, apparatus, device, and storage medium. Background Technology

[0002] With the development of educational informatization, Optical Character Recognition (OCR) technology has been widely used in educational scenarios such as intelligent grading, automatic homework correction, digital courseware, and smart classroom note analysis. However, exam papers, with their complex layouts containing multiple elements such as question boxes, question stems, and answer areas, typically require structured detection and recognition of these elements to achieve comprehensive and accurate identification of complex document layouts.

[0003] Currently, in order to ensure the structured effect of complex layout documents, a single-element statistical strategy is usually adopted to separately count the structured results of each element in the complex layout document. This results in too many structured indicators in the complex layout document, making it impossible to intuitively reflect the overall structured effect of the complex layout document. Summary of the Invention

[0004] This application provides a document structure evaluation method, apparatus, device, and storage medium to achieve overall structure evaluation of any complex document layout, ensuring a comprehensive and intuitive representation of the document structure effect and improving the efficiency and accuracy of document structure evaluation.

[0005] In a first aspect, embodiments of this application provide a document structure evaluation method, the method comprising: Obtain structured annotation information of complex layout documents and structured detection information output by a pre-trained document structuring model, wherein the structured annotation information includes multi-dimensional attribute information of a first type of element and the structured detection information includes multi-dimensional attribute information of a second type of element. Based on the multi-attribute information of the first type of elements and the multi-attribute information of the second type of elements, determine the set of element matching pairs between the first type of elements and the second type of elements, as well as the set of missed detections of the first type of elements and the set of false detections of the second type of elements. The structured deviation score of the complex layout document is determined based on the matching difference degree of each element matching pair in the element matching pair set and the number of elements in the first type of element missed detection set and the second type of element false detection set.

[0006] Secondly, embodiments of this application provide a document structure evaluation device, the device comprising: The structured information acquisition module is used to acquire structured annotation information of complex layout documents and structured detection information output by a pre-trained document structured model. The structured annotation information includes multi-dimensional attribute information of a first type of element, and the structured detection information includes multi-dimensional attribute information of a second type of element. The feature matching module is used to determine the set of feature matching pairs between the first type of features and the second type of features, as well as the set of missed detections of the first type of features and the set of false detections of the second type of features, based on the multi-attribute information of the first type of features and the multi-attribute information of the second type of features. The structured evaluation module is used to determine the structured deviation score of the complex layout document based on the matching difference degree of each element matching pair in the element matching pair set and the number of elements in the first type of element missed detection set and the second type of element false detection set.

[0007] Thirdly, embodiments of this application provide an electronic device, which includes: A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the document structure evaluation method provided in the first aspect of this application.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to execute the document structure evaluation method provided in the first aspect of this application.

[0009] Fifthly, embodiments of this application provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the document structure evaluation method provided in the first aspect of this application.

[0010] The technical solution provided in this application first obtains the structured annotation information of a complex layout document and the structured detection information output by a pre-trained document structuring model. The structured annotation information includes multi-dimensional attribute information of a first type of element, and the structured detection information includes multi-dimensional attribute information of a second type of element. Then, based on the multi-dimensional attribute information of the first and second types of elements, a set of element matching pairs between the first and second types of elements, a set of missed detections of the first type of element, and a set of false detections of the second type of element are determined. Therefore, based on the matching difference degree of each element matching pair in the element matching pair set and the number of elements in the first and second types of missed detections and false detections, a structured deviation score for the complex layout document is determined, thereby achieving an overall structured evaluation of the complex layout document. While using the matching difference degree of each element matching pair in the element matching pair set to represent the structured effect of a single element in a complex layout document, the structured deviation score of the complex layout document can further represent the overall structured effect of the complex layout document, thereby optimizing the structured indicators of the complex layout document, ensuring a comprehensive and intuitive representation of the document's structured effect, and improving the efficiency and accuracy of document structured evaluation. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart of a document structure evaluation method provided in this application embodiment; Figure 2 A flowchart illustrating the matching process between the first type of elements and the second type of elements provided in this application embodiment; Figure 3 A flowchart illustrating the structured scoring process for complex layout documents provided in this application embodiment; Figure 4 A schematic diagram of a document structuring evaluation device provided in this application embodiment; Figure 5 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0015] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or solution described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0016] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more, that is, at least two. "At least one" means one or more. "Any" means any one or any several.

[0017] Considering that using a single-factor statistical strategy to assess the structured effect of complex layout documents would result in too many structured indicators, making it difficult to intuitively reflect the overall structured effect of the complex layout document, this application provides a new scheme for structured evaluation of complex layout documents. By performing pairwise matching between each labeled first-type element and each second-type element output by the model in the complex layout document, the matching difference degree of each element matching pair in the element matching pair set represents the structured effect of a single element in the complex layout document. Furthermore, the structured deviation score of the complex layout document is analyzed by using the matching difference degree of each element matching pair in the element matching pair set, the number of missed detections of first-type elements, and the number of false detections of second-type elements, to represent the overall structured effect of the complex layout document. This optimizes the structured indicators of complex layout documents, ensures a comprehensive and intuitive reflection of the document's structured effect, and improves the efficiency and accuracy of document structured evaluation.

[0018] Figure 1 This is a flowchart illustrating a document structuring evaluation method provided in an embodiment of this application. This method can be executed by the document structuring evaluation device provided in this application. The document structuring evaluation device can be implemented using any software and / or hardware method. Exemplarily, this document structuring evaluation device can be applied to any electronic device, including but not limited to tablet computers, mobile phones (such as foldable phones, large-screen phones, etc.), wearable devices, in-vehicle devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), smart TVs, smart screens, high-definition TVs, 4K TVs, smart speakers, smart projectors, and other various computing devices. This disclosure does not impose any restrictions on the specific type of electronic device.

[0019] Specifically, such as Figure 1 As shown, the method may include the following steps: S110, Obtain the structured annotation information of complex layout documents and the structured detection information output by the pre-trained document structuring model, wherein the structured annotation information includes the multi-dimensional attribute information of the first type of elements and the structured detection information includes the multi-dimensional attribute information of the second type of elements.

[0020] Complex layout documents are various documents designed and arranged using multiple heterogeneous elements, combined according to multi-level logic, and under strict spatial constraints, such as exam papers, magazines, and books. The individual elements in a complex layout document refer to all the specific components that constitute its visual and informational content, including content elements used to convey various textual information, such as titles, body text, tables, and figure captions.

[0021] This application can use an exam paper as an example to provide an exemplary description of various elements in a complex document layout.

[0022] Specifically, a complex document like an exam paper can include various elements such as pages, layout, columns, question boxes of different sizes, question stems, answer areas, icons, paragraphs, text lines, text blocks, and text units.

[0023] Furthermore, in order to achieve efficient readability of complex layout documents, there will be corresponding hierarchical nesting relationships between the various elements in the complex layout document, so as to organize the various elements into a logically clear reading path, making it easier for users to understand the information structure and text content in the complex layout document.

[0024] Understandably, to achieve comprehensive content recognition of complex-formatted documents, it is typically necessary to perform structured detection and recognition of each element within the document. Furthermore, each element in a complex-formatted document possesses multiple attributes, including element type, location, associated text, and hierarchical nesting relationships. Therefore, this application constructs a document structuring model to detect each element in a complex-formatted document and determine its multi-dimensional attribute information. In this case, the accuracy of the document structuring model directly impacts the structuring effect of the complex-formatted document.

[0025] Therefore, this application can continuously optimize the accuracy of the document structuring model by evaluating the structuring effect of complex layout documents.

[0026] In some embodiments, the document structuring model can typically be trained using cross-validation (e.g., 5-fold cross-validation). This involves proportionally dividing all training samples for the document structuring model into a training set and a validation set. The training set is used to train the document structuring model, while the validation set is used to evaluate the structuring accuracy of the trained model, thereby optimizing and updating the document structuring model. Therefore, the complex layout documents in this application can serve as training samples in the aforementioned validation set for training the document structuring model.

[0027] In this application, to achieve structured evaluation of complex layout documents, the document first undergoes manual annotation to pre-annotate various attribute information of each element in the complex layout document, such as element type, location, related text, and hierarchical nesting relationship, in order to obtain structured annotation information of the complex layout document. This structured annotation information may include multi-attribute information of the first type of elements.

[0028] The first category of elements can be the various elements manually annotated in a complex document layout. The multivariate attribute information of each first category element can be represented by tuples.

[0029] Suppose there are n first-class elements in a complex document layout, and each first-class element has four attributes: element type, location, related text, and hierarchical nesting relationship. Then, the structured annotation information of a complex document layout can be represented as G={g1,g2,...,g...} n}, g i Let g be the i-th first-class element. Furthermore, each first-class element g... i Multi-attribute information can be represented as a tuple (t g ,b g ,s g ,r g ).

[0030] Among them, t g It can be the first type of element g i Type information, such as question box, question stem, etc.; b g It can be the first type of element g i The coordinates of its position in a complex document layout, such as the coordinates of its outline; g It can be the first type of element g i The specific text content; r g It can be the first type of element g i The hierarchical nesting relationship.

[0031] It should be noted that, due to the hierarchical nesting relationships between various elements in complex document layouts, this application can use a tree diagram to construct the topological structure between these elements, clearly expressing their hierarchical nesting relationships. Therefore, for the hierarchical nesting relationship r in the multi-attribute information of the first type of element... g In this application, a tree diagram can be used to describe each first-class element g. i Hierarchical nesting relationship r in multi-attribute information g .

[0032] Furthermore, to achieve structured evaluation of complex layout documents, this application can also input the complex layout document into a pre-trained document structuring model, which will then detect various elements within the complex layout document and output corresponding structured detection information. This structured annotation information may include multi-attribute information of the second type of elements.

[0033] The second type of element can be any element detected by the document structure model in a complex layout document. The multivariate attribute information of each second type of element can also be represented by tuples.

[0034] Suppose that a complex layout document contains m second-class elements output by the document structuring model, and each second-class element also possesses four attributes: element type, location, related text, and hierarchical nesting relationship. Then, the structured detection information of a complex layout document can be represented as P = {p1, p2, ..., p...} m}, p i This is the i-th second-class element. Furthermore, each second-class element p... i Multi-attribute information can be represented as a tuple (t p ,b p ,s p ,r p ).

[0035] It is understandable that the second type of element p i Multi-attribute information (t) p ,b p ,s p ,r p ) and the first type of element g i Multi-attribute information (t) g ,b g ,s g ,r g The meanings of the various attributes in ) are the same.

[0036] S120, Based on the multi-attribute information of the first type of elements and the multi-attribute information of the second type of elements, determine the set of element matching pairs between the first type of elements and the second type of elements, as well as the set of missed detections of the first type of elements and the set of false detections of the second type of elements.

[0037] After obtaining the structured annotation information and structured detection information of complex layout documents, this application can use the multi-attribute information of each first-class element in the structured annotation information and the multi-attribute information of each second-class element in the structured detection information to analyze whether any first-class element and any second-class element have a large similarity under each attribute, thereby determining whether the two first-class elements and second-class elements match.

[0038] Therefore, by traversing each first-class element, this application can determine whether the first-class element matches any of the second-class elements using the aforementioned method, and select a second-class element from the various second-class elements that matches the first-class element, thus forming a corresponding element matching pair. Then, for the next traversed first-class element, the same method can be used to determine whether the first-class element matches any of the remaining unmatched second-class elements. By repeating this process, multiple element matching pairs between first-class and second-class elements can be obtained, thus forming the element matching pair set in this application.

[0039] For example, the feature matching pair set M may include multiple feature matching pairs (g i ,p i ).

[0040] Furthermore, due to the limitations of document structure models in terms of accuracy, they may have a certain rate of missed detections and false detections for various elements in complex document layouts. Therefore, after performing matching analysis on the first and second categories of elements and obtaining the corresponding set of element matching pairs, there may be first and second category elements that fail to match.

[0041] At this point, since the first type of elements are genuine elements manually annotated in the complex layout document, the unmatched elements of the first type are considered missed elements, thus forming the first type of missed element set. Conversely, since the second type of elements are elements detected by the document structure model in the complex layout document, the unmatched elements of the second type are considered falsely detected elements, thus forming the second type of falsely detected element set.

[0042] In some embodiments, the present application may determine the first type of missing feature set and the second type of false feature set by the following method: based on the feature matching pair set, determine the unmatched first type of features and the second type of features; form the unmatched first type of features into the first type of missing feature set, and form the unmatched second type of features into the second type of false feature set.

[0043] In other words, after determining the set of matching pairs between the first and second categories of elements, this application can use the first and second categories of elements that match each element in the matching pair set to determine the remaining unmatched first and second categories of elements. Then, the unmatched first categories of elements are grouped into a corresponding set of missed detections for the first category of elements, and the unmatched second categories of elements are grouped into a corresponding set of false detections for the second category of elements.

[0044] It should be noted that the number of elements in the first type of missed detection set and the second type of false detection set can be multiple or zero.

[0045] For example, the first type of missed feature set can be represented as U g The set of false positives for the second type of element can be represented as U. p .

[0046] S130, determine the structured deviation score of complex layout documents based on the matching difference degree of each feature matching pair in the feature matching pair set and the number of features in the first type of missed feature set and the second type of false feature set.

[0047] For each feature matching pair in the feature matching pair set, the specific degree of matching between the first type of features and the second type of features in each pair can represent the structuring effect of a complex layout document. Moreover, the number of features in the first type of missed feature set can represent the structured missed features in a complex layout document, while the number of features in the second type of false positive feature set can represent the structured false positive features in a complex layout document.

[0048] Therefore, in order to accurately evaluate the structured effect of complex layout documents, this application can analyze the specific matching degree between the first type of elements and the second type of elements in each element matching pair in the element matching pair set, in order to determine how much distance still exists between these two first type of elements and the second type of elements to achieve a completely consistent match, thereby determining the matching difference degree of each element matching pair.

[0049] Furthermore, the structured false negative rate of each element in a complex layout document is determined by analyzing the number of elements in the first type of false negative set. The structured false positive rate of each element in a complex layout document is determined by analyzing the number of elements in the second type of false positive set.

[0050] Then, by conducting a unified comprehensive analysis of the matching difference degree of each element matching pair in the element matching pair set, as well as the aforementioned structured false negative rate and structured false positive rate, the overall deviation probability of complex layout documents after structured detection is determined, thereby obtaining the structured deviation score of complex layout documents, which intuitively reflects the overall structured effect of complex layout documents.

[0051] The technical solution provided in this application first obtains the structured annotation information of a complex layout document and the structured detection information output by a pre-trained document structuring model. The structured annotation information includes multi-dimensional attribute information of a first type of element, and the structured detection information includes multi-dimensional attribute information of a second type of element. Then, based on the multi-dimensional attribute information of the first and second types of elements, a set of element matching pairs between the first and second types of elements, a set of missed detections of the first type of element, and a set of false detections of the second type of element are determined. Therefore, based on the matching difference degree of each element matching pair in the element matching pair set and the number of elements in the first and second types of missed detections and false detections, a structured deviation score for the complex layout document is determined, thereby achieving an overall structured evaluation of the complex layout document. While using the matching difference degree of each element matching pair in the element matching pair set to represent the structured effect of a single element in a complex layout document, the structured deviation score of the complex layout document can further represent the overall structured effect of the complex layout document, thereby optimizing the structured indicators of the complex layout document, ensuring a comprehensive and intuitive representation of the document's structured effect, and improving the efficiency and accuracy of document structured evaluation.

[0052] As an optional implementation scheme in this application, in order to ensure accurate matching of elements in complex layout documents, this application can provide a detailed explanation of the specific matching process between the first type of elements and the second type of elements.

[0053] Figure 2 A flowchart illustrating the method for matching first-class and second-class elements as provided in embodiments of this application. Figure 2 As shown, the method may specifically include the following steps: S210, Based on the multi-attribute information of the first type of elements and the multi-attribute information of the second type of elements, determine the multi-attribute similarity between the first type of elements and the second type of elements.

[0054] Considering that both the first and second categories of elements possess multiple attribute information such as element type, location, related text, and hierarchical nesting relationship, this application can determine the specific information values ​​of the two first and second categories of elements under the same attribute from the multi-attribute information of any first category element and any second category element, thereby calculating the similarity between the two first and second categories of elements under each attribute.

[0055] Therefore, by traversing each first-class element, this application can use the aforementioned similarity calculation method to calculate the similarity between the first-class element and each second-class element under each attribute, thus obtaining the multi-attribute similarity between the first-class element and each second-class element. By repeating this process, the multi-attribute similarity between any two first-class elements and second-class elements can be obtained.

[0056] Taking the four attributes of element type, location, related text, and hierarchical nesting relationship in a complex document as an example, the first type of element g i The multi-attribute information is (t) g ,b g ,s g ,r g ), second type of element p i The multi-attribute information is (t) p ,b p ,s p ,r p ).

[0057] So, for the first type of element g i With the second type of element p i Similarity S under the attribute of feature type t In other words, if So S t =1; and if So S t =0.

[0058] For the first type of element g i With the second type of element p i Similarity S under the attribute of location b In this regard, the Intersection Over Union (IoU) algorithm can be used to calculate the first type of feature g. i The bounding box of the annotation and the second type of feature p i The IoU value between the detected bounding boxes is denoted as the similarity S under the corresponding attribute. b Therefore, the first type of element g i With the second type of element p i Similarity S under the attribute of location b The formula for calculating IoU can be:

[0059] Where A represents the area of ​​the bounding box.

[0060] Regarding the first type of element g i With the second type of element p iSince the detection of the location attribute is not strict, this application can modify the IoU algorithm to use the first type of element g. i The bounding box of the annotation and the second type of feature p i The minimum bounding box area within the detected bounding box is used to replace the union of the areas of two bounding boxes, and this is used to calculate the first type of feature g. i With the second type of element p i Similarity S under the attribute of location b .

[0061] Among them, the first type of element g i With the second type of element p i Similarity S under the attribute of location b The variation of the IoU calculation formula used can be:

[0062] For the first type of element g i With the second type of element p i Similarity S under the attribute involving text s In this regard, the Minimum Edit Distance (MED) algorithm can be used to calculate the first type of feature g. i The text content s g Transform into second-class element p i The text content s p The minimum number of operations, combined with the second type of element p i Chinese text content s p The character length is used to calculate the similarity S under the corresponding attribute. s Therefore, the first type of element g i With the second type of element p i Similarity S under the attribute involving text s The calculation formula can be:

[0063] in, The minimum number of operations required to transform the text content sg in the first type of element gi into the text content sp in the second type of element pi. The length of the text content sp in the second type of element pi is denoted by .

[0064] For the first type of element g i With the second type of element p i Similarity S under the attribute of hierarchical nesting relationship rIn this regard, the Tree Edit Distance based Similarity (TEDS) algorithm can be used to calculate the first type of feature g. i The hierarchical nesting relationship r in g Transform into second-class element p i The hierarchical nesting relationship r in p The minimum number of operations, combined with the first type of element g. i Mid-level nested relationship r g Second type of element p i Mid-level nested relationship r p The maximum number of feature nodes is used to calculate the similarity S under the corresponding attribute. r Therefore, the first type of element g i With the second type of element p i Similarity S under the attribute of hierarchical nesting relationship r The calculation formula can be:

[0065] in, For the first type of element g i The hierarchical nesting relationship r in g Transform into second-class element p i The hierarchical nesting relationship r in p The minimum number of operations. For the first type of element g i Mid-level nested relationship r g The maximum number of element nodes. Second type of element p i Mid-level nested relationship r p The maximum number of element nodes.

[0066] S220, Based on multi-attribute similarity and preset multi-attribute constraints, determine the set of element matching pairs between the first type of elements and the second type of elements.

[0067] Considering that both the first and second categories of elements possess multiple attributes such as element type, location, associated text, and hierarchical nesting, this application can pre-define a corresponding matching constraint for each attribute to ensure accurate matching between them. This multi-attribute constraint represents the specific conditions that the similarity between any two first-category and second-category elements must meet under each attribute if they match.

[0068] Therefore, after determining the multi-attribute similarity between any two first-type and second-type elements, this application can perform a matching analysis on these two first-type and second-type elements by judging whether the similarity of these two first-type and second-type elements under each attribute satisfies the matching constraint condition corresponding to that attribute in the above multi-attribute constraints. That is, if the similarity of these two first-type and second-type elements under each attribute satisfies the above multi-attribute constraints, then it can be determined that these two first-type and second-type elements match and form a corresponding element matching pair.

[0069] Therefore, by using the above multi-attribute constraints to analyze the multi-attribute similarity between any two first-class elements and second-class elements, multiple element matching pairs between the first-class elements and second-class elements can be determined, thus forming the element matching pair set in this application.

[0070] In some embodiments, considering that if there are multiple similar elements in a complex layout document, when matching the first type of elements and the second type of elements pairwise, there may be cases where the same first type of element matches with multiple second type of elements, or multiple first type of elements match with the same second type of element, making the element matching pairs inaccurate and unable to accurately represent the structured effect of the complex layout document.

[0071] Therefore, in order to ensure accurate matching between the first type of elements and the second type of elements, this application can further explain the specific matching process between the first type of elements and the second type of elements in the following way: determine candidate element matching pairs between the first type of elements and the second type of elements based on multi-attribute similarity and preset multi-attribute constraints; determine multiple candidate element matching pair sets based on heterogeneous element matching pairs in the candidate element matching pairs; and determine the corresponding element matching pair set based on the overall similarity of each candidate element matching pair set.

[0072] Specifically, after obtaining the multi-attribute similarity between any two first-class and second-class elements, this application can perform matching analysis on these two first-class and second-class elements by determining whether the similarity of any two first-class and second-class elements under each attribute satisfies the matching constraint condition corresponding to that attribute in the preset multi-attribute constraint conditions. That is, if the similarity of these two first-class and second-class elements under each attribute satisfies the above-mentioned multi-attribute constraint conditions, it can be determined that these two first-class and second-class elements have a possibility of matching, and thus form a corresponding candidate element matching pair.

[0073] At this point, the candidate element matching pairs may contain the same first-type elements and / or second-type elements. Therefore, to ensure accurate matching between the first-type and second-type elements, this application can compare the first-type and second-type elements in each candidate element matching pair to iteratively filter out multiple heterogeneous element matching pairs formed by different elements under different matching paths, and then form a candidate element matching pair set for each matching path. Thus, multiple candidate element matching pair sets can be obtained under multiple matching paths.

[0074] For example, suppose there are candidate element matching pairs (g1, p2), (g1, p3), (g2, p1), (g2, p3), (g3, p1), and (g3, p3) between the first and second types of elements. Then, (g1, p2), (g2, p1), and (g3, p3) can be heterogeneous element matching pairs under a matching path, forming a set of candidate element matching pairs. Similarly, (g1, p3) and (g2, p1) can also be heterogeneous element matching pairs under a matching path, forming a set of candidate element matching pairs. (g2, p1) and (g3, p3) can also be heterogeneous element matching pairs under a matching path, forming a set of candidate element matching pairs. (g2, p3) and (g3, p1) can also be heterogeneous element matching pairs under a matching path, forming a set of candidate element matching pairs. Thus, we obtain a set of four candidate element matching pairs.

[0075] Only when there is an optimal set of matching element pairs between the first and second categories of elements can the structured effect of complex document layouts be optimized. Therefore, for the multiple candidate sets of matching element pairs mentioned above, this application can calculate the overall similarity of the candidate set of matching element pairs by comprehensively analyzing the multi-attribute similarity of each heterogeneous element pair in the candidate set. Then, the candidate set of matching element pairs with the highest overall similarity is taken as the optimal matching result between the first and second categories of elements, and this is recorded as the final set of matching element pairs.

[0076] In some embodiments, in order to ensure accurate matching between the first type of elements and the second type of elements, the overall similarity of any candidate element matching pair set can be determined by the following method: using preset attribute weights, the multi-attribute similarity between the first type of elements and the second type of elements in the candidate element matching pair is weighted and summed to obtain the matching similarity of the candidate element matching pair; based on the matching similarity of each candidate element matching pair in any candidate element matching pair set, the overall similarity of the candidate element matching pair set is determined.

[0077] In other words, for each candidate element matching pair between the first and second types of elements, this application can obtain the multi-attribute similarity between the first and second types of elements in that candidate element matching pair. However, for the structured effect of complex document layouts, the emphasis on different attributes of each element varies. For example, in document analysis scenarios, the focus is more on the accurate identification of text content in each element; while in target detection scenarios such as charts or graphics in documents, the focus is more on the accurate detection of the location of each element.

[0078] Therefore, in order to further improve the accuracy of structured evaluation of complex layout documents, this application can analyze the emphasis of the current structured task on the different attributes of each element, and pre-set different weights for each attribute as attribute weights in this application.

[0079] Then, after obtaining the multi-attribute similarity between the first and second categories of elements in each candidate element matching pair, this application can use multiple preset attribute weights to perform a weighted summation of the multi-attribute similarity between the first and second categories of elements in the candidate element matching pair to calculate the matching similarity between the first and second categories of elements in the candidate element matching pair. Therefore, for each candidate element matching pair, the matching similarity can be calculated using the above method.

[0080] Furthermore, for the multiple candidate element matching pair sets composed of heterogeneous element matching pairs in each candidate element matching pair set, this application can directly determine the matching similarity of each candidate element matching pair in each candidate element matching pair set, and add the matching similarities of each candidate element matching pair in each candidate element matching pair set to obtain the sum of the similarities of the candidate element matching pair set. Then, by normalizing the sum of the similarities of each candidate element matching pair set, the overall similarity of each candidate element matching pair set can be obtained.

[0081] The technical solution provided in this application calculates the multi-attribute similarity between the first type of elements and the second type of elements based on the multi-attribute information of the first type of elements and the multi-attribute information of the second type of elements. Combined with preset multi-attribute constraints, the first type of elements and the second type of elements are matched pairwise to obtain the corresponding set of element matching pairs. This achieves accurate matching between the first type of elements and the second type of elements, and improves the efficiency and accuracy of document structure evaluation.

[0082] According to one or more embodiments of this application, in order to ensure the accuracy of structured evaluation of complex layout documents, this application can provide a detailed explanation of the specific determination process of structured deviation scoring for complex layout documents.

[0083] Figure 3 A flowchart illustrating the structured scoring process for complex layout documents provided in this application embodiment. Figure 3 As shown, the method may specifically include the following steps: S310, Determine the matching difference of a feature pair based on the multi-attribute similarity of each feature pair in the feature matching pair set.

[0084] By performing pairwise matching analysis on the first and second categories of elements, we can obtain the corresponding set of element matching pairs, the set of missed detections of the first category of elements, and the set of false detections of the second category of elements.

[0085] For each element matching pair in the element matching pair set, this application can calculate the multi-attribute similarity between the first type of elements and the second type of elements in the element matching pair during the element matching process. Therefore, this application can determine the overall difference between the first type of elements and the second type of elements in the element matching pair by comprehensively analyzing the multi-attribute similarity between the first type of elements and the second type of elements in the element matching pair, thereby determining the matching difference degree of the element matching pair.

[0086] In some embodiments, this application may use preset attribute weights to perform a weighted summation of the multi-attribute similarity of each element matching pair to calculate the matching similarity of the element matching pair, thereby analyzing the complementarity of the matching similarity of the element matching pair, which is denoted as the matching difference of the element matching pair.

[0087] In other embodiments, the matching difference degree of each element matching pair can also be determined by the following method: determining the multi-attribute difference degree of the element matching pair based on the multi-attribute similarity of each element matching pair in the element matching pair set; and using preset attribute weights, performing a weighted summation of the multi-attribute difference degrees of each element matching pair in the element matching pair set to obtain the matching difference degree of the element matching pair.

[0088] Specifically, for each element matching pair in the element matching pair set, this application can calculate the complementarity number of the similarity of the element matching pair under each attribute based on the multi-attribute similarity between the first type of elements and the second type of elements in the element matching pair, thereby obtaining the multi-attribute difference of the element matching pair.

[0089] Taking a complex document layout where each element possesses four attributes—element type, location, associated text, and hierarchical nesting relationship—as an example, the first type of element g in each element matching pair in the element matching pair set... i The multi-attribute information is (t) g ,b g ,s g ,r g ), second type of element pi The multi-attribute information is (t) p ,b p ,s p ,r p ).

[0090] So, for the first type of element g i With the second type of element p i Difference d under the attribute of feature type t In other words, if , then d t =0; and if , then d t =1.

[0091] For the first type of element g i With the second type of element p i The degree of difference d under the attribute of location b For example, due to the first type of element g i With the second type of element p i Similarity S under the attribute of location b The IoU algorithm is used for calculation. Therefore, the first type of element g... i With the second type of element p i The degree of difference d under the attribute of location b The calculation formula can be:

[0092] Regarding the first type of element g i With the second type of element p i Since the detection of the location attribute is not strict, this application can modify the IoU algorithm to use the first type of element g. i The bounding box of the annotation and the second type of feature p i The minimum bounding box area within the detected bounding box is used to replace the union of the areas of two bounding boxes, and this is used to calculate the first type of feature g. i With the second type of element p i Similarity S under the attribute of location b So, the first type of element g i With the second type of element p i The degree of difference d under the attribute of location b The calculation formula can also be:

[0093] For the first type of element g i With the second type of element p i Difference d under the attribute involving text s For example, due to the first type of element gi With the second type of element p i Similarity S under the attribute involving text s Using the MED algorithm, combined with the second type of element p i Chinese text content s p The length of the character is used for calculation. Therefore, the first type of element g... i With the second type of element p i Difference d under the attribute involving text s The calculation formula can be:

[0094] For the first type of element g i With the second type of element p i The difference d under the attribute of hierarchical nesting relationship r For example, due to the first type of element g i With the second type of element p i Similarity S under the attribute of hierarchical nesting relationship r Using the TEDS algorithm, combined with the first type of element g i Mid-level nested relationship r g Second type of element p i Mid-level nested relationship r p The maximum number of feature nodes is used for calculation. Therefore, the first type of feature g... i With the second type of element p i The difference d under the attribute of hierarchical nesting relationship r The calculation formula can be:

[0095] In summary, following the above method, the multi-attribute dissimilarity of each feature matching pair in the feature matching pair set can be calculated.

[0096] Furthermore, for the structured evaluation of complex document layouts, the emphasis on different attributes of each element varies. For example, in document analysis scenarios, the focus is more on the accurate identification of text content within each element; while in target detection scenarios such as charts or graphics within a document, the focus is more on the accurate detection of the location of each element. Therefore, to further improve the accuracy of structured evaluation of complex document layouts, this application can analyze the emphasis of the current structured task on the different attributes of each element and pre-set different weights for each attribute as attribute weights in this application.

[0097] Therefore, after obtaining the multi-attribute difference degree of each feature matching pair in the feature matching pair set, this application can use multiple preset attribute weights to perform a weighted summation of the multi-attribute difference degrees between the first type of features and the second type of features in each feature matching pair to calculate the matching difference degree of that feature matching pair. Thus, for each feature matching pair in the feature matching pair set, the matching difference degree of each feature matching pair can be calculated using the above method.

[0098] Taking a complex document layout where each element possesses four attributes—element type, location, associated text, and hierarchical nesting relationship—as an example, the formula for calculating the matching difference of each element pair can be:

[0099] in, , , and These are the pre-defined attribute weights for four attributes: feature type, location, related text, and hierarchical nesting relationship. , , and For each feature pair, match the first type of feature g. i With the second type of element p i The degree of difference under four attributes: feature type, location, text involved, and hierarchical nesting relationship.

[0100] S320. Based on the total number of first-class elements in the structured annotation information, the matching difference degree of each element matching pair in the element matching pair set, the preset missed detection weight and false detection weight, and the number of elements in the first-class element missed detection set and the second-class element false detection set, determine the structured deviation score of the complex layout document.

[0101] For structured documents with complex layouts, the emphasis on missed and false detections for each element varies. Therefore, to further improve the accuracy of structured evaluation for complex documents, this application can also pre-set different weights for missed and false detections by analyzing the emphasis of the current structured task on missed and false detections, so as to analyze the missed and false detection situation of structured documents in complex layouts later.

[0102] After obtaining the matching difference degree of each feature matching pair in the feature matching pair set, this application can sum the matching difference degrees of each feature matching pair in the feature matching pair set to obtain the corresponding total matching difference degree. Furthermore, using preset missed detection weights and false detection weights, the number of elements in the first type of missed detection set and the number of elements in the second type of false detection set are weighted and summed. Combined with the aforementioned total matching difference degree, the overall structured deviation result of the complex layout document is calculated. Then, using the total number of first-type elements in the structured annotation information, the overall structured deviation result of the complex layout document is normalized to obtain the structured deviation score of the complex layout document, thereby achieving accurate structured evaluation of the complex layout document.

[0103] For example, the formula for calculating the structured deviation score of a complex layout document can be:

[0104] Where M is the set of feature matching pairs. N is the total number of first-class features in the structured annotation information, used to represent the normalization factor. The sum of the matching differences of each feature matching pair in the feature matching pair set M is the total matching difference. , These are the preset weights for missed detections and false detections, respectively. This represents the number of elements in the first type of missed detection set. This represents the number of elements in the second type of false detection set.

[0105] The technical solution provided in this application determines the structured deviation score of a complex layout document based on the matching difference degree of each element matching pair in the element matching pair set and the number of elements in the first type of missing element set and the second type of false element set. This achieves the overall structured evaluation of the complex layout document. While using the matching difference degree of each element matching pair in the element matching pair set to represent the structured effect of a single element in a complex layout document, the structured deviation score of the complex layout document can be further used to represent the overall structured effect of the complex layout document. This optimizes the structured indicators of the complex layout document, ensures a comprehensive and intuitive representation of the document's structured effect, and improves the efficiency and accuracy of the document structured evaluation.

[0106] Figure 4 This is a schematic block diagram of a document structuring evaluation device provided in an embodiment of this application. Figure 4 As shown, the device 400 may include: The structured information acquisition module 410 is used to acquire structured annotation information of complex layout documents and structured detection information output by a pre-trained document structured model, wherein the structured annotation information includes multi-dimensional attribute information of a first type of element and the structured detection information includes multi-dimensional attribute information of a second type of element. The feature matching module 420 is used to determine the feature matching pair set between the first type of features and the second type of features, as well as the first type of feature missed detection set and the second type of feature false detection set, based on the multi-attribute information of the first type of features and the multi-attribute information of the second type of features. The structured evaluation module 430 is used to determine the structured deviation score of the complex layout document based on the matching difference degree of each element matching pair in the element matching pair set and the number of elements in the first type of element missed detection set and the second type of element false detection set.

[0107] In some implementations, the feature matching module 420 may include: The similarity calculation unit is used to determine the multi-attribute similarity between the first type of elements and the second type of elements based on the multi-attribute information of the first type of elements and the multi-attribute information of the second type of elements. The feature matching unit is used to determine the set of feature matching pairs between the first type of features and the second type of features based on the multi-attribute similarity and preset multi-attribute constraints.

[0108] In some implementations, the feature matching unit can be specifically used for: Based on the multi-attribute similarity and the preset multi-attribute constraints, candidate element matching pairs between the first type of elements and the second type of elements are determined. Based on the heterogeneous element matching pairs in the candidate element matching pairs, a set of multiple candidate element matching pairs is determined; The corresponding set of matching elements is determined based on the overall similarity of each set of candidate element matching pairs.

[0109] In some implementations, the similarity calculation unit can also be used for: Using preset attribute weights, the multi-attribute similarity between the first type of elements and the second type of elements in the candidate element matching pair is weighted and summed to obtain the matching similarity of the candidate element matching pair. The overall similarity of the candidate element matching pair set is determined based on the matching similarity of each candidate element matching pair in any candidate element matching pair set.

[0110] In some implementations, the feature matching module 420 may further include a missed detection / false detection determination unit. This missed detection / false detection determination unit can be used for: Based on the set of matching element pairs, determine the first and second categories of unmatched elements; Unmatched first-category features are grouped into a first-category feature missed detection set, and unmatched second-category features are grouped into a second-category feature false detection set.

[0111] In some implementations, the structured assessment module 430 may include: The matching difference determination unit is used to determine the matching difference degree of the feature matching pair based on the multi-attribute similarity of each feature matching pair in the feature matching pair set. The structured evaluation unit is used to determine the structured deviation score of the complex layout document based on the total number of first-type elements in the structured annotation information, the matching difference degree of each element matching pair in the element matching pair set, the preset missed detection weight and false detection weight, and the number of elements in the first-type element missed detection set and the second-type element false detection set.

[0112] In some implementations, the matching difference determination unit can be specifically used for: Based on the multi-attribute similarity of each element matching pair in the set of element matching pairs, determine the multi-attribute difference of the element matching pair. Using preset attribute weights, the multi-attribute differences of each feature matching pair in the feature matching pair set are weighted and summed to obtain the matching difference of the feature matching pair.

[0113] In this embodiment, the structured annotation information of the complex layout document and the structured detection information output by the pre-trained document structuring model are first obtained. The structured annotation information includes multi-dimensional attribute information of the first type of elements, and the structured detection information includes multi-dimensional attribute information of the second type of elements. Then, based on the multi-dimensional attribute information of the first and second types of elements, the set of element matching pairs between the first and second types of elements, as well as the set of missed detections of the first type of elements and the set of false detections of the second type of elements, are determined. Therefore, based on the matching difference degree of each element matching pair in the element matching pair set and the number of elements in the set of missed detections of the first type of elements and the set of false detections of the second type of elements, the structured deviation score of the complex layout document is determined, thereby achieving the overall structured evaluation of the complex layout document. While using the matching difference degree of each element matching pair in the element matching pair set to represent the structured effect of a single element in the complex layout document, the structured deviation score of the complex layout document can further represent the overall structured effect of the complex layout document, thereby optimizing the structured indicators of the complex layout document, ensuring a comprehensive and intuitive representation of the document's structured effect, and improving the efficiency and accuracy of document structured evaluation.

[0114] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 4 The apparatus 400 shown can execute any of the method embodiments provided in this application, and the foregoing and other operations and / or functions of each module in the apparatus 400 are respectively for implementing the corresponding processes in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0115] The apparatus 400 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly manifested as execution by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0116] Figure 5 A schematic block diagram of an electronic device provided in an embodiment of this application.

[0117] like Figure 5 As shown, the electronic device 500 may include: The system includes a memory 510 and a processor 520. The memory 510 stores computer programs and transfers the program code to the processor 520. In other words, the processor 520 can retrieve and run the computer program from the memory 510 to implement the methods described in the embodiments of this application.

[0118] For example, the processor 520 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0119] In some embodiments of this application, the processor 520 may include, but is not limited to: General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0120] In some embodiments of this application, the memory 510 includes, but is not limited to: Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0121] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 510 and executed by the processor 520 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0122] like Figure 5 As shown, the electronic device may further include: Transceiver 530, which can be connected to processor 520 or memory 510.

[0123] The processor 520 can control the transceiver 530 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 530 may include a transmitter and a receiver. The transceiver 530 may further include antennas, and the number of antennas may be one or more.

[0124] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0125] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, this application also provides a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0126] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0127] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0129] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0130] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A document structure evaluation method, characterized in that, include: Obtain structured annotation information of complex layout documents and structured detection information output by a pre-trained document structuring model, wherein the structured annotation information includes multi-dimensional attribute information of a first type of element and the structured detection information includes multi-dimensional attribute information of a second type of element. Based on the multi-attribute information of the first type of elements and the multi-attribute information of the second type of elements, determine the set of element matching pairs between the first type of elements and the second type of elements, as well as the set of missed detections of the first type of elements and the set of false detections of the second type of elements. The structured deviation score of the complex layout document is determined based on the matching difference degree of each element matching pair in the element matching pair set and the number of elements in the first type of element missed detection set and the second type of element false detection set.

2. The method according to claim 1, characterized in that, The step of determining the set of element matching pairs between the first type of elements and the second type of elements based on the multi-attribute information of the first type of elements and the multi-attribute information of the second type of elements includes: Based on the multi-attribute information of the first type of elements and the multi-attribute information of the second type of elements, determine the multi-attribute similarity between the first type of elements and the second type of elements; Based on the multi-attribute similarity and the preset multi-attribute constraints, a set of element matching pairs between the first type of elements and the second type of elements is determined.

3. The method according to claim 2, characterized in that, The step of determining the set of element matching pairs between the first type of elements and the second type of elements based on the multi-attribute similarity and preset multi-attribute constraints includes: Based on the multi-attribute similarity and the preset multi-attribute constraints, candidate element matching pairs between the first type of elements and the second type of elements are determined. Based on the heterogeneous element matching pairs in the candidate element matching pairs, a set of multiple candidate element matching pairs is determined; The corresponding set of matching elements is determined based on the overall similarity of each set of candidate element matching pairs.

4. The method according to claim 3, characterized in that, The method further includes: Using preset attribute weights, the multi-attribute similarity between the first type of elements and the second type of elements in the candidate element matching pair is weighted and summed to obtain the matching similarity of the candidate element matching pair. The overall similarity of the candidate element matching pair set is determined based on the matching similarity of each candidate element matching pair in any candidate element matching pair set.

5. The method according to claim 1, characterized in that, Determining the first type of missed detection set and the second type of false detection set includes: Based on the set of matching element pairs, determine the first and second categories of unmatched elements; Unmatched first-category features are grouped into a first-category feature missed detection set, and unmatched second-category features are grouped into a second-category feature false detection set.

6. The method according to claim 1, characterized in that, The step of determining the structured deviation score of the complex layout document based on the matching difference degree of each element matching pair in the element matching pair set and the number of elements in the first type of missed element detection set and the second type of false element detection set includes: Based on the multi-attribute similarity of each feature matching pair in the feature matching pair set, the matching difference of the feature matching pair is determined. The structured deviation score of the complex layout document is determined based on the total number of first-type elements in the structured annotation information, the matching difference degree of each element matching pair in the element matching pair set, the preset missed detection weight and false detection weight, and the number of elements in the first-type element missed detection set and the second-type element false detection set.

7. The method according to claim 6, characterized in that, The step of determining the matching difference of a feature matching pair based on the multi-attribute similarity of each feature matching pair in the feature matching pair set includes: Based on the multi-attribute similarity of each element matching pair in the set of element matching pairs, determine the multi-attribute difference of the element matching pair. Using preset attribute weights, the multi-attribute differences of each feature matching pair in the feature matching pair set are weighted and summed to obtain the matching difference of the feature matching pair.

8. A structured assessment device for document 4 documents, characterized in that, include: The structured information acquisition module is used to acquire structured annotation information of complex layout documents and structured detection information output by a pre-trained document structured model. The structured annotation information includes multi-dimensional attribute information of a first type of element, and the structured detection information includes multi-dimensional attribute information of a second type of element. The feature matching module is used to determine the set of feature matching pairs between the first type of features and the second type of features, as well as the set of missed detections of the first type of features and the set of false detections of the second type of features, based on the multi-attribute information of the first type of features and the multi-attribute information of the second type of features. The structured evaluation module is used to determine the structured deviation score of the complex layout document based on the matching difference degree of each element matching pair in the element matching pair set and the number of elements in the first type of element missed detection set and the second type of element false detection set.

9. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to perform the document structure evaluation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs that cause a computer to perform the document structuring evaluation method as described in any one of claims 1-7.

Images