Document parsing evaluation method and system based on multi-modal semantic consistency

Abstract

The application discloses a document parsing evaluation method and system based on multi-modal semantic consistency. The method comprises the following steps: acquiring multi-modal contract document data and preprocessing to obtain preprocessed data; extracting contract element features from the preprocessed data to generate a fusion feature vector; based on the fusion feature vector, using a double constraint loss function and a three-level judgment mechanism to evaluate the semantic consistency of the document elements to obtain a cross-modal semantic consistency quantitative result; based on the cross-modal semantic consistency quantitative result, calculating the basic and semantic enhancement indicators and integrating them into a comprehensive evaluation index; outputting the comprehensive evaluation index and the corresponding related suggestions to guide the contract intelligent parsing and algorithm optimization. By implementing the method of the application, intelligent and accurate document parsing can be realized, and the problems of geometric positioning dependence, cross-modal association loss and insufficient logic verification in existing document parsing technology can be solved.

Description

Technical Field

[0001] This invention relates to the field of document parsing technology, and more specifically to a document parsing evaluation method and system based on multimodal semantic consistency. Background Technology

[0002] In the field of document parsing, identifying and understanding key elements such as tables, charts, formulas, headings, and paragraphs is a core task. However, current evaluation systems suffer from several technical flaws, which limit the development of document parsing technology and its application effectiveness.

[0003] Existing solutions typically rely on traditional evaluation metrics, such as Intersection over Union (IoU) and Mean Precision (MP), to assess detection results by calculating the overlap of bounding box coordinates. While this method can classify and evaluate text or graphic elements based on geometric location, it can lead to double misclassification due to both bounding boxes and surface labels. Specifically, this strategy first filters candidate objects based on the overlap of bounding boxes, then performs matching verification based on category labels such as text boxes or tables, neglecting the verification of the elements' deeper semantic attributes. This results in problems such as incorrectly segmenting large text boxes that should be considered as a whole into multiple smaller text boxes, or ignoring the importance of certain indivisible text boxes, thus compromising their integrity.

[0004] Furthermore, documents often contain multimodal information, including image modalities, text modalities, and structural modalities. However, current evaluation methods primarily focus on the single image modality, failing to effectively utilize the semantic relationships between textual descriptions and visual elements. For example, when a visual chart (Figure XX) appearing in a document may conflict with a reference in the text, traditional methods cannot identify this semantic inconsistency. Moreover, due to the lack of cross-modal data interfaces and feature mapping mechanisms, existing systems struggle to establish an effective visual-text-structure cross-modal semantic consistency evaluation framework.

[0005] The various elements within a document follow specific layout and semantic logic rules, such as the relationship between titles and body text, and the adjacency of charts and their explanatory text. However, current evaluation systems do not adequately consider this logical consistency as part of the evaluation dimensions. This means that even if the position and classification of each individual contract element, such as the signature area or clause box, are accurate, a lack of necessary logical connections between them can affect the validity and legal effect of the entire document. For example, if the implicit rule that the client's signature area must be adjacent to the client's information is not correctly modeled, it may lead to problems with the abnormal placement of the signature area.

[0006] Therefore, it is necessary to design a new method to achieve intelligent and accurate document parsing, and to solve the problems of geometric positioning dependence, lack of cross-modal association, and insufficient logical verification in existing document parsing technologies. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a document parsing evaluation method and system based on multimodal semantic consistency.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: a document parsing evaluation method based on multimodal semantic consistency, comprising:

[0009] Acquire multimodal contract document data and preprocess it to obtain preprocessed data;

[0010] Contract element features are extracted from the preprocessed data to generate a fused feature vector;

[0011] Based on the fused feature vector, the semantic consistency of document elements is evaluated using a dual-constraint loss function and a three-level judgment mechanism to obtain cross-modal semantic consistency quantification results.

[0012] The calculation basis and semantic enhancement index are based on the cross-modal semantic consistency quantification results and integrated into a comprehensive evaluation index;

[0013] The comprehensive evaluation index and corresponding suggestions are output to guide intelligent contract analysis and algorithm optimization.

[0014] The further technical solution is as follows: the extraction of contract element features from the preprocessed data to generate a fused feature vector includes:

[0015] The preprocessed data is optimized by dynamically adjusting processing weights and using cross-modal spatiotemporal alignment techniques.

[0016] The bounding boxes at different levels in the optimized preprocessed data are labeled and analyzed. The correlation is enhanced by combining convolutional layers and Transformer encoders to obtain visual layout features.

[0017] The citation relationships of the text in the preprocessed data after optimization are analyzed, and weights are assigned according to the position information to generate a comprehensive feature vector containing text semantics and citation details, so as to obtain text semantic features;

[0018] Using the encoded position data in the optimized preprocessed data, the correct relationship between the clauses and the cited text is confirmed through logical verification, forming a final feature vector that integrates all logical relationships to obtain structural modal features;

[0019] The visual layout features, text semantic features, and structural modality features are fused to obtain a fused feature vector.

[0020] The further technical solution is as follows: the bounding boxes at different levels in the preprocessed data are labeled and analyzed, and the correlation is enhanced by combining convolutional layers and a Transformer encoder to obtain visual layout features, including:

[0021] The preprocessed data is processed and optimized using the LayoutLMv3 model, and the bounding boxes are hierarchically marked.

[0022] Based on the aforementioned tags, the preprocessed image patches and text words are fused using the native cross-modal self-attention mechanism, and the weights are adjusted according to the importance of the elements based on the hierarchical self-attention sub-layers to obtain the output of the Transformer layer.

[0023] By combining convolutional layers to capture visual details and concatenating these details with the output of the Transformer layer, a vector containing hierarchical relationships, position coordinates, and visual morphology is generated through a fully connected layer to obtain visual layout features.

[0024] Its further technical solution is as follows: Based on the fused feature vector, the semantic consistency of document elements is evaluated using a dual-constraint loss function and a three-level judgment mechanism to obtain cross-modal semantic consistency quantification results, including:

[0025] Calculate the intersection-union ratio loss of the fused feature vector relative to the standard bounding box and the semantic distance between the visual element and the text description;

[0026] The weights are dynamically adjusted based on the document type, and the total loss value is calculated by combining the intersection-union loss and the semantic distance.

[0027] A three-level judgment mechanism is executed based on the fused feature vector to obtain the judgment result;

[0028] The cross-modal semantic consistency of document elements is quantified by combining the judgment results and the total loss value to obtain the cross-modal semantic consistency quantification result.

[0029] Its further technical solution is: the three-level judgment mechanism based on the fused feature vector to obtain the judgment result includes:

[0030] Based on the fused feature vector, the consistency between element type and text annotation is compared, the consistency between chart number and text reference is verified, and the rationality of logical position is analyzed to obtain the judgment result.

[0031] Its further technical solution is as follows: the calculation basis and semantic enhancement index based on the cross-modal semantic consistency quantification results are integrated into a comprehensive evaluation index, including:

[0032] Based on the cross-modal semantic consistency quantification results, semantic enhancement metrics, reference matching rate, type semantic accuracy, and structural logic conformity are calculated.

[0033] The semantic enhancement metrics, type semantic accuracy, reference matching rate, and structural logic conformity are weighted and summed to obtain a comprehensive evaluation index.

[0034] The further technical solution is as follows: the calculation of semantic enhancement indicators, reference matching rate, type semantic accuracy, and structural logic conformity based on the cross-modal semantic consistency quantification results includes:

[0035] The original bounding box precision and recall of the preprocessed data are calculated, and the bounding box precision is corrected according to the cross-modal semantic consistency quantization result to obtain a semantic enhancement index; wherein, the semantic enhancement index includes the corrected bounding box precision and recall.

[0036] Extract and standardize all reference identifiers from the preprocessed data, extract the actual reference content from the detection results and text, unify the format, set rules and thresholds for exact matching, fault-tolerant matching and invalid reference exclusion, and calculate the reference matching rate to obtain the reference matching rate.

[0037] Define a contract core element type library for the preprocessed data and construct a semantic database between types; for each detected element, calculate the semantic distance between the type label and the labeled type label, determine the type matching score based on the semantic distance, and calculate the type semantic accuracy.

[0038] The actual reading order in the preprocessed data is analyzed, and the actual reading sequence is generated. The coherence of the reading order and the rationality of the regional association are calculated, and the overall structural logic conformity is calculated.

[0039] This invention also provides a document parsing evaluation system based on multimodal semantic consistency, comprising:

[0040] The acquisition unit is used to acquire multimodal contract document data and perform preprocessing to obtain preprocessed data;

[0041] An extraction unit is used to extract contract element features from the preprocessed data to generate a fused feature vector;

[0042] The consistency evaluation unit is used to evaluate the semantic consistency of document elements based on the fused feature vector using a dual-constraint loss function and a three-level judgment mechanism, so as to obtain the cross-modal semantic consistency quantification result.

[0043] The comprehensive index determination unit is used to calculate the basic and semantic enhancement indexes based on the cross-modal semantic consistency quantification results, and integrate them into a comprehensive evaluation index;

[0044] The output unit is used to output the comprehensive evaluation index and corresponding suggestions to guide intelligent contract analysis and algorithm optimization.

[0045] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method.

[0046] The present invention also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0047] The advantages of this invention compared to existing technologies are as follows: This invention acquires and preprocesses multimodal contract document data, extracts contract element features including visual, textual, and structural features to generate a fused feature vector, and uses a dual-constraint loss function and a three-level judgment mechanism to evaluate the semantic consistency between document elements based on the vector. It then calculates a comprehensive evaluation index covering both basic and semantic enhancement aspects, thereby achieving intelligent and accurate parsing of contract documents. This process not only overcomes the limitations of traditional parsing techniques that rely solely on geometric position for target detection and evaluation, but also compensates for the lack of cross-modal correlation information. Furthermore, by introducing logical-level verification indicators such as subject association conformity, it strengthens the ability to verify the internal logical consistency of the document. Finally, the output comprehensive evaluation index and related suggestions provide a quantitative basis for the intelligent parsing of contract documents and also point the way for algorithm optimization, significantly improving the accuracy and reliability of document parsing.

[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A schematic diagram illustrating an application scenario of the document parsing evaluation method based on multimodal semantic consistency provided in this embodiment of the invention;

[0051] Figure 2 A flowchart illustrating the document parsing evaluation method based on multimodal semantic consistency provided in an embodiment of the present invention;

[0052] Figure 3 for Figure 2 A schematic diagram of the sub-process of step S120 in the process;

[0053] Figure 4 for Figure 3 A schematic diagram of the sub-process of step S122 in the process;

[0054] Figure 5 for Figure 2 A schematic diagram of the sub-process of step S130 in the process;

[0055] Figure 6 for Figure 2 A schematic diagram of the sub-process of step S140 in the process;

[0056] Figure 7 for Figure 6 A schematic diagram of the sub-process of step S141 in the process;

[0057] Figure 8 A schematic block diagram of a document parsing evaluation system based on multimodal semantic consistency provided in an embodiment of the present invention;

[0058] Figure 9 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0061] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0062] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0063] Please see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram illustrating an application scenario of the document parsing and evaluation method based on multimodal semantic consistency provided in an embodiment of the present invention. Figure 2 This is a schematic flowchart illustrating a document parsing evaluation method based on multimodal semantic consistency provided in this invention. This method is applied in a server. The server interacts with the terminal, acquiring and preprocessing multimodal contract document data, extracting contract element features to generate a fused feature vector, and using a dual-constraint loss function and a three-level judgment mechanism to evaluate the semantic consistency of document elements. It calculates basic and semantic enhancement indices, ultimately integrating them into a comprehensive evaluation index. The entire process combines dynamically adjusted weights, cross-modal spatiotemporal alignment technology, data processing optimization using the LayoutLMv3 model, and convolutional layers and Transformer encoders to enhance correlation, thereby achieving effective capture and fusion of visual layout features, textual semantic features, and structural modality features. Furthermore, through precise matching rules and semantic distance analysis, it ensures the accuracy of citation relationships and the consistency of type labels, while evaluating the coherence of reading order and the rationality of regional associations. This solves the problems of geometric positioning dependence, missing cross-modal associations, and insufficient logical verification in existing document parsing technologies, achieving intelligent and accurate document parsing and providing specific directions for improving intelligent contract parsing and algorithm optimization.

[0064] Figure 2 This is a flowchart illustrating the document parsing and evaluation method based on multimodal semantic consistency provided in an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S150.

[0065] S110. Obtain multimodal contract document data and perform preprocessing to obtain preprocessed data.

[0066] In this embodiment, the preprocessed data refers to contract document data that has been processed through steps such as image denoising, binarization, OCR recognition, text standardization, table parsing, and logical reconstruction, resulting in a clear structure, accurate content, and multimodal information (such as text, images, and layout).

[0067] First, contract document data needs to be obtained from various sources. These documents may exist in different formats, such as PDF, images (JPG, PNG, etc.), Word documents, or other electronic formats. For physical contract documents, they need to be converted into digital format by scanning or photographing.

[0068] Next, image preprocessing is performed:

[0069] Denoising: Reduce or eliminate noise in an image to improve image quality.

[0070] Binarization: Converting a color or grayscale image into a black and white image to facilitate subsequent text recognition.

[0071] Tilt correction: Corrects document tilt caused by incorrect scanning angle.

[0072] Segmentation: Divide the page into smaller parts, such as paragraphs, tables, and images, for easier processing.

[0073] For contracts in image format, OCR technology is used to convert the text in the image into editable text. The text format is standardized, removing unnecessary spaces, line breaks, etc., to ensure consistency. The text content is segmented, and important information (such as dates, amounts, and clause numbers) is highlighted according to specific rules.

[0074] If the document contains tables, specialized algorithms are needed to identify and extract the table structure and content. This allows for the recovery of the document's internal logical relationships, such as references between clauses and document correspondences.

[0075] Create bounding boxes for each element in the document (such as headings, body text, tables, charts, etc.) and record their position coordinates. Divide the elements hierarchically based on their importance or type, such as first-level headings, second-level headings, etc. This helps with subsequent semantic understanding and feature fusion.

[0076] After the above steps, the "preprocessed data" contains clear and structured information, which not only covers the text content of the original document, but also the positional information of visual elements, tabular data, and logical relationships between various parts, providing a solid foundation for subsequent feature extraction, cross-modal semantic consistency quantification, and comprehensive evaluation.

[0077] S120. Extract contract element features from the preprocessed data to generate a fused feature vector.

[0078] In this embodiment, the fused feature vector refers to a vector that comprehensively represents each element within the contract document by combining multi-dimensional information such as visual layout features, textual semantic features, and structural modality features. This vector not only includes the basic attributes of each element (such as position and type) but also reflects the logical relationships and reference details between elements, thus providing a solid foundation for subsequent document parsing and evaluation.

[0079] To optimize data processing during document parsing, this embodiment proposes a dynamic modality priority adjustment mechanism. This mechanism flexibly adjusts the processing weights of different modalities based on the specific document type and quality characteristics. For example, when processing scanned documents with blurred images, the importance of the OCR text is increased, with its confidence weight rising from 0.5 to 0.7; while in plain text documents, due to potential structural deficiencies, the density of layout feature acquisition is enhanced. This dynamic adjustment ensures that the model can more accurately capture key information from the document under different circumstances.

[0080] In one embodiment, please refer to Figure 3 The above step S120 may include steps S121 to S125.

[0081] S121. The preprocessed data is optimized by dynamically adjusting the processing weights and using cross-modal spatiotemporal alignment techniques.

[0082] This step aims to improve data quality by intelligently adjusting the processing priorities of different modalities and ensuring precise temporal and spatial matching between image and text data. For example, when processing handwritten scanned documents, the confidence weight of OCR text is increased; for plain text documents, the acquisition density of layout features is enhanced.

[0083] Specifically, to ensure a strict match between visual elements (such as tables and charts) and their corresponding text descriptions in a document, this invention employs cross-modal spatiotemporal alignment technology. The specific method includes:

[0084] Document page number synchronization: This is achieved by numbering each page of a document and ensuring that these numbers remain consistent throughout the entire document processing process.

[0085] Coordinate mapping: Projection transformation technology is used to align the coordinates between the scanned image and the text stream, ensuring that visual elements and their corresponding text descriptions correspond precisely in the document's logical flow.

[0086] S122. Mark and analyze the bounding boxes at different levels in the optimized preprocessed data, and enhance the correlation by combining convolutional layers and Transformer encoders to obtain visual layout features.

[0087] In this embodiment, the visual layout feature refers to the use of the improved LayoutLMv3 architecture to strengthen the hierarchical relationship between elements through a self-attention mechanism, capture the nesting characteristics including "first-level headings" and "second-level headings", and finally output a vector containing hierarchical relationship, position coordinates and visual shape.

[0088] In one embodiment, please refer to Figure 4 The above step S122 may include steps S1221 to S1223.

[0089] S1221. The preprocessed data after optimization is processed using the LayoutLMv3 model, and the bounding boxes are hierarchically marked.

[0090] S1222. Based on the marker, the preprocessed image patch and text word units are fused and optimized using the native cross-modal self-attention mechanism, and the weights are adjusted according to the importance of the elements based on the hierarchical self-attention sub-layer to obtain the output of the Transformer layer.

[0091] In this embodiment, the output of the Transformer layer refers to the set of feature vectors generated after processing by a multi-head self-attention mechanism and a feedforward neural network, which contains complex relationships between elements of the input sequence. Specifically, these feature vectors not only encode the basic information of the text or image, but also integrate semantic associations and structural information between different elements, providing rich input for further feature extraction, logical verification, and final task processing. In short, the output of the Transformer layer is a feature representation that enhances semantic understanding and structural awareness.

[0092] S1223. Combine convolutional layers to capture visual details, and concatenate the visual details with the output of the Transformer layer. Generate a vector containing hierarchical relationships, position coordinates and visual shapes through a fully connected layer to obtain visual layout features.

[0093] In this embodiment, based on the improved LayoutLMv3 architecture, this invention proposes a novel visual layout feature extraction method. This method not only enhances the understanding of hierarchical relationships between elements but also integrates semantic features. The specific steps are as follows:

[0094] The system uses LayoutLMv3's existing trimodal input (image pixels, OCR text, and bounding box coordinates), and then hierarchically labels the input bounding boxes (L1 for first-level headings, L2 for second-level headings, and L0 for body text). This labeling method provides an important basis for subsequent attention allocation.

[0095] By utilizing LayoutLMv3's native cross-modal self-attention mechanism, preliminary fusion encoding of image patches and text terms is performed.

[0096] A new hierarchical self-attention sub-layer is added, assigning different weights to different types of elements: for L1 type elements (first-level headings), position codes are assigned a weight of 1.2 times; for L2 type elements (second-level headings), a weight of 1.1 times; while L0 type elements (body text) retain a weight of 1.0 times. The formula is as follows:

[0097] ,in, This is the hierarchical weight matrix. The feature dimensions are defined as follows: Attention refers to the attention mechanism. Q (Query) refers to the current input vector in the model. K (Key) refers to the feature vector of all elements in the input sequence. Q is compared with all K values ​​to determine which elements are more relevant to the current Q. V (Value) refers to the specific information vector of each element in the input sequence. When Q matches K, V is extracted, which is represented by multiplying it with the attention weights calculated from Q and K and summing the results to obtain the final attention output. T (Time Step) refers to the position index in the sequence data. For example, in a text sequence of length N, T can represent the position of the 1st to the Nth word.

[0098] The functionality of the LayoutLMv3 convolutional layer is retained to capture visual details (such as the edge features of the dashed border of the signature area and the gradient changes of the indentation of the clauses).

[0099] The output of the Transformer layer is concatenated with the bounding box contour features extracted by the convolutional layer, and then compressed into a visual layout feature vector containing hierarchical relationships, position coordinates, and visual morphology through a fully connected layer.

[0100] S123. Analyze the reference relationships of the text in the preprocessed data after optimization, assign weights according to the position information, and generate a comprehensive feature vector containing text semantics and reference details to obtain text semantic features.

[0101] In this embodiment, textual semantic features refer to the construction of a comprehensive feature vector that reflects both the text content and the details of the citations by parsing the citation relationships in the text and assigning different weights based on positional information. This helps in understanding the relationships between elements within a document, especially the accuracy of citations when referring to charts or other specific sections.

[0102] Leveraging the powerful text semantic understanding capabilities of LayoutLMv3, this invention extends the reference resolution module, strengthening the "text-visual" association. The specific process includes:

[0103] A new reference entity recognition layer is added after the vector output of the Transformer to recognize phrases such as "as shown in Figure X" and extract reference trigger words (such as "see figure" and "reference table").

[0104] By combining the bounding box position information, if the difference between the y-coordinate of the trigger word and the y-coordinate of an image element does not exceed 50 (normalized units), a correlation is considered to exist between the two. The identified citation relationship is assigned a semantic weight of 1.1 times, thereby enhancing the text's constraint characteristics on visual elements.

[0105] The processed text semantic vector is fused with the reference association features to generate the final text semantic feature vector containing reference relationships.

[0106] S124. Using the encoded position data in the optimized preprocessed data, the correct relationship between the clause and the referenced text is confirmed through logical verification, forming a final feature vector that integrates all logical relationships to obtain structural modal features.

[0107] In this embodiment, structural modal features refer to calculating the distance and sequence relationship between clauses and cited text to ensure that there is a correct logical connection between them. This step helps to verify the rationality of the document structure, such as whether the paragraph numbering is consecutive and whether the clause citations are accurate, and finally generates a feature vector that integrates all logical relationships.

[0108] A logic verification layer was added to the layout detection model to calculate the distance and sequence relationship between elements, ensuring the consistency of the document's internal logic. The specific steps are as follows:

[0109] Extract the relative position data of the bounding boxes encoded by LayoutLMv3 (such as the x / y coordinate difference between elements) to avoid redundant calculations.

[0110] The page pixel distance between the clause and the cited text (such as "Clause 3.1" and "as shown in Figure X") is calculated, and a decay factor is introduced: Distance Feature = Where the distance unit is the normalized coordinate difference, and 50 is the attenuation threshold.

[0111] Perform continuity checks on paragraph numbers (such as "1.1", "1.2") and calculate sequence consistency scores based on text semantic similarity of LayoutLMv3 (≥0.9 is considered continuous).

[0112] The features output by the Transformer are concatenated with distance and sequence features, and then incorporated into the original features through residual connections. The final output is a structural modal feature vector containing logical relationships. This process ensures the rationality and accuracy of the document's internal logical structure.

[0113] S125. The visual layout features, text semantic features, and structural modality features are fused to obtain a fused feature vector.

[0114] By organically combining visual layout features, textual semantic features, and structural modality features, a fusion feature vector is formed that comprehensively describes all elements within a contract document and their interrelationships. This vector serves as the foundation for subsequent document parsing evaluation based on multimodal semantic consistency, significantly improving the accuracy and efficiency of contract parsing.

[0115] S130. Based on the fused feature vector, the semantic consistency of document elements is evaluated using a dual-constraint loss function and a three-level judgment mechanism to obtain the cross-modal semantic consistency quantification result.

[0116] In this embodiment, the cross-modal semantic consistency quantification result refers to quantifying the degree of cross-modal semantic consistency among document elements by calculating the Intersection over Union (IoU) Loss and semantic distance between visual and textual elements in the document, combining the total loss value after dynamically adjusting the weights, and performing three-level judgments on element type matching degree, reference relationship consistency, and logical position rationality. This process not only considers the geometric positioning accuracy of elements but also deeply analyzes the semantic relationships between different modalities, thus providing a more comprehensive and accurate evaluation metric.

[0117] In one embodiment, please refer to Figure 5 The above step S130 may include steps S131 to S134.

[0118] S131. Calculate the cross-union ratio loss of the fused feature vector relative to the standard bounding box and the semantic distance between the visual element and the text description.

[0119] In this embodiment, the Intersection over Union (IoU) loss refers to comparing the detected element bounding boxes with the standard or expected bounding boxes, and measuring the degree of overlap between them by calculating the ratio of their intersection area to their union area. The smaller the IoU loss, the better the match between the predicted box and the standard box.

[0120] Semantic distance refers to the semantic similarity between visual elements (such as tables and charts in images) and their corresponding textual descriptions, calculated using models like Sentence-BERT. Semantic distance measures the semantic difference between the two; a smaller distance indicates higher semantic consistency.

[0121] S132. Dynamically adjust the weights according to the document type, and calculate the total loss value by combining the intersection-union ratio loss and the semantic distance.

[0122] In this embodiment, the total loss value refers to a comprehensive loss value calculated by combining the intersection-union ratio loss and semantic distance, and dynamically adjusting the weights of each part according to the document type (such as a layout document or a plain text document). For example, in a layout document, the layout may be more important, so the intersection-union ratio loss will be given a greater weight; while in a plain text document, more emphasis will be placed on semantic distance considerations.

[0123] S133. A three-level judgment mechanism is executed based on the fused feature vector to obtain the judgment result.

[0124] In this embodiment, the judgment result refers to performing the following three levels of judgments in sequence through the analysis of the fused feature vector:

[0125] First-level judgment: Compare the consistency between the element type and the text annotation, such as the matching degree between "table" and the text annotation "Table".

[0126] Second-level judgment: Verify the consistency between the chart number and the text reference, ensuring the consistency between the chart number such as "Figure X-X" and the text reference such as "See Figure X-X".

[0127] Third-level judgment: Analyze the rationality of the logical position, for example, check whether the "abstract" element appears reasonably in the first paragraph of the document.

[0128] Based on the fused feature vector, compare the consistency between the element type and the text annotation, verify the consistency between the chart number and the text reference, and analyze the rationality of the logical position to obtain the judgment result.

[0129] S134. Quantify the cross-modal semantic consistency of the document elements based on the judgment result and the total loss value to obtain the cross-modal semantic consistency quantification result.

[0130] Finally, comprehensively quantify the cross-modal semantic consistency of the document elements by integrating the results of all the above steps, including the intersection over union loss, semantic distance, dynamically adjusted total loss value, and the result of the third-level judgment. This quantification result can provide an objective basis for the quality of document parsing and guide subsequent data processing and algorithm optimization work.

[0131] In this embodiment, the dual-constraint loss function aims to comprehensively evaluate the cross-modal consistency of document elements by combining geometric position information and multi-modal semantic associations. Its specific form is as follows: ; where represents the intersection over union between the element bounding box and the standard box loss, which is used to measure the accuracy of visual elements in spatial position, represents the semantic distance between the visual element and the text description, calculated based on the Sentence-BERT model, reflecting the consistency between the two at the semantic level, is a dynamic weight parameter, which is adjusted according to different document types. For layout documents, is set to 0.4; for pure text documents, is set to 0.6. This means that in layout documents, more attention is paid to the consistency of layout, while in pure text documents, more attention is paid to semantic consistency.

[0132] To further ensure semantic consistency among document elements, a three-level judgment mechanism is introduced, which considers three aspects: element type matching degree, reference relationship consistency, and logical position rationality.

[0133] First-level judgment: Element type matching degree;

[0134] This level of judgment focuses on whether the element types in the comparison document match the corresponding text annotations. For example, the visual element "table" should correspond to the text annotation "Table". The specific implementation method uses cosine similarity to calculate the semantic distance between the two elements and sets a threshold to determine whether they match. If the calculated cosine distance is below a certain preset threshold, the two are considered semantically matched; otherwise, they are considered mismatched.

[0135] Second-level judgment: Consistency of reference relationships;

[0136] This level focuses on the consistency between figure / table numbers and text references within the document. For example, the figure / table number "Figure XX" should correspond exactly to the text reference "See Figure XX". This step typically involves string matching techniques to check whether the figure / table number and the referenced text are completely identical or very close. Considering that slight textual differences may exist in practical applications (such as different punctuation), methods such as edit distance can be used to tolerate these minor differences, thereby improving the accuracy of the matching.

[0137] Level 3 judgment: Logical position rationality;

[0138] The final level of evaluation focuses on analyzing the logical placement of document elements. For example, elements like "Summary" should theoretically appear at the beginning of the document. Therefore, the system will calculate the probability of the "Summary" element appearing in the first paragraph as one of the evaluation criteria. In addition, other logical order requirements can be considered, such as ascending order of clause numbers, to ensure that the document structure conforms to conventional reading habits and standards.

[0139] In summary, the dual-constraint loss function and its accompanying document-level semantic consistency judgment mechanism together constitute a comprehensive and meticulous document parsing framework. It not only considers the accuracy of element spatial layout but also delves into the semantic consistency between different modalities, effectively improving the accuracy and reliability of document parsing. This method is particularly suitable for scenarios requiring highly accurate parsing, such as legal contracts and academic papers.

[0140] S140. Calculate the basis and semantic enhancement index based on the cross-modal semantic consistency quantification results, and integrate them into a comprehensive evaluation index.

[0141] In this embodiment, the comprehensive evaluation index is a numerical value that comprehensively measures the quality of document parsing, obtained by weighted summation of semantic enhancement indicators, reference matching rate, type semantic accuracy, and structural logic conformity.

[0142] In one embodiment, please refer to Figure 6 The above step S140 may include steps S141 to S142.

[0143] S141. Calculate the semantic enhancement index, reference matching rate, type semantic accuracy, and structural logic conformity based on the cross-modal semantic consistency quantification results.

[0144] In one embodiment, please refer to Figure 7 The above step S141 may include steps S1411 to S1414.

[0145] S1411. Calculate the original bounding box precision and recall of the preprocessed data, and correct the bounding box precision based on the cross-modal semantic consistency quantization result to obtain a semantic enhancement index; wherein, the semantic enhancement index includes the corrected bounding box precision and recall.

[0146] In this embodiment, the semantic enhancement metric refers to the bounding box accuracy and recall after correction based on the cross-modal semantic consistency quantization result, which is used to reflect the degree of consistency between the detected elements in geometric position and semantic information.

[0147] Specifically, preserve bounding box accuracy Recall rate Traditional metrics, such as semantic consistency scores, are corrected: ; ;in The cross-modal semantic consistency score (range 0-1) for document elements is obtained by calculating the cosine similarity between the detection result and the labeled content; the closer to 1, the higher the semantic consistency. The coefficient weights can be adjusted according to the scenario.

[0148] S1412. Extract and standardize all reference identifiers of the preprocessed data, extract the actual reference content from the detection results and text, unify the format, set rules and thresholds for exact matching, fault-tolerant matching and invalid reference exclusion, and calculate the reference matching rate to obtain the reference matching rate.

[0149] In this embodiment, the reference matching rate refers to the ratio of matching between visual element numbers and text references, and evaluates the accuracy and completeness of the correlation between the two.

[0150] The Reference Match Rate (RMR) measures the proportion of visual element IDs that match text references. First, reference identifiers are extracted from the detection results, and all actual references are extracted from the text. These references are then standardized by removing redundant modifiers and unifying their format. Next, matching rules and thresholds are set, including exact matching, error-tolerant matching, and exclusion of invalid references. Finally, the reference match rate is calculated using a specific formula.

[0151] Specifically, citation identifiers (such as "see Section 3.1", "Document 2", "refer to Clause 5.2") are extracted from the detection results, and all actual cited content (such as "Section 3.1 Payment Method" and "Document 2: Service List" in the table of contents) are extracted from the text modality. Here, "detection results" refers to information or elements automatically extracted from documents (such as contracts, agreements, etc.) using certain algorithms or technologies. This information or elements may include, but are not limited to:

[0152] Bounding box position: For images or scanned documents, the position and size of a specific region identified by object detection algorithms, such as the position of text blocks, tables, signature areas, etc.

[0153] Text content extraction: The extraction of text content from a document using OCR (Optical Character Recognition) technology or other text extraction methods. This may include clause numbers, specific clause content, citation markers (such as "see Clause 3.1"), document names, etc.

[0154] Element type classification: Classify the identified text or graphic elements, such as distinguishing which are signature areas, which are risk warning boxes, and which are main clauses.

[0155] Citation matching: This function attempts to match citations in the document with actual existing clauses or files to verify the accuracy of the citations.

[0156] Semantic analysis results: The semantic similarity between the extracted content and the labeled content is calculated to assess the degree of consistency between the two. The "cross-modal semantic consistency score" mentioned here is based on this concept and is used to measure the semantic matching degree between the detected content and the expected content.

[0157] In summary, "detection results" refer to the processed set of information about document content, used for subsequent analysis, evaluation, and improvement of the algorithm's accuracy and effectiveness. During this process, the detection results are compared with pre-defined standards or manually labeled results to calculate various metrics (such as P'_box, RMR, TSA, SLC, etc.), thereby evaluating the algorithm's performance.

[0158] Standardize the citation format (e.g., convert “3.1 Clause”, “3.1”, and “3.1” to “3.1 Clause”; convert “Document 2” and “Appendix 2” to “Document 2”), remove redundant modifiers (e.g., “see”, “refer to”), and retain core identifiers (e.g., “3.1 Clause”, “Document 2”).

[0159] A perfect match is defined as a detection identifier that is completely identical to the text reference (e.g., "Article 3.1" and "Article 3.1"). For minor formatting differences (e.g., "Article 3.1.1" misdetected as "Article 3.1"), similarity is calculated using the Levenshtein Distance. An edit distance ≤ 1 is considered a partial match (weight 0.6). If the detection identifier has no corresponding reference in the text (e.g., "Article 9.9" does not actually exist), it is considered invalid and not included in the matching calculation; the calculation formula is: Where: n is the total number of citations in the text modality; The matching value for the i-th reference (complete match = 1, partial match = 0.6, no match = 0); The weight of the i-th reference (core references such as "reference to liability for breach of contract clause" have a weight of 1.2, and auxiliary references such as "reference to document description" have a weight of 0.8).

[0160] S1413. Define the contract core element type library of the preprocessed data and construct a semantic database between types; for each detected element, calculate the semantic distance between the type label and the labeled type label, determine the type matching score based on the semantic distance, and calculate the type semantic accuracy.

[0161] In this embodiment, the type semantic accuracy refers to the semantic distance score between the detected element type and the labeled type, which measures the accuracy of different type element recognition and their semantic fit.

[0162] This metric evaluates the semantic fit between the detected element type and the multimodal annotation. First, a core contract element type library is defined, and a contract type semantic database is constructed to calculate the semantic distance between types. For each detected element, the semantic distance *d* between its type label and the annotated type label is calculated, and the accuracy is calculated based on this distance.

[0163] First, define a core element type library for contracts, including 12 standard types such as "Party A's Signature Area", "Party B's Signature Area", "Indivisible Core Clauses", "Mergeable Supplementary Clauses", "Risk Warning Box", and "Document Description".

[0164] Construct a semantic database for contract types: Calculate the semantic distance between types based on the ontology in the legal domain (such as the clause classification of the "Contract Law") (e.g., the semantic distance between "Party A's signature area" and "Party B's signature area" = 0.5, and the semantic distance between "Signature area" and "Remarks box" = 2.3).

[0165] For each detected element, extract its type label (such as "Party B's signature area") and the labeled type label (such as "Party A's signature area") to obtain the semantic distance d between the two.

[0166] Accuracy calculation and weight adjustment, specifically using the calculation formula ; where: k is the total number of detected elements; is the type matching score of the j-th element: if d ≤ 1 (semantically similar), = 1; if 1 < d ≤ 2 (semantically related but different), = 0.5; if d > 2 (semantically irrelevant), = 0; is the weight of the j-th element: core elements (such as signature area, core clauses) , auxiliary elements (such as remarks box, page number) .

[0167] S1414. Analyze the actual reading order in the preprocessed data, generate an actual reading sequence, calculate the coherence of the reading order and the rationality of regional association, and calculate the overall structural logic compliance degree.

[0168] In this embodiment, the structural logic compliance degree refers to evaluating the matching degree between the layout of document elements and the logical structure by analyzing the actual reading order and the rationality of regional association, ensuring the internal logic coherence of the document.

[0169] This index examines the matching degree between the element layout and the document logical structure. First, evaluate the coherence of the reading order. Based on the reading habits of a large number of users, summarize the conventional reading order, and calculate the length of the longest common subsequence between the actual reading sequence and the reference sequence to evaluate the matching degree. Second, consider the rationality of regional association, that is, associated elements should be adjacent in the reading order. Calculate the overall structural logic compliance degree according to these factors: , where is the reading order coherence score, is the regional association rationality score.

[0170] Specifically, the conventional reading order of the contract document is: title - directory - signatory information - text clauses (in increasing number) - signature area - document description, which is the natural order summarized based on the reading habits of a large number of users and the contract format specifications.

[0171] The detected elements are extracted and sorted according to the actual page layout and reading order (simulating the human eye's scanning path, prioritizing vertical coordinates and then horizontal coordinates for elements of the same height), generating an "actual reading sequence". Using the normal reading order as the baseline sequence, the length of the longest common subsequence (LCS) between the actual sequence and the baseline sequence is calculated. The matching formula is: For example, the standard reading order contains 6 key elements. In the actual reading sequence, the order of "Main Text Clauses" and "Contractor Information" is reversed, while the order of other elements remains the same. The LCS length is 5. =5 / 6≈0.83;

[0172] The contract contains "content-related element groups" (such as clause number-corresponding clause content, document name-document details box), and their layout must meet the requirement of "readability continuity," meaning that related elements should be adjacent in the reading order. The scoring formula for a single related group is: If related elements are completely disjointed in reading order (e.g., "Clause No. 5.2" appears after "Clause No. 3.1", skipping "3.2-5.1" in between), the score for that group will be halved. Overall regional association rationality: (n is the total number of content-related element groups, with core groups such as "clause number - content" having a weight of ×1.2); weights are assigned according to the importance of reading experience, using the following formula: .

[0173] S142. The semantic enhancement index, type semantic accuracy, reference matching rate, and structural logic conformity are weighted and summed to obtain a comprehensive evaluation index.

[0174] The comprehensive evaluation index is calculated using the analytic hierarchy process (AHP) and aims to comprehensively measure the quality of document parsing. The specific formula is as follows: The weights of each indicator here reflect their relative importance in the overall evaluation and can be adjusted according to the actual situation. This multi-level, multi-dimensional evaluation system helps to more accurately reflect the overall quality and effectiveness of document parsing.

[0175] S150. Output the comprehensive evaluation index and corresponding suggestions to guide intelligent contract analysis and algorithm optimization.

[0176] The comprehensive evaluation index refers to a comprehensive assessment of the performance of a contract intelligence parsing system based on multiple different indicators. These indicators include, but are not limited to, the following aspects:

[0177] Accuracy: Measures the accuracy with which a system identifies and extracts information.

[0178] Recall: The percentage of all relevant information that the system correctly identifies.

[0179] Precision: The proportion of truly relevant information among all the information identified by the system.

[0180] Cross-modal semantic consistency score: measures the semantic similarity between text extracted from a document and the expected content.

[0181] Risk Mention Recognition Rate (RMR): The percentage of identified risk warning boxes out of the total number of risk warning boxes.

[0182] Term Section Accuracy (TSA): The accuracy rate at which the system correctly identifies and locates a specific term.

[0183] Semantic Logic Coherence (SLC): This checks whether the extracted information retains the logical structure and meaning of the original text.

[0184] Based on the aforementioned comprehensive evaluation index, targeted improvement suggestions are proposed to guide the further development and algorithm optimization of intelligent contract parsing technology. These suggestions can be divided into several aspects:

[0185] Data augmentation: If it is found that certain types of contracts or clauses are not well identified, the generalization ability of the model can be improved by adding more diverse training data.

[0186] Algorithm tuning: For specific problems (such as low precision or recall), consider using more advanced machine learning or deep learning algorithms, or fine-tune the parameters of existing algorithms.

[0187] Feature engineering: By analyzing which features are most critical to improving model performance, the feature selection and construction process can be further optimized.

[0188] User feedback mechanism: Establish an effective user feedback channel to collect problems and needs in real-world usage scenarios, which will serve as an important basis for subsequent improvements.

[0189] Continuous monitoring and iteration: Regularly reassess system performance and continuously iterate and optimize based on the latest business needs and technological advancements.

[0190] In this way, the S150 step can not only quantify the current performance level of the intelligent contract parsing system, but also provide clear directions and strategies for future technological improvements, ensuring that the system can continuously improve its applicability and reliability in practice.

[0191] In this embodiment, semantic connections are established between text and image information to achieve a deeper understanding of the contract content. Based on the contract type and the importance of its clauses, the weights of each evaluation indicator are flexibly adjusted to ensure that the evaluation results truly reflect actual application needs. Multiple evaluation indicators are integrated, and a weighted summation method is used to generate a Comprehensive Evaluation Index (CEI), providing a quantitative evaluation standard for the overall effectiveness of contract parsing. An innovative "layout-semantic" dual-constraint mechanism effectively solves the geometric positioning dependency problem and fills the gap in cross-modal association. The introduction of a multimodal information evaluation dimension breaks through the traditional limitations of geometric matching; a dynamic weight adjustment strategy is designed to enhance the relevance and accuracy of the evaluation. While improving the accuracy of contract parsing, legal risks are significantly reduced, making it particularly suitable for scenarios such as sales contracts and labor contracts, and contributing to the intelligent and compliant development of contract processing.

[0192] In traditional document parsing techniques, the evaluation of object detection often heavily relies on the matching degree of geometric positions. However, this method ignores the semantic relationship between text content and other visual elements (such as charts, images, etc.), which may result in inaccurate or incomplete parsing results.

[0193] In this embodiment, a document cross-modal semantic consistency quantification model is introduced, utilizing a dual-constraint loss function calculation unit and a three-level judgment unit to evaluate the semantic consistency between data from different modalities. This model not only considers geometric location information, but more importantly, it analyzes the semantic relationships between text and images. For example, in a contract document, the textual description of a clause and its associated illustration should be consistent. Relying solely on geometric location matching might overlook this deeper semantic relationship. This invention, by constructing a "layout-semantic" dual-constraint mechanism, ensures that both factors are considered simultaneously during the parsing process, thereby improving the accuracy of the parsing.

[0194] Traditional document parsing methods typically focus on processing single-modal data, such as only text content or only image information, lacking the ability to uncover potential relationships between multiple modalities. This limits the depth and breadth of the overall understanding of the document.

[0195] In this embodiment, the proposed multimodal data collaborative processing mechanism for documents includes a visual layout feature extraction unit, a text semantic extraction unit, and a structural modality extraction unit, enabling comprehensive feature extraction of document elements from multiple dimensions (visual, textual, and structural). This mechanism not only identifies information from each individual modality but also reveals hidden correlations between different modalities. For example, in a contract document, the textual description of a clause may require specific charts to fully express its meaning. The method of this invention effectively captures these cross-modal correlations, overcoming the shortcomings of single-modal processing methods and enhancing the comprehensiveness and accuracy of document parsing.

[0196] To further improve the quality of document parsing, especially for professional documents such as contracts, this invention introduces a series of contract-specific evaluation indicators, such as subject association compliance, and incorporates a hierarchical evaluation mechanism. These unique evaluation indicators go beyond surface content matching, delving into the contract's logical level to examine the inherent consistency and rationality among various clauses. For example, the "subject association compliance" indicator measures whether the parties involved in the contract and their interactions are reasonable and consistent. Furthermore, the hierarchical evaluation mechanism allows for setting different weights based on the requirements of different levels, making the final Comprehensive Evaluation Index (CEI) more closely reflect the needs of real-world application scenarios.

[0197] Through the above measures, the method of this embodiment not only solves the problem of relying on geometric position for target detection evaluation in the prior art, but also fills the gap of missing cross-modal association and enhances the ability of document logic verification, providing users with a more accurate and reliable document parsing solution.

[0198] The aforementioned document parsing evaluation method based on multimodal semantic consistency acquires and preprocesses multimodal contract document data, extracts contract element features including visual, textual, and structural elements to generate a fused feature vector, and uses a dual-constraint loss function and a three-level judgment mechanism to evaluate the semantic consistency between document elements based on the vector. It then calculates a comprehensive evaluation index covering both basic and semantic enhancement aspects, thereby achieving intelligent and accurate parsing of contract documents. This process not only overcomes the limitations of traditional parsing techniques that rely solely on geometric position for target detection evaluation but also compensates for the lack of cross-modal correlation information. Furthermore, by introducing logical-level verification indicators such as subject association conformity, it strengthens the ability to verify the internal logical consistency of the document. Finally, the output comprehensive evaluation index and related suggestions provide a quantitative basis for intelligent contract document parsing and also point the way for algorithm optimization, significantly improving the accuracy and reliability of document parsing.

[0199] Figure 8This is a schematic block diagram of a document parsing and evaluation system 300 based on multimodal semantic consistency provided in an embodiment of the present invention. Figure 8 As shown, corresponding to the above-described document parsing evaluation method based on multimodal semantic consistency, the present invention also provides a document parsing evaluation system 300 based on multimodal semantic consistency. This document parsing evaluation system 300 includes a unit for executing the above-described document parsing evaluation method based on multimodal semantic consistency, and the system can be configured in a server. Specifically, please refer to... Figure 8 The document parsing evaluation system 300 based on multimodal semantic consistency includes an acquisition unit 301, an extraction unit 302, a consistency evaluation unit 303, a comprehensive index determination unit 304, and an output unit 305.

[0200] The acquisition unit 301 is used to acquire multimodal contract document data and perform preprocessing to obtain preprocessed data; the extraction unit 302 is used to extract contract element features from the preprocessed data to generate a fused feature vector; the consistency evaluation unit 303 is used to evaluate the semantic consistency of document elements based on the fused feature vector using a dual-constraint loss function and a three-level judgment mechanism to obtain a cross-modal semantic consistency quantification result; the comprehensive index determination unit 304 is used to calculate the basis and semantic enhancement index based on the cross-modal semantic consistency quantification result and integrate them into a comprehensive evaluation index; the output unit 305 is used to output the comprehensive evaluation index and corresponding related suggestions to guide intelligent contract parsing and algorithm optimization.

[0201] In one embodiment, the extraction unit 302 includes:

[0202] The system comprises the following subunits: an optimization subunit, which optimizes the preprocessed data by dynamically adjusting processing weights and using cross-modal spatiotemporal alignment techniques; a visual feature extraction subunit, which labels and analyzes bounding boxes at different levels in the optimized preprocessed data, and enhances correlation by combining convolutional layers and a Transformer encoder to obtain visual layout features; a text feature extraction subunit, which parses the citation relationships of text in the optimized preprocessed data, assigns weights based on positional information, and generates a comprehensive feature vector containing text semantics and citation details to obtain text semantic features; a structural feature extraction subunit, which utilizes the encoded positional data in the optimized preprocessed data, verifies the correct relationship between clauses and citation text through logical checks, and forms a final feature vector integrating all logical relationships to obtain structural modal features; and a fusion subunit, which fuses the visual layout features, text semantic features, and structural modal features to obtain a fused feature vector.

[0203] In one embodiment, the visual feature extraction subunit includes:

[0204] The labeling subunit is used to hierarchically label the bounding boxes by processing and optimizing the preprocessed data using the LayoutLMv3 model. The output subunit is used to fuse the optimized preprocessed image patches and text words based on the labels using the native cross-modal self-attention mechanism, and adjust the weights according to the importance of the elements based on the hierarchical self-attention sublayer to obtain the output of the Transformer layer. The fully connected subunit is used to capture visual details by combining convolutional layers, and concatenate the visual details with the output of the Transformer layer, and generate a vector containing hierarchical relationships, position coordinates and visual shapes through the fully connected layer to obtain visual layout features.

[0205] In one embodiment, the consistency evaluation unit 303 includes:

[0206] The distance calculation subunit is used to calculate the intersection-union ratio (IU / U) loss of the fused feature vector relative to the standard bounding box and the semantic distance between the visual element and the text description; the total loss calculation subunit is used to dynamically adjust the weights according to the document type and calculate the total loss value by combining the IU / U and the semantic distance; the judgment subunit is used to perform a three-level judgment mechanism based on the fused feature vector to obtain the judgment result; the quantization subunit is used to quantify the cross-modal semantic consistency of document elements by combining the judgment result and the total loss value to obtain the cross-modal semantic consistency quantization result.

[0207] In one embodiment, the judgment subunit is used to compare the consistency between element type and text annotation based on the fused feature vector, verify the consistency between chart number and text reference, and analyze the rationality of logical position to obtain a judgment result.

[0208] In one embodiment, the comprehensive index determination unit 304 includes:

[0209] The index calculation subunit is used to calculate the semantic enhancement index, reference matching rate, type semantic accuracy, and structural logic conformance based on the cross-modal semantic consistency quantification result; the weighted summation subunit is used to perform weighted summation of the semantic enhancement index, type semantic accuracy, reference matching rate, and structural logic conformance to obtain a comprehensive evaluation index.

[0210] In one embodiment, the index calculation subunit includes:

[0211] The first calculation module is used to calculate the original bounding box precision and recall of the preprocessed data, and to correct the bounding box precision based on the cross-modal semantic consistency quantification results to obtain a semantic enhancement index; wherein the semantic enhancement index includes the corrected bounding box precision and recall. The second calculation module is used to extract and standardize all reference identifiers of the preprocessed data, extract the actual reference content from the detection results and text, unify the format, set rules and thresholds for exact matching, fault-tolerant matching, and exclusion of invalid references, and calculate the reference matching rate to obtain the reference matching rate. The third calculation module is used to define the contract core element type library of the preprocessed data and construct a semantic database between types; for each detected element, calculate the semantic distance between the type label and the labeled type label, determine the type matching score based on the semantic distance, and calculate the type semantic precision. The fourth calculation module is used to analyze the actual reading order in the preprocessed data, generate the actual reading sequence, calculate the reading order coherence and regional correlation rationality, and calculate the overall structural logic conformity.

[0212] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the document parsing evaluation system 300 based on multimodal semantic consistency and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0213] The aforementioned document parsing and evaluation system 300 based on multimodal semantic consistency can be implemented as a computer program, which can be used in, for example... Figure 9 It runs on the computer device shown.

[0214] Please see Figure 9 , Figure 9 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0215] See Figure 9 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0216] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a document parsing evaluation method based on multimodal semantic consistency.

[0217] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0218] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a document parsing evaluation method based on multimodal semantic consistency.

[0219] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0220] The processor 502 is used to run the computer program 5032 stored in the memory to implement all the steps of the document parsing evaluation method based on multimodal semantic consistency.

[0221] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0222] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0223] Therefore, the present invention also provides a storage medium. This storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform all steps of the document parsing evaluation method based on multimodal semantic consistency.

[0224] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0225] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0226] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0227] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the system of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0228] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0229] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A document parsing evaluation method based on multimodal semantic consistency, characterized in that, include: Acquire multimodal contract document data and preprocess it to obtain preprocessed data; Contract element features are extracted from the preprocessed data to generate a fused feature vector; Based on the fused feature vector, the semantic consistency of document elements is evaluated using a dual-constraint loss function and a three-level judgment mechanism to obtain cross-modal semantic consistency quantification results. The calculation basis and semantic enhancement index are based on the cross-modal semantic consistency quantification results and integrated into a comprehensive evaluation index; Output the comprehensive evaluation index and corresponding suggestions to guide intelligent contract analysis and algorithm optimization; The process of evaluating the semantic consistency of document elements based on the fused feature vector using a dual-constraint loss function and a three-level judgment mechanism to obtain cross-modal semantic consistency quantification results includes: Calculate the intersection-union ratio loss of the fused feature vector relative to the standard bounding box and the semantic distance between the visual element and the text description; The weights are dynamically adjusted based on the document type, and the total loss value is calculated by combining the intersection-union loss and the semantic distance. A three-level judgment mechanism is executed based on the fused feature vector to obtain the judgment result; The cross-modal semantic consistency of document elements is quantified by combining the judgment results and the total loss value to obtain the cross-modal semantic consistency quantification result; The three-level judgment mechanism based on the fused feature vector to obtain the judgment result includes: Based on the fused feature vector, the consistency between element type and text annotation is compared, the consistency between chart number and text reference is verified, and the rationality of logical position is analyzed to obtain the judgment result.

2. The document parsing evaluation method based on multimodal semantic consistency according to claim 1, characterized in that, The step of extracting contract element features from the preprocessed data to generate a fused feature vector includes: The preprocessed data is optimized by dynamically adjusting processing weights and using cross-modal spatiotemporal alignment techniques. The bounding boxes at different levels in the optimized preprocessed data are labeled and analyzed. The correlation is enhanced by combining convolutional layers and Transformer encoders to obtain visual layout features. The citation relationships of the text in the preprocessed data after optimization are analyzed, and weights are assigned according to the position information to generate a comprehensive feature vector containing text semantics and citation details, so as to obtain text semantic features; Using the encoded position data in the optimized preprocessed data, the correct relationship between the clauses and the cited text is confirmed through logical verification, forming a final feature vector that integrates all logical relationships to obtain structural modal features; The visual layout features, text semantic features, and structural modality features are fused to obtain a fused feature vector.

3. The document parsing evaluation method based on multimodal semantic consistency according to claim 2, characterized in that, The bounding boxes at different levels in the preprocessed data are labeled and analyzed, and the correlation is enhanced by combining convolutional layers and a Transformer encoder to obtain visual layout features, including: The preprocessed data is processed and optimized using the LayoutLMv3 model, and the bounding boxes are hierarchically marked. Based on the aforementioned tags, the preprocessed image patches and text words are fused using the native cross-modal self-attention mechanism, and the weights are adjusted according to the importance of the elements based on the hierarchical self-attention sub-layers to obtain the output of the Transformer layer. By combining convolutional layers to capture visual details and concatenating these details with the output of the Transformer layer, a vector containing hierarchical relationships, position coordinates, and visual morphology is generated through a fully connected layer to obtain visual layout features.

4. The document parsing evaluation method based on multimodal semantic consistency according to claim 1, characterized in that, The calculation basis and semantic enhancement index based on the cross-modal semantic consistency quantification results are integrated into a comprehensive evaluation index, including: Based on the cross-modal semantic consistency quantification results, semantic enhancement metrics, reference matching rate, type semantic accuracy, and structural logic conformity are calculated. The semantic enhancement metrics, type semantic accuracy, reference matching rate, and structural logic conformity are weighted and summed to obtain a comprehensive evaluation index.

5. The document parsing evaluation method based on multimodal semantic consistency according to claim 4, characterized in that, The calculation of semantic enhancement metrics, reference matching rate, type semantic accuracy, and structural logic conformance based on the cross-modal semantic consistency quantification results includes: The original bounding box precision and recall of the preprocessed data are calculated, and the bounding box precision is corrected according to the cross-modal semantic consistency quantization result to obtain a semantic enhancement index; wherein, the semantic enhancement index includes the corrected bounding box precision and recall. Extract and standardize all reference identifiers from the preprocessed data, extract the actual reference content from the detection results and text, unify the format, set rules and thresholds for exact matching, fault-tolerant matching and invalid reference exclusion, and calculate the reference matching rate to obtain the reference matching rate. Define a contract core element type library for the preprocessed data and construct a semantic database between types; for each detected element, calculate the semantic distance between the type label and the labeled type label, determine the type matching score based on the semantic distance, and calculate the type semantic accuracy. The actual reading order in the preprocessed data is analyzed, and the actual reading sequence is generated. The coherence of the reading order and the rationality of the regional association are calculated, and the overall structural logic conformity is calculated.

6. A document parsing and evaluation device based on multimodal semantic consistency, characterized in that, The document parsing evaluation method based on multimodal semantic consistency as described in any one of claims 1 to 5 includes: The acquisition unit is used to acquire multimodal contract document data and perform preprocessing to obtain preprocessed data; An extraction unit is used to extract contract element features from the preprocessed data to generate a fused feature vector; The consistency evaluation unit is used to evaluate the semantic consistency of document elements based on the fused feature vector using a dual-constraint loss function and a three-level judgment mechanism, so as to obtain the cross-modal semantic consistency quantification result. The comprehensive index determination unit is used to calculate the basic and semantic enhancement indexes based on the cross-modal semantic consistency quantification results, and integrate them into a comprehensive evaluation index; The output unit is used to output the comprehensive evaluation index and corresponding suggestions to guide intelligent contract analysis and algorithm optimization.

7. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.

Images