A multi-modal document review method, system, and storage medium

Abstract

The application provides a multi-modal document review method, system and storage medium, comprising: dynamically determining a processing unit participating in review and a task execution plan based on distribution information of each modal content in a document to be reviewed; identifying a cross-modal association mark when analyzing text content, sending a cross-modal semantic verification request to a target modal processing unit, and receiving a returned measured semantic relationship and confidence; comparing the measured semantic relationship with a text description, and in response to detecting semantic inconsistency, calculating a comprehensive arbitration score based on multi-dimensional weighted arbitration, and outputting a graded review conclusion. The application realizes automatic verification and intelligent arbitration of cross-modal logical consistency, and improves review efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of intelligent document processing technology, specifically to a multimodal document review method, system, and storage medium. Background Technology

[0002] Currently, in fields such as engineering construction, financial auditing, medical registration and application, and legal compliance, complex business documents generally contain information in multiple modalities, including text, images, tables, and formulas. Existing technologies typically employ rule-based single-modal checks, independent AI model checks, or pipeline-style multi-step reviews. However, when processing multimodal documents, existing solutions often treat the content of different modalities in isolation, lacking an effective verification mechanism for cross-modal logical consistency between text descriptions, figure annotations, data tables, and technical parameter formulas. This makes it difficult to automatically identify deep logical errors in documents, such as "inconsistencies between text and images" and "conflicts between text and tables."

[0003] Furthermore, when different detection sources (such as rule engines and AI models) make inconsistent judgments on the same content, the lack of an automated intelligent arbitration mechanism leads to a high false positive rate or requires extensive manual review. Meanwhile, existing review processes typically employ fixed pipelines, failing to dynamically adapt to document types (such as plain text contracts or technical documents containing numerous drawings), resulting in wasted computational resources or missed reviews. Therefore, there is an urgent need for a multimodal document review technology solution that can address the shortcomings in cross-modal information collaborative processing and intelligent decision-making capabilities. Summary of the Invention

[0004] To address the technical problem of insufficient cross-modal information collaborative processing and intelligent decision-making capabilities in existing technologies for multimodal document review, this application proposes a multimodal document review method, system, and storage medium to achieve automatic identification of cross-modal semantic contradictions in the document to be reviewed, intelligent determination of conflict results, and on-demand scheduling of review resources.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal document review method includes: dynamically determining the processing units and task execution plans involved in the review based on the distribution information of each modality in the document to be reviewed; identifying cross-modal association markers during the parsing of the text content of the document to be reviewed; in response to the cross-modal association markers, sending a cross-modal semantic verification request carrying entity constraint information and expected semantic relationships to the corresponding target modal processing unit, and receiving a verification response returned by the target modal processing unit containing measured semantic relationships and confidence levels; comparing the measured semantic relationships in the verification response and the relevant semantic records in the shared semantic storage unit with the corresponding descriptions in the text content; in response to the detection of semantic inconsistencies, calculating a comprehensive arbitration score based on multidimensional weighted arbitration, and outputting a graded review conclusion based on the comprehensive arbitration score.

[0006] Preferably, the step of dynamically determining the processing units and task execution plan participating in the review based on the distribution information of each modal content in the document to be reviewed includes: counting the number of elements of each modal content in the document to be reviewed and the distribution density of cross-modal association tags, and generating a weight proportion vector of each modal content; determining the activation state and execution priority of each processing unit based on the weight proportion vector; the task execution plan is represented in the form of a dependency graph between task units, wherein task units without dependencies are executed in parallel, and task units with dependencies are executed sequentially according to the topological sorting order.

[0007] Preferably, the data structure of the cross-modal semantic verification request and the verification response includes: the cross-modal semantic verification request includes: a request identifier; an anchor type field, the value of which is at least one enumerated value among image type, table type, and formula type; an anchor location identifier; the original text sentence that triggers the cross-modal association marker; an entity list, each entity record in the entity list including entity name, entity value, and unit of measurement; an expected relationship list, each record in the expected relationship list including subject entity name, relationship type, and object entity name; the verification response includes: a request identifier corresponding to the cross-modal semantic verification request; an anchor location identifier; a measured entity list, each record in the measured entity list including entity name, measured value, and corresponding confidence level; a measured relationship list; and a conflict flag list, each conflict record in the conflict flag list including conflict entity name, expected value, measured value, and a conflict severity field, the value of which includes high severity, medium severity, and low severity.

[0008] Preferably, in the verification response returned by the target modality processing unit, if the confidence level of any entity in the measured entity list is lower than the confidence level threshold, an additional cross-modal semantic verification request is generated and routed to the target modality processing unit; the additional cross-modal semantic verification request carries parameters to expand the recognition range or parameters specifying the feature type; the routing process executes at most a preset number of iterations, and the measured semantic relationship in the final verification response is used as the input for subsequent comparison steps.

[0009] Preferably, the calculation of the comprehensive arbitration score based on multidimensional weighted arbitration includes: calculating the comprehensive arbitration score S according to S = w1·Crule + w2·Cmodel + w3·Ccross; where Crule is the rule constraint confidence, representing the hard constraint judgment result of the rule base on the detected semantically inconsistent content, and its value is 0 or 1; Cmodel is the model inference confidence, representing the degree of confidence of the natural language processing module or visual feature extraction module in the inference of the content, and its value range is greater than or equal to 0 and less than or equal to 1; Ccross is the cross-modal corroboration weight, which takes a value in the range of greater than or equal to 0 and less than or equal to 1 when there is positive corroboration, and takes a value in the range of greater than or equal to negative 1 and less than 0 when there is negative conflict. The values ​​of w1, w2, and w3 are dynamic weight coefficients, read from the weight configuration library according to the type of the document to be reviewed; based on the comprehensive arbitration score S, the graded review conclusion is output according to the following rules: when S is not lower than the first preset threshold, it is judged as correct and the error report is rejected; when S is lower than the first preset threshold but not lower than the second preset threshold, a doubtful warning is generated and pushed to the manual review queue; when S is lower than the second preset threshold, an error review conclusion is generated; wherein, for the financial document to be reviewed, the numerical difference of semantically inconsistent content is additionally calculated before performing the above calculation. If the numerical difference exceeds the preset numerical difference threshold, an error review conclusion is directly generated and the calculation of the comprehensive arbitration score is no longer performed.

[0010] Preferably, when the comprehensive arbitration score S is not lower than the first preset threshold, the corresponding entity name and its value are written into the temporary whitelist of this review session; in the subsequent processing of the document to be reviewed, when the entity queried in the shared semantic storage unit or the actual entity in the verification response matches the record in the temporary whitelist, the confidence level recorded in the temporary whitelist is directly used as the model inference confidence level, and the model inference step for that entity is skipped.

[0011] Preferably, each semantic record in the shared semantic storage unit includes the following fields: record identifier, subject field, predicate field, object field, source processing unit identifier, source location information, confidence level, and timestamp; wherein, the subject field includes the subject name and standardized specification name; the predicate field indicates the semantic relationship type between the subject and the object; the object field includes the value and unit of measurement; the source location information includes the document page number and paragraph number; the shared semantic storage unit supports the following operation interfaces: a write interface, allowing each processing unit to write the extracted semantic information into the shared semantic storage unit in the form of semantic records; a query interface, receiving the subject field and predicate field as query conditions, returning a list of matching object values ​​and corresponding confidence levels for cross-modal numerical comparison; and a conflict detection interface, receiving two semantic records as input, returning a conflict flag and the magnitude of the difference to trigger the multidimensional weighted arbitration.

[0012] Furthermore, this invention also provides a multimodal document review system, comprising: an arbitration scheduling unit configured to dynamically generate a task execution plan based on the distribution information of each modality content in the document to be reviewed, and to perform multidimensional weighted arbitration and output a graded review conclusion in response to semantic inconsistencies; a text processing module configured to parse the text content of the document to be reviewed, identify cross-modal association markers, and send a cross-modal semantic verification request carrying entity constraint information and expected semantic relationships to a target modal processing module in response to the cross-modal association markers, and to perform semantic comparison based on the received verification response; at least one non-text modal processing module configured to extract the measured semantic relationships of the corresponding content area in response to the cross-modal semantic verification request, and generate a verification response containing the measured semantic relationships and confidence levels; a message passing layer configured to route the cross-modal semantic verification request and the verification response between the text processing module and the at least one non-text modal processing module; and a shared semantic storage unit configured to store the semantic records extracted by each processing module and support cross-module queries and conflict detection.

[0013] Preferably, the system further includes a weight configuration library module, which stores pre-set arbitration weight parameter combinations according to document type; before executing the multi-dimensional weighted arbitration, the arbitration scheduling unit reads the corresponding weight parameter combination from the weight configuration library module according to the document type identifier of the document to be reviewed, as the weight coefficient used to execute the multi-dimensional weighted arbitration.

[0014] In addition, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multimodal document review method as described in any of the preceding claims.

[0015] Beneficial effects: This invention employs an event-driven cross-modal semantic verification mechanism, enabling text processing units to proactively discover cross-modal reference anchors during parsing and trigger targeted verification requests. This achieves point-to-point verification without centralized scheduling, effectively identifying deep logical contradictions between text descriptions and chart data, thus solving the problem of missing cross-modal logical consistency verification in existing technologies. Through a hybrid arbitration algorithm based on multi-dimensional confidence, it comprehensively considers hard constraints from the rule base, model inference confidence, and cross-modal corroboration weights for weighted judgment, and sets tiered thresholds to output different levels of review conclusions. This achieves intelligent arbitration of conflicts in multi-source detection results, reducing interference from false positives from a single model and decreasing the workload of manual review. Adaptive task orchestration based on modal density topology graphs dynamically activates the required processing units and generates directed acyclic graph execution plans based on document content, achieving dynamic adaptation of the review process, avoiding waste of computational resources, and improving review efficiency. Furthermore, through a unified semantic storage and query interface provided by a shared semantic storage unit and a temporary whitelist mechanism, it enables the sharing and reuse of semantic information during the review process, further improving the accuracy and efficiency of the review. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the multimodal document review method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the multimodal document review system according to an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0019] Example 1: like Figure 1 As shown, this embodiment provides a multimodal document review method. This method is applied to electronic devices with computing capabilities and is used to automatically review complex documents containing multiple modal information such as text, images, tables, and formulas. The method includes the following steps: Step S100: Based on the distribution information of each modal content in the document to be reviewed, dynamically determine the processing units and task execution plans to participate in the review.

[0020] Specifically, the distribution information of each modality in the document to be reviewed refers to the statistical and positional characteristics of different modal elements obtained during the document parsing phase. Distribution information includes, but is not limited to, the number of image elements, the distribution density of table elements, the embedding position of formula elements, and the density of reference relationships between text paragraphs and non-text modal elements. A processing unit refers to a software module, containerized service, or hardware acceleration unit with specific modal processing capabilities, such as a text processing unit, image processing unit, or table processing unit. Dynamically determining the processing units participating in the review means that the system does not pre-set a fixed processing pipeline before the review begins, but rather activates the corresponding processing units "on demand" based on the distribution information. For example, if the distribution information shows that the number of image elements in the document to be reviewed is zero and there are no image reference markers, the system will not activate the image processing unit, thereby saving computing resources. The task execution plan refers to the execution order, dependencies, and data flow paths between the activated processing units. By dynamically determining processing units and task execution plans based on distributed information, this embodiment can adaptively adjust the review process according to the actual content characteristics of the document, avoiding resource waste or review omissions caused by the "one-size-fits-all" fixed pipeline in traditional technologies, and achieving optimized allocation of review resources.

[0021] Step S200: During the parsing of the text content of the document to be reviewed, cross-modal association markers are identified; in response to the cross-modal association markers, a cross-modal semantic verification request carrying entity constraint information and expected semantic relationships is sent to the corresponding target modal processing unit, and a verification response containing measured semantic relationships and confidence levels is received from the target modal processing unit.

[0022] Specifically, cross-modal association markers refer to specific characters or symbols in the text content used to reference or point to other modal content, such as expressions like "as shown in Figure 3", "see Table 2", and "obtained from formula (5)". When parsing the text content, the system monitors the appearance of these markers in real time. Once a cross-modal association marker is identified, the system triggers an event-driven verification mechanism to generate a cross-modal semantic verification request. The entity constraint information carried in the request refers to the specific entity parameters mentioned in the text that need to be verified, such as material name, numerical value, and unit of measurement; the expected semantic relationship refers to the logical relationship between entities claimed in the text, such as "equal to", "greater than", and "contains". The target modal processing unit refers to the processing unit corresponding to the modality pointed to by the cross-modal association marker. For example, if the marker points to an image, then the target modal processing unit is an image processing unit. After receiving the request, the target modal processing unit performs feature extraction in the corresponding content area (such as the specified figure or table), obtains the measured semantic relationship, calculates the confidence of the extraction result, and then encapsulates this information into a verification response and returns it. Through the above process, this embodiment realizes active interaction and accurate verification between text modal and non-text modal, breaking the barrier of independent processing of each modality in the prior art, and can effectively identify cross-modal logical contradictions.

[0023] Step S300: Compare the measured semantic relationship in the verification response and the relevant semantic records in the shared semantic storage unit with the corresponding description in the text content. In response to the detection of semantic inconsistency, calculate the comprehensive arbitration score based on multi-dimensional weighted arbitration, and output the graded review conclusion based on the comprehensive arbitration score.

[0024] Specifically, the shared semantic storage unit is a global data storage area in the system used to store the semantic information extracted by each processing unit, supporting cross-modal data reading, writing, and querying. The comparison process involves verifying the consistency between the measured semantic relationships obtained in step S200 (such as the measured values ​​extracted from the drawings), the relevant historical records that may exist in the shared semantic storage unit (such as values ​​confirmed in previous steps), and the original description in the text content. If there is a contradiction among the three, such as a discrepancy between the text description value and the measured value, it is determined to be semantically inconsistent. In response to the detection of semantic inconsistency, the system does not directly output an erroneous conclusion but instead initiates a multi-dimensional weighted arbitration mechanism. Multi-dimensional weighted arbitration calculation refers to comprehensively considering information from multiple dimensions, such as the hard constraint judgments of the rule base, the inference confidence of the model, and the cross-modal corroboration weights, to obtain a quantified comprehensive arbitration score through weighted calculation. The graded review conclusion refers to outputting review results of different severity levels, such as "error," "questionable," or "correct," based on the numerical range of the comprehensive arbitration score. Through multi-dimensional weighted arbitration, this embodiment can effectively filter false alarms caused by a single detection source, improve the accuracy and robustness of the review results, and realize intelligent judgment of conflicting information from multiple sources.

[0025] Example 2: Based on step S100 of Embodiment 1, this embodiment provides a detailed explanation of the specific implementation process of "dynamically determining the processing units and task execution plans participating in the review based on the distribution information of each modal content in the document to be reviewed".

[0026] Specifically, before formally commencing deep semantic review, the system first performs a pre-scanning step to construct a modality density topology graph. This pre-scanning process includes: Step S101: Count the number of elements in each modality of the document to be reviewed and the distribution density of cross-modal association tags, and generate the weight ratio vector of each modality.

[0027] In this embodiment, the system scans the document to be reviewed on a page-by-page basis or in units of a preset fixed character length. The number of elements refers to the number of non-text modal objects such as images, tables, and formulas. The distribution density of cross-modal association markers refers to the frequency of occurrence of anchor references such as "as shown in Figure X" or "see Table Y" within a unit scan area. The weight proportion vector is a multi-dimensional numerical vector, such as [text weight, image weight, table weight, formula weight], where the value of each dimension is calculated by weighting the element quantity proportion of the corresponding modality with the reference density. For example, if a technical document contains a large number of engineering drawings and these drawings are frequently referenced in the text, the weight proportion of the image modality will significantly increase; conversely, if the document is a purely textual contract, the weight proportion of the image modality will be close to zero. By generating the weight proportion vector, the system can quantify the content composition characteristics of the document.

[0028] Step S102: Determine the activation state and execution priority of each processing unit based on the weight ratio vector.

[0029] Specifically, the system presets activation thresholds for each processing unit. When the weight percentage of a modality exceeds its activation threshold, the system activates the corresponding processing unit (such as an image processing unit or a table processing unit). If the threshold is not exceeded, the processing unit remains dormant and does not participate in the current review task, thus avoiding unnecessary resource consumption. Furthermore, the weight percentage directly determines the execution priority. Modalities with high weight percentages often carry the core logical information of the document, and their corresponding processing units will be assigned higher execution priority to prioritize the processing of key information. For example, in a financial audit report scenario, the table modality typically has a higher weight percentage, and the system will assign a higher priority to the table processing unit, prioritizing the extraction of key values ​​from the table to provide a verification benchmark for subsequent text parsing.

[0030] Step S103: The task execution plan is represented in the form of a dependency graph between task units, in which task units without dependencies are executed in parallel, and task units with dependencies are executed sequentially according to the topological sorting order.

[0031] In this embodiment, the task execution plan is concretized as a directed acyclic graph. Nodes in the graph represent specific task instances of each processing unit, and edges represent dependencies between tasks. The logic for determining dependencies lies in the data flow: if the execution of task B requires waiting for the output of task A as input, then there exists an edge from A to B, meaning A and B have a dependency relationship and must be executed sequentially; if there is no data dependency between task C and task D, then there is no edge connection between them and they can be executed in parallel. For example, in the review of engineering tender documents, after the "text parsing task" identifies the anchor point "as shown in Figure 3," it needs to trigger the "drawing recognition task" to extract the parameters in Figure 3. Therefore, the "drawing recognition task" depends on the output of the "text parsing task," and the two must be executed sequentially. However, if the document contains both "..." Figure 1 "and" Figure 2 If the references are unrelated and do not relate to each other, then it applies to... Figure 1 The recognition task and the target Figure 2 The identification tasks are independent of each other, and the system will schedule these two tasks to be executed in parallel. By constructing a Directed Acyclic Graph (DAG), the system can maximize the use of computing resources and achieve efficient workflow while ensuring logical correctness. This embodiment achieves dynamic adaptation of the review process through the above mechanism, effectively solving the problem that the fixed pipeline in the prior art cannot adapt to different types of documents, and significantly improving review efficiency.

[0032] Example 3: Based on step S200 of Example 1, this embodiment provides a detailed explanation of the interaction protocol and iterative verification mechanism in the cross-modal semantic verification process.

[0033] The cross-modal semantic verification request includes: a request identifier; an anchor type field, the value of which is at least one enumerated value among image type, table type, and formula type; an anchor location identifier; the original text sentence that triggers the cross-modal association marker; an entity list, each entity record in the entity list containing the entity name, entity value, and unit of measurement; an expected relationship list, each record in the expected relationship list containing the subject entity name, relationship type, and object entity name; and a confidence threshold.

[0034] Specifically, the request identifier is used to uniquely identify a verification interaction session, ensuring a one-to-one correspondence between requests and responses and preventing message misalignment in asynchronous communication environments. The anchor type field indicates which type of parsing algorithm the target modal processing unit should use. For example, if the value is "image type," the system routes the request to the image processing unit; if the value is "table type," it routes to the table processing unit. Anchor location identifiers (such as "Figure 3" and "Table 2") are used to precisely locate the specific position of the content to be verified in the document. The original text sentence that triggers the cross-modal association marker provides contextual information, which helps the target modal processing unit make auxiliary judgments when semantics are ambiguous. The entity list is the core payload of the verification request, carrying entity constraint information extracted from the text side. For example, when the text description is "The main beam is made of Q345 steel, with a plate thickness of 16mm," the entity list will contain two records: {Entity Name: Main Beam Material, Entity Value: Q345, Unit of Measurement: null} and {Entity Name: Plate Thickness, Entity Value: 16, Unit of Measurement: mm}. The expected relationship list defines the logical relationships that entities should satisfy, such as "main beam material" equaling "Q345". The confidence threshold is a quality control standard set by the initiator to determine the credibility of the response result. Through the above structured data protocol, this embodiment achieves precise "event-driven" verification, avoids the indiscriminate transmission and processing of all data, and significantly reduces system overhead.

[0035] The verification response includes: a request identifier corresponding to the cross-modal semantic verification request; an anchor point location identifier; a list of tested entities, each record in the list of tested entities including the entity name, the tested value and the corresponding confidence level; a list of tested relationships; and a list of conflict flags, each conflict record in the list of conflict flags including the conflict entity name, the expected value, the tested value, and a conflict severity field, the conflict severity field including high severity, medium severity and low severity.

[0036] Specifically, the measured entity list consists of actual data obtained by the target modal processing unit within the area specified by the anchor point positioning identifier (such as a drawing or table) through visual feature extraction or structured data parsing. For example, the image processing unit identifies the measured value of "main beam material" as "Q235" in Figure 3 with a confidence level of 0.97. The conflict flag list is a preliminary diagnostic result generated by the target modal processing unit when generating a response, automatically comparing the expected value with the measured value. The conflict severity field is graded according to the entity type and the magnitude of the difference. For example, conflicts involving critical safety parameters (such as material and load-bearing capacity) are marked as "high severity," while conflicts involving non-critical descriptive text are marked as "low severity." This mechanism allows the system to pre-classify conflicts during the response phase, facilitating the rapid identification of high-risk issues in subsequent arbitration processes.

[0037] In the verification response returned by the target modality processing unit, if the confidence level of any entity in the measured entity list is lower than the confidence level threshold, an additional cross-modal semantic verification request is generated and routed to the target modality processing unit. The additional cross-modal semantic verification request carries parameters to expand the recognition range or parameters specifying the feature type. The routing process executes at most a preset number of iterations, and the measured semantic relationship in the final verification response is used as the input for subsequent comparison steps.

[0038] Specifically, in actual review scenarios, document images may be blurry, obscured, or formatted incorrectly, leading to insufficient confidence in the initial extraction by the target modality processing unit. In this case, the system triggers an iterative verification mechanism. The "expand recognition range" parameter, carried in the cross-modal semantic verification request, instructs the target modality processing unit to expand the search area, for example, from the local area where the anchor point is located to the entire page of the document; the "specify feature type" parameter instructs the target modality processing unit to use a specific recognition model, such as a model specifically for handwritten digits or stamped text. A preset number of iterations (e.g., 3 times) is used to limit the maximum number of retries and prevent the system from entering an infinite loop. If the confidence is still insufficient after multiple iterations, the system will use the last response result as the basis for subsequent comparisons, but will mark "insufficient confidence" in the final review conclusion, recommending manual review. This embodiment effectively improves the system's robustness when facing low-quality document data through an iterative mechanism, ensuring the accuracy of the review results.

[0039] Example 4: This embodiment, based on step S300 of embodiment 1, provides a detailed explanation of the specific implementation process of multi-dimensional weighted arbitration calculation and hierarchical review conclusion output. When the system detects semantic inconsistency, the arbitration scheduling unit performs multi-dimensional weighted arbitration calculation.

[0040] The comprehensive arbitration score is calculated based on multidimensional weighted arbitration, including: calculating the comprehensive arbitration score S according to S=w1·Crule+w2·Cmodel+w3·Ccross.

[0041] Specifically, this formula integrates confidence information from three dimensions to address the high false positive rate of a single detection source. Crule represents the rule constraint confidence, indicating the hard constraint judgment result of the rule base on the detected content; its value is either 0 or 1. The rule base pre-sets hard rules such as industry standards and terminology specifications. If the detected content violates the rules (e.g., material numbers do not conform to national standards), Crule is 0; if it conforms, it is 1. Crule reflects the rigid requirements for compliance. Cmodel represents the model inference confidence, indicating the degree of confidence of the natural language processing module or visual feature extraction module in inferring the content; its value ranges from greater than or equal to 0 to less than or equal to 1. Cmodel reflects the soft probability of the AI ​​model's semantic understanding and can handle ambiguous semantic scenarios that rules cannot cover. Ccross represents the cross-modal corroboration weight; it takes a value greater than or equal to 0 and less than or equal to 1 when there is positive corroboration, and a value greater than or equal to negative 1 and less than 0 when there is negative conflict. The introduction of this parameter is one of the key points of this invention, enabling the arbitration process to use the consistency or contradiction of multimodal information to correct the judgment. For example, if the numerical value described in the text can be found and matches in the table, it is considered positive evidence; if the text description contradicts the drawing annotations, it is considered negative conflict. w1, w2, and w3 are dynamic weighting coefficients, read from the weighting configuration library based on the type of document to be reviewed. This dynamic weighting mechanism allows the system to adapt to the review focus of different business scenarios. For example, for engineering documents, there may be a greater emphasis on rule compliance, resulting in a higher weight for w1; for creative documents, there may be a greater emphasis on model reasoning, resulting in a higher weight for w2.

[0042] Based on the comprehensive arbitration score S, the graded review conclusion is output according to the following rules: Specifically, the system presets a first and a second threshold, with the first threshold being greater than the second. When S is not lower than the first threshold, it indicates that the multi-source evidence is highly consistent, is judged as correct, and the erroneous report is rejected to avoid false alarms interfering with users. When S is lower than the first threshold but not lower than the second threshold, it indicates that the evidence is contradictory or lacks confidence, is judged as questionable, generates a questionable warning, and pushes it to the manual review queue for human intervention. When S is lower than the second threshold, it indicates that the multi-source evidence is highly conflicting, is judged as incorrect, and an erroneous review conclusion is generated. Through tiered output, the system achieves a human-machine collaborative mode of "machines handling high-confidence issues and humans handling low-confidence issues," optimizing review efficiency.

[0043] For financial documents under review, the numerical difference of semantically inconsistent content is calculated before performing the above calculation. If the numerical difference exceeds the preset numerical difference threshold, an incorrect review conclusion is generated directly, and the calculation of the comprehensive arbitration score is no longer performed.

[0044] Specifically, financial documents require extremely high accuracy in their numerical values, as even minor discrepancies can represent significant financial risks. Therefore, the system employs a special pre-validation logic for financial documents. If the detected numerical difference (e.g., percentage difference) exceeds a preset threshold (e.g., 0.5%), it is directly flagged as an error, skipping the complex weighted calculation process to ensure the rigor of the financial data and prevent omissions due to weighting.

[0045] When the overall arbitration score S is not lower than the first preset threshold, the corresponding entity name and its value will be written into the temporary whitelist for this review session.

[0046] Specifically, the temporary whitelist is a dynamic knowledge base maintained by the system during the current review session. When an entity is determined to be "correct" by arbitration, the system records it in the temporary whitelist. During the subsequent processing of the document to be reviewed, when an entity queried from the shared semantic storage unit or a tested entity in the verification response matches a record in the temporary whitelist, the confidence level recorded in the temporary whitelist is directly used as the model inference confidence level, skipping the model inference step for that entity.

[0047] This mechanism enables "progressive learning" during the review process. As the review progresses, the system accumulates more and more confirmed entities. When encountering the same entity again, there is no need to repeat complex model reasoning or cross-modal verification; the previously correct conclusion can be directly reused. This not only improves review efficiency but also ensures the consistency of judgments for the same entity within the same document, avoiding contradictory judgments for the same entity in different locations.

[0048] Example 5: This embodiment provides a detailed description of the specific data structure and operation interface of the shared semantic storage unit mentioned in Embodiment 1. Each semantic record in the shared semantic storage unit includes the following fields: record identifier, subject field, predicate field, object field, source processing unit identifier, source location information, confidence level, and timestamp; wherein, the subject field includes the subject name and standardized specification name; the predicate field indicates the semantic relationship type between the subject and the object; the object field includes the value and unit of measurement; the source location information includes the document page number and paragraph number; the shared semantic storage unit supports the following operation interfaces: a write interface, which allows each processing unit to write the extracted semantic information into the shared semantic storage unit in the form of semantic records; a query interface, which receives the subject field and predicate field as query conditions and returns a list of matching object values ​​and corresponding confidence levels for cross-modal numerical comparison; and a conflict detection interface, which receives two semantic records as input and returns a conflict flag and difference magnitude to trigger multi-dimensional weighted arbitration.

[0049] Specifically, the shared semantic storage unit acts as a "global semantic blackboard" for the entire multimodal review system. Its core function is to break down data silos between different modal processing units and achieve real-time sharing and interoperability of semantic information. Record identifiers are used to uniquely identify a semantic record, facilitating indexing and management. Standardized names in the main fields are a key technical means to achieve accurate cross-modal matching. In actual documents, different modalities may use different names for the same entity. For example, the text modality may use "main beam material," while the table modality may use "material grade." If only string matching is performed, the matching will fail. This embodiment introduces standardized names (such as a unified mapping to "steel_grade"), enabling semantic records written by different processing units to be associated based on this standard name, thus effectively solving the matching omission problem caused by synonyms and near-synonyms. The predicate field defines the logical relationships between entities, such as "has_value" (value is...), "greater_than" (greater than...), etc., providing a structured data foundation for subsequent logical verification. The source processing unit identifier and source location information record the origin of the data, enabling the document's specific page number and paragraph to be quickly located when a conflict is discovered, facilitating manual review or tracing.

[0050] Furthermore, the three operation interfaces provided by the shared semantic storage unit are integrated throughout the review process. The write interface is typically called after each processing unit completes single-modal information extraction. For example, after parsing a parameter table, the table processing unit writes several extracted semantic records into the storage unit in batches. The query interface primarily serves the cross-modal verification stage. When the text processing unit parses a description, it can use the query interface to retrieve whether other modalities have recorded the values ​​of related entities. If so, a direct comparison is performed without triggering the complex non-text modal recognition process again, thus improving review efficiency. The conflict detection interface is the pre-trigger for the arbitration mechanism. When the record returned by the query interface has a potential conflict with the current parsing result, the system calls the conflict detection interface to calculate the specific difference magnitude (such as the absolute value or percentage of the numerical difference) and generate a conflict flag. This difference magnitude data can be directly used as the input parameter for the multi-dimensional weighted arbitration in Example 4. Especially for financial documents, this difference magnitude directly determines whether the logic for quickly generating erroneous review conclusions is triggered. Through the above-described structured data definition and standardized interface design, this embodiment constructs a highly cohesive and loosely coupled semantic storage center, providing solid data support for multi-agent collaborative review.

[0051] Example 6: This embodiment provides a multimodal document review system, which is used to execute the multimodal document review method described in the above embodiment. For example... Figure 2As shown, the system includes an arbitration scheduling unit, a text processing module, at least one non-text modal processing module, a message passing layer, and a shared semantic storage unit.

[0052] The arbitration scheduling unit is configured to dynamically generate a task execution plan based on the distribution information of each modality in the document to be reviewed, and to perform multi-dimensional weighted arbitration in response to semantic inconsistencies and output hierarchical review conclusions.

[0053] Specifically, the arbitration scheduling unit is the central control node of the entire system, equivalent to the "brain" in a multi-agent collaborative architecture. At the hardware level, it can be implemented by a high-performance processor or application-specific integrated circuit running a scheduling algorithm. During system initialization, the arbitration scheduling unit is responsible for pre-scanning the documents to be reviewed, calculating the modality density topology graph, and generating a task execution plan in the form of a directed acyclic graph. During the review process, when a conflict report is received from other modules, the arbitration scheduling unit reads the rule base and model confidence, performs weighted calculations, and finally outputs the review conclusion. The existence of this unit enables the system to coordinate and schedule the processing resources of each modality from a global perspective, avoiding resource conflicts caused by the disordered operation of various modules.

[0054] The text processing module is configured to parse the text content of the document to be reviewed, identify cross-modal association markers, and in response to the cross-modal association markers, send a cross-modal semantic verification request carrying entity constraint information and expected semantic relationships to the target modal processing module, and perform semantic comparison based on the received verification response.

[0055] Specifically, the text processing module serves as the entry point for text modal information within the system, integrating natural language processing algorithms and a regular expression matching engine. Physically, the text processing module connects to other non-text modal processing modules via a message passing layer. When the text processing module parses an anchor point as shown in Figure 3, it does not directly call the image processing module's interface. Instead, it generates a standardized cross-modal semantic verification request message and publishes it to the message passing layer. This design allows the text processing module to focus solely on message publishing without needing to be aware of the specific implementation details or physical location of the image processing module, thus achieving decoupling between modules.

[0056] At least one non-text modal processing module is configured to extract the measured semantic relationships of the corresponding content region in response to a cross-modal semantic verification request, and generate a verification response containing the measured semantic relationships and confidence levels.

[0057] Specifically, non-text modal processing modules include, but are not limited to, image processing modules, table processing modules, and formula processing modules. These modules exist in the system as independent service processes or containerized instances, receiving requests by subscribing to specific topics on the messaging layer. For example, the image processing module subscribes to the "image type" verification request topic. Upon receiving a request, the image processing module uses a visual feature extraction algorithm to identify the specified document area and encapsulates the identification result into a verification response message for return. Because each non-text modal processing module runs independently, the system can be easily horizontally scaled, for example, by increasing the number of image processing module instances to handle document review tasks containing a large number of drawings, without modifying the code of other modules.

[0058] The messaging layer is configured to route cross-modal semantic verification requests and responses between the text processing module and at least one non-text modal processing module.

[0059] Specifically, the messaging layer serves as the communication infrastructure connecting various processing modules, implementing a publish-subscribe message routing model. In the system architecture, the messaging layer acts as a "neural pathway," shielding the complexity of underlying network communication and eliminating the need for direct point-to-point connections between processing modules. When a text processing module sends a request, the messaging layer routes it to a non-text modal processing module that has subscribed to that type based on the anchor type field in the request; conversely, when a non-text modal processing module generates a response, the messaging layer accurately routes it back to the text processing module that initiated the request based on the request identifier. This architectural design significantly improves the system's scalability and maintainability. When adding new modal processing capabilities (such as 3D model review), only the corresponding processing module needs to be added and subscribed to the relevant message topic, without requiring intrusive modifications to the existing system.

[0060] A shared semantic storage unit is configured to store semantic records extracted by each processing module and supports cross-module queries and conflict detection.

[0061] Specifically, the shared semantic storage unit can physically be a distributed database or a high-performance in-memory database, which can be read and written by all processing modules. It acts as the system's "global semantic blackboard," breaking down data silos between different modal processing modules. After extracting semantic information, each module stores the data in this unit by calling the write interface; during cross-modal comparisons, it retrieves historical extraction results from other modules through the query interface. This centralized semantic storage design ensures that when the same entity is mentioned multiple times in different modalities and locations, the system can obtain a consistent semantic context, providing a solid data foundation for subsequent conflict detection and arbitration.

[0062] Furthermore, the system also includes a weight configuration library module, which stores pre-set combinations of arbitration weight parameters according to document type.

[0063] Specifically, the weight configuration library module is a data storage unit independent of the arbitration scheduling unit, and it internally maintains a "document type - weight parameter" mapping table. For example, for the "engineering tender document" type, the stored weight parameter combination may focus on rule constraints (w1 is larger); for the "creative design document" type, the stored weight parameter combination may focus on model inference (w2 is larger). Before performing multi-dimensional weighted arbitration, the arbitration scheduling unit reads the corresponding weight parameter combination from the weight configuration library module according to the document type identifier of the document to be reviewed, as the weight coefficients used to perform multi-dimensional weighted arbitration.

[0064] This mechanism enables the system to flexibly adjust arbitration strategies according to different business scenarios, avoiding the incompatibility of a single arbitration standard in the review of documents in different fields, thereby improving the accuracy and professionalism of the review results.

[0065] Through the system architecture described above, this embodiment constructs a highly cohesive, loosely coupled multi-agent collaborative review platform. Each processing module focuses on information extraction for a specific modality, communicates lightweightly through a message passing layer, shares data through a shared semantic storage unit, and finally makes global decisions through an arbitration and scheduling unit. This architecture not only improves the system's processing efficiency and scalability but also ensures the rigor and consistency of cross-modal review logic.

[0066] Example 7: This embodiment uses a specific engineering tender document review scenario as an example to explain in detail the application process and effect of the above-mentioned multimodal document review method in actual business. The document to be reviewed is set as a tender document containing 200 pages of technical specifications and 85 engineering drawings. The technical specifications, page 3, paragraph 2, describes "the main beam uses Q345 steel with a plate thickness of 16mm (as shown in Figure 3)," while the material shown in the drawing annotations in Figure 3 is "Q235" with a plate thickness of "14mm."

[0067] At the start of the review process, the system first performs a pre-scan step. The arbitration scheduling unit counts the number of elements in each modality of the document, identifying 85 image elements, and noting a high distribution density of cross-modal association markers in the text (such as "as shown in Figure X"). Based on this distribution information, the system generates a modality density topology graph (MDTM), calculating that the weight proportion vector of the image modality is significantly higher than that of a regular plain text document. Accordingly, the arbitration scheduling unit dynamically activates the image processing unit and determines its execution priority, while simultaneously generating a dependency graph (DAG) between task units, where the "text parsing task" serves as the starting node, and the "drawing recognition task" depends on the output of the "text parsing task," forming a serial execution path.

[0068] The text processing unit begins parsing the technical specification document according to the DAG plan. When parsing to paragraph 2 on page 3, it identifies the cross-modal association marker "as shown in Figure 3". In response to this marker, the text processing unit sends a cross-modal semantic verification request to the corresponding target modal processing unit—the image processing unit. The entity constraint information carried in this request includes the entity list: [{Entity Name: Main Beam Material, Entity Value: Q345, Unit of Measurement: null}, {Entity Name: Plate Thickness, Entity Value: 16, Unit of Measurement: mm}]; the expected semantic relationships include: [{Subject Entity Name: Main Beam Material, Relationship Type: equals, Object Entity Name: Q345}, {Subject Entity Name: Plate Thickness, Relationship Type: equals, Object Entity Name: 16mm}]. The request also includes the anchor type field "Image Type" and the anchor positioning identifier "Figure 3".

[0069] After receiving the request through the message passing layer, the image processing unit locates the area in Figure 3 of the document and uses a visual feature extraction algorithm to identify the annotations on the drawing. The verification response generated by the identification results includes a list of measured entities: [{Entity Name: Main Beam Material, Measured Value: Q235, Confidence Level: 0.97}, {Entity Name: Plate Thickness, Measured Value: 14, Unit of Measurement: mm, Confidence Level: 0.95}]. When generating the response, the image processing unit automatically compares the expected values ​​with the measured values ​​and finds inconsistencies between "Main Beam Material" and "Plate Thickness". Therefore, it generates two conflict records in the conflict flag list of the verification response, both marked as "High Severity". At the same time, the image processing unit writes the extracted measured semantic information into the shared semantic storage unit, recording the source location information as "Figure 3".

[0070] After receiving the verification response, the text processing unit detects a semantic inconsistency and triggers the arbitration scheduling unit to execute multi-dimensional weighted arbitration. The arbitration scheduling unit reads the weight parameter combination of the engineering documents from the weight configuration library and sets w1=0.5, w2=0.3, and w3=0.2. Since Q345 and Q235 are both compliant steel grades, the rule library does not hit the hard constraint, so the rule constraint confidence level Crule=0; the image recognition confidence level is relatively high, so the model inference confidence level Cmodel=0.96 is taken; the measured value contradicts the expected value, indicating a negative conflict, so the cross-modal corroboration weight Ccross=-0.1 is taken. Substituting into the formula, the comprehensive arbitration score S=0.5×0+0.3×0.96+0.2×(-0.1)=0.268 is calculated.

[0071] The system presets a first threshold of 0.8 and a second threshold of 0.5. Since the calculated comprehensive arbitration score S=0.268 is lower than the second threshold, the system determines this inconsistency as an error and generates an erroneous review conclusion: "The description of the main beam material and plate thickness in paragraph 2 on page 3 of the technical specification is inconsistent with the annotation in Figure 3; verification is recommended." Through the above process, this embodiment fully demonstrates the entire process from document pre-scanning, cross-modal verification request generation, non-text modal information extraction, conflict detection, and multi-dimensional weighted arbitration. It effectively identifies deep-seated logical errors of "text-figure discrepancies" in engineering bidding documents, verifying the effectiveness and accuracy of the method provided by this invention in complex document review scenarios.

[0072] Example 8: This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multimodal document review method as described in any of the above embodiments.

[0073] Specifically, a computer-readable storage medium can be any tangible medium that can contain or store a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this embodiment, a computer-readable storage medium includes, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic storage, optical disk, or any suitable combination thereof. When a computer program is executed by a processor, the processor performs the following steps according to the program instructions: dynamically determining the processing units and task execution plans involved in the review based on the distribution information of each modality in the document to be reviewed; identifying cross-modal association markers during the parsing of the text content of the document to be reviewed; in response to the cross-modal association markers, sending a cross-modal semantic verification request carrying entity constraint information and expected semantic relationships to the corresponding target modal processing unit, and receiving a verification response from the target modal processing unit containing measured semantic relationships and confidence levels; comparing the measured semantic relationships in the verification response and the relevant semantic records in the shared semantic storage unit with the corresponding descriptions in the text content; in response to the detection of semantic inconsistencies, calculating a comprehensive arbitration score based on multi-dimensional weighted arbitration, and outputting a graded review conclusion based on the comprehensive arbitration score.

[0074] It should be understood that the processor in this embodiment can be an electronic component with data processing capabilities, such as a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), or field-programmable gate array (FPGA). By storing the above method steps in the form of a computer program on a computer-readable storage medium, the technical solution of the present invention can be implemented through standard computing devices, enhancing the versatility and portability of the solution and facilitating its deployment and application in various servers, terminal devices, or cloud platforms.

[0075] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention, such as adjusting the specific value of the arbitration weight, replacing the specific communication protocol of the message passing layer, or adding or removing the types of non-text modal processing modules, should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal document review method, characterized in that, include: Based on the distribution information of each modal content in the document to be reviewed, the processing units and task execution plans participating in the review are dynamically determined; the distribution information includes the number of elements and the distribution density of cross-modal association tags; During the process of parsing the text content of the document to be reviewed, cross-modal association markers are identified; in response to the cross-modal association markers, a cross-modal semantic verification request carrying entity constraint information and expected semantic relationships is sent to the corresponding target modal processing unit, and a verification response containing measured semantic relationships and confidence levels is received from the target modal processing unit. The measured semantic relationships in the verification response and the relevant semantic records in the shared semantic storage unit are compared with the corresponding descriptions in the text content. In response to the detection of semantic inconsistency, a comprehensive arbitration score is calculated based on multi-dimensional weighted arbitration, and a graded review conclusion is output based on the comprehensive arbitration score. The semantic records include the semantic information extracted by each of the processing units.

2. The multimodal document review method according to claim 1, characterized in that, The dynamic determination of processing units and task execution plans for review based on the distribution information of each modal content in the document to be reviewed includes: The number of elements in each modality of the document to be reviewed and the distribution density of cross-modal association tags are counted to generate a weight vector of each modality. Based on the weight ratio vector, the activation state and execution priority of each processing unit are determined; The task execution plan is represented in the form of a dependency graph between task units, in which task units without dependencies are executed in parallel, and task units with dependencies are executed sequentially according to topological sorting order.

3. The multimodal document review method according to claim 2, characterized in that: The cross-modal semantic verification request includes: a request identifier; an anchor type field, the value of which is at least one enumerated value among image type, table type, and formula type; an anchor positioning identifier; the original text sentence that triggers the cross-modal association marker; an entity list, each entity record in the entity list containing an entity name, entity value, and unit of measurement; and an expected relationship list, each record in the expected relationship list containing a subject entity name, relationship type, and object entity name. The verification response includes: a request identifier corresponding to the cross-modal semantic verification request; an anchor point location identifier; a list of measured entities, each record in the list of measured entities including an entity name, a measured value, and a corresponding confidence level; a list of measured relationships; and a list of conflict flags, each conflict record in the list of conflict flags including a conflict entity name, an expected value, a measured value, and a conflict severity field, the conflict severity field including high severity, medium severity, and low severity.

4. The multimodal document review method according to claim 3, characterized in that, In the verification response returned by the target modality processing unit, if the confidence level of any entity in the measured entity list is lower than the confidence level threshold, an additional cross-modal semantic verification request is generated and routed to the target modality processing unit. The additional cross-modal semantic verification request carries parameters that expand the recognition range or parameters that specify the feature type; The routing process executes at most a preset number of iterations, and then uses the measured semantic relationship in the final verification response as the input for subsequent comparison steps.

5. The multimodal document review method according to claim 1, characterized in that, The comprehensive arbitration score calculated based on multidimensional weighted arbitration includes: The comprehensive arbitration score S is calculated according to S = w1·Crule + w2·Cmodel + w3·Ccross; Wherein, Crule is the rule constraint confidence score, representing the hard constraint judgment result of the rule base on the detected semantically inconsistent content, and its value is 0 or 1; Cmodel is the model inference confidence score, representing the confidence level of the natural language processing module or visual feature extraction module in the inference of the content, and its value ranges from greater than or equal to 0 to less than or equal to 1; Ccross is the cross-modal corroboration weight, which takes a value in the range of greater than or equal to 0 to less than or equal to 1 when there is positive corroboration, and takes a value in the range of greater than or equal to negative 1 to less than 0 when there is negative conflict; w1, w2, and w3 are dynamic weight coefficients, which are read from the weight configuration library according to the type of the document to be reviewed; Based on the comprehensive arbitration score S, the graded review conclusion is output according to the following rules: When S is not lower than the first preset threshold, it is judged as correct and the error report is rejected; When S is lower than the first preset threshold and not lower than the second preset threshold, a questionable warning is generated and pushed to the manual review queue. When S is lower than the second preset threshold, an incorrect audit conclusion is generated; Specifically, for the financial documents to be reviewed, the numerical difference of semantically inconsistent content is calculated before performing the above calculation. If the numerical difference exceeds a preset numerical difference threshold, an incorrect review conclusion is generated directly, and the calculation of the comprehensive arbitration score is no longer performed.

6. The multimodal document review method according to claim 5, characterized in that, When the comprehensive arbitration score S is not lower than the first preset threshold, the corresponding entity name and its value are written into the temporary whitelist of this review session; In the subsequent processing of the document to be reviewed, when the entity queried in the shared semantic storage unit or the actual entity in the verification response matches the record in the temporary whitelist, the confidence level of the record in the temporary whitelist is directly used as the confidence level of the model inference, and the model inference step for that entity is skipped.

7. The multimodal document review method according to claim 1, characterized in that, Each semantic record in the shared semantic storage unit includes the following fields: record identifier, subject field, predicate field, object field, source processing unit identifier, source location information, confidence level, and timestamp; The subject field includes the subject name and the standardized specification name; the predicate field indicates the semantic relationship type between the subject and the object; the object field includes the value and the unit of measurement; and the source location information includes the document page number and the paragraph number. The shared semantic storage unit supports the following operation interfaces: A write interface is provided for each of the processing units to write the extracted semantic information into the shared semantic storage unit in the form of semantic records; The query interface accepts the subject field and predicate field as query conditions and returns a list of matching object values ​​and corresponding confidence scores for cross-modal numerical comparison. The conflict detection interface receives two semantic records as input and returns a conflict flag and the magnitude of the difference to trigger the multidimensional weighted arbitration.

8. A multimodal document review system, characterized in that, include: The arbitration scheduling unit is configured to dynamically generate a task execution plan based on the distribution information of each modality in the document to be reviewed, and to perform multi-dimensional weighted arbitration in response to semantic inconsistencies and output a graded review conclusion; the distribution information includes the number of elements and the distribution density of cross-modal association tags; The text processing module is configured to parse the text content of the document to be reviewed, identify cross-modal association markers, and in response to the cross-modal association markers send a cross-modal semantic verification request carrying entity constraint information and expected semantic relationships to the target modal processing module, and perform semantic comparison based on the received verification response; At least one non-text modal processing module is configured to, in response to the cross-modal semantic verification request, extract the measured semantic relationships of the corresponding content region and generate a verification response containing the measured semantic relationships and confidence levels; A messaging layer is configured to route the cross-modal semantic verification request and the verification response between the text processing module and the at least one non-text modal processing module; A shared semantic storage unit is configured to store semantic records extracted by each processing module and supports cross-module queries and conflict detection.

9. The multimodal document review system according to claim 8, characterized in that, The system also includes a weight configuration library module, which stores pre-set combinations of arbitration weight parameters according to document type; Before executing the multi-dimensional weighted arbitration, the arbitration scheduling unit reads the corresponding weight parameter combination from the weight configuration library module according to the document type identifier of the document to be reviewed, so as to use it as the weight coefficient for executing the multi-dimensional weighted arbitration.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the multimodal document review method as described in any one of claims 1 to 7.

Images