A method for content understanding and business rule matching of large models for multimodal data

By establishing a closed-loop process from multimodal data to business rules, the problem of multimodal data being difficult to utilize directly is solved. This enables the automation, credibility, and traceability of structured evidence generation and rule matching, thereby improving the efficiency of enterprise business processing and the accuracy of intelligent decision-making.

CN123115809APending Publication Date: 2026-11-17WUXI BAISHANG ZHONGWANG DATA TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611208930.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-11
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

There is a technological gap between the semantic understanding output of multimodal large models and the deterministic input of traditional rule engines in existing technologies. There is a lack of an intermediate bridge to convert multimodal content understanding results into rule-computable structured business evidence, resulting in insufficient accuracy and reliability of automated decision-making and a high degree of reliance on human intervention.

Method used

By integrating multimodal data parsing, multimodal large-scale model semantic understanding, structured business rule modeling, and intelligent matching decision-making technologies, a closed-loop process is established from multimodal business data to business rules for automatic matching and interpretable decision-making. This includes multimodal content fragment generation, cross-modal association, structured evidence generation, rule matching, and conflict verification, forming a computable and traceable evidence layer.

Benefits of technology

It enables automated, reliable, and traceable business rule matching in complex business scenarios, reducing manual review costs and improving enterprise business processing efficiency and intelligent decision-making capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115809A_ABST
    Figure CN123115809A_ABST
Patent Text Reader

Abstract

This invention belongs to the field of computer data processing technology, specifically relating to a method for understanding large-scale model content and matching business rules for multimodal data. By integrating multimodal data parsing technology, multimodal large-scale model semantic understanding technology, structured business rule modeling technology, and intelligent matching decision technology, it establishes a complete closed-loop processing flow from multimodal business data input to automatic business rule matching, conflict verification, and interpretable decision output. It transforms multimodal content understanding results into computable evidence of business rules, and combines intelligent rule matching, conflict verification, and interpretable decision output to achieve automated, reliable, and traceable business rule matching in complex business scenarios. This solves problems such as the difficulty of directly utilizing multimodal data by business rules, low rule matching accuracy, high reliance on manual intervention, and lack of reliable explanation for matching results, thereby reducing manual review costs and improving enterprise business processing efficiency and intelligent decision-making capabilities.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of computer data processing technology, specifically relating to a method for understanding the content of large models and matching business rules for multimodal data. Background Technology

[0002] As enterprises deepen their digital transformation, business systems generate a large amount of unstructured or semi-structured data in various forms, including text, images, tables, audio, and video. In complex business scenarios such as contract review, financial risk control, insurance claims, customer service quality inspection, and intelligent approval, key business information is no longer confined to structured fields in databases but is scattered across contract texts, scanned attachments, spreadsheets, customer service recordings, surveillance videos, and various multimedia materials. Enterprises urgently need to uniformly understand and extract business semantics from this multi-source heterogeneous data and automatically match corresponding approval rules, risk control rules, compliance rules, and operating procedures to reduce manual review costs and improve business processing efficiency and intelligent decision-making capabilities.

[0003] In existing technologies, a typical implementation combines a multimodal content parsing system, a multimodal large-scale model, and a traditional business rule engine. First, the system independently processes different modalities of data using tools such as text parsing, OCR recognition, table parsing, speech-to-text, and video analysis. Second, the recognition results of each modality are converted into text descriptions or structured fields, which are then input into the multimodal large-scale model for content summarization, entity extraction, risk identification, or classification. The model is required to output results according to JSON or predefined field templates. Next, the fields output by the large-scale model are mapped to the input parameters of the traditional rule engine, which then compares the field values ​​with rule thresholds based on pre-configured conditional expressions. Finally, the system outputs the matched rule number or execution action; tasks with low confidence or missing fields are transferred to manual review. Some solutions also assign fixed priorities to rules, executing them in a preset order when multiple rules are matched simultaneously.

[0004] However, a key drawback of the aforementioned existing technical solutions lies in the technical gap between the semantic understanding output of the multimodal large model and the deterministic input of traditional rule engines. There is a lack of an intermediate bridge to convert the multimodal content understanding results into rule-computable structured business evidence. Specifically, the large model outputs natural language descriptions or simple labels, while the rule engine requires evidence units with clear types, definite values, original sources, and credibility annotations. Existing solutions only perform format conversion through fixed field templates or manual configuration of mapping relationships, failing to address issues such as cross-modal information fusion verification, evidence source tracing, conflict resolution of multi-source evidence, and automatic verification of the rule matching process. This deficiency prevents the large model's understanding capabilities from truly driving the rule engine to make reliable automated decisions. When facing complex scenarios such as joint judgment of cross-modal evidence, synonym recognition, implicit conditional reasoning, and rule conflict detection, the system still heavily relies on manual intervention, making it difficult to meet the accuracy, reliability, and auditability requirements of enterprise business for automated decision-making. Therefore, this invention provides a method for large model content understanding and business rule matching for multimodal data. Summary of the Invention

[0005] The purpose of this invention is to provide a method for understanding large-scale model content and matching business rules for multimodal data. By integrating multimodal data parsing technology, multimodal large-scale model semantic understanding technology, structured business rule modeling technology, and intelligent matching decision technology, a complete closed-loop processing flow is established from multimodal business data input to automatic business rule matching, conflict verification, and interpretable decision output. By transforming the results of multimodal content understanding into computable evidence of business rules, and combining intelligent rule matching, conflict verification, and interpretable decision output, automated, reliable, and traceable business rule matching is achieved in complex business scenarios. This solves the problems in existing technologies, such as the difficulty of directly utilizing multimodal data for business rules, low rule matching accuracy, high dependence on manual intervention, and lack of reliable interpretation of matching results. This reduces the cost of manual review and improves the efficiency of enterprise business processing and intelligent decision-making capabilities.

[0006] The specific technical solution adopted by this invention is as follows:

[0007] A method for content understanding and business rule matching of large models for multimodal data.

[0008] It receives business data in at least two modalities from text, images, tables, audio, and video; establishes business processing tasks; identifies each data modality; performs entity recognition and segmentation on text; performs optical character recognition and layout analysis on images; performs structure parsing and numerical standardization on tables; performs speech recognition and speaker separation on audio; performs keyframe extraction and audio track transcription on video; and generates multimodal content fragments in a unified format. Each fragment records the modality type, processing content, location coordinates, and confidence level.

[0009] Based on spatial location, timestamp, document structure, business entities, and semantic similarity, establish the association relationship between cross-modal content fragments to form a multimodal content association graph;

[0010] The understanding target is determined according to the task type and the applicable domain of the rules. Relevant fragments are selected from the association graph to construct the context. A multimodal big model is called to perform joint semantic understanding and output structured semantic understanding results. The results include entities, attributes, events, relationships, intentions, risk tags and at least one original content fragment identifier on which they are based.

[0011] Based on the business semantic model and evidence template, the semantic understanding results are converted into structured business evidence units. Each business evidence unit records the business object, evidence category, standardized value, value type, source fragment identifier, identification confidence, and source reliability. Consistent fusion is performed on multi-source evidence supporting the same fact, conflicting evidence is marked, and a business evidence set is formed after data format, value range, and integrity verification.

[0012] Obtain applicable business rules and parse them into structured rule objects containing conditions, required evidence, priorities, dependencies, and execution actions;

[0013] Based on the aforementioned business evidence set, a rule matching request is constructed, and the following steps are executed sequentially: a first-level rule pre-screening is performed using business type and key fields; a second-level semantic recall is performed using the similarity between the evidence semantic vector and the rule semantic vector; a third-level fine-grained matching is performed to determine the state in which the rule conditions are met; and candidate rules are cross-coded and reordered to obtain highly relevant candidate rules.

[0014] The conditions of highly relevant candidate rules are calculated. Numerical comparison and logical combination conditions are completed by the rule engine, while natural language implicit conditions or cross-modal joint judgment conditions are verified by the large model. The comprehensive matching score is calculated based on semantic similarity, field consistency, condition satisfaction, evidence coverage and evidence credibility to determine the rule hit status.

[0015] For multiple matching rules, conflict detection is performed on actions, conditions, and priorities. Conflicts are resolved according to strategies such as rule validity, priority of special rules, priority of high priority, and sufficiency of evidence. The proposed actions are then reverse-verified to determine whether their rule basis and supporting evidence are sufficient.

[0016] Based on the rule hit status, evidence integrity, and conflict resolution results, the system performs tiered business actions such as automatically approving, rejecting, requesting supplementary materials, or transferring the request to manual review, generating interpretable matching results. These results include the hit rules, matching scores, associated evidence, and the location of the evidence in the original modality.

[0017] Record the complete decision-making path from data input to action output, obtain human feedback and business execution results, use the feedback to optimize multimodal preprocessing parameters, cross-modal association weights, rule recall thresholds and matching score weights, and update and release after offline evaluation;

[0018] The structured business evidence unit constitutes an intermediate evidence layer between multimodal understanding output and rule execution, enabling rule matching to be based on computable and traceable structured evidence, rather than directly relying on the natural language output of a large model.

[0019] In step 1:

[0020] In step 101: Receiving multimodal service data and establishing a specific service processing task,

[0021] The multimodal business data includes one or more of the following: text files, scanned images, spreadsheets, audio files, video files, and structured business fields. The multimodal business data originates from contract systems, invoice systems, approval systems, work order systems, customer service systems, knowledge bases, multimedia acquisition devices, or external business interfaces. Upon receiving the business data, a unique task identifier is generated for this processing step, and the following task attributes are obtained: business type, business object identifier, data source, file or data source identifier, applicable rule domain, user permission scope, task creation time, and business context information.

[0022] When multiple files or multiple modal data belong to the same business matter, they are merged into the same business processing task based on the business number, contract number, work order number, personnel identifier, organization identifier, time range, or file association.

[0023] The system generates task objects:

[0024] Task={task id business type business object_id ,source list[] ,rule domain ,context information permission scope ,create time ,processing status}

[0025] In step 102: Identifying data modes and selecting the corresponding preprocessing steps:

[0026] The system identifies the modality type of input data based on file extension, header information, media encoding, data structure, and content characteristics. When a single file contains multiple modalities, the system recognizes it as a composite multimodal file and breaks it down into text areas, image areas, table areas, audio tracks, video frames, and subtitle information.

[0027] Based on the modality type, the corresponding preprocessing component is called: text data enters the text parsing process, images or scanned documents enter the image parsing process, table data enters the table structure parsing process, audio data enters the speech processing process, and video data enters the video timing analysis process.

[0028] For unrecognized data types, mark them as unknown modalities and call a general parsing component or transfer them to manual verification based on the file content.

[0029] In step 103: the preprocessing for modal normalization includes text data processing, image data processing, tabular data processing, audio data processing, and video data processing;

[0030] Text data processing: For text data, the system performs format conversion, text extraction, page number recognition, heading level recognition, paragraph segmentation, sentence segmentation, and text cleaning.

[0031] Then perform: named entity recognition, keyword extraction, event element extraction, business term recognition, negative word and condition word recognition, and text semantic vector generation.

[0032] For long documents, divide them into blocks based on heading level, paragraph boundaries, and semantic continuity. Set a predetermined overlap between adjacent text blocks to reduce the truncation of key information due to block division.

[0033] Image data processing: For images or scanned documents, the system first performs image rotation correction, perspective correction, noise reduction, contrast enhancement, and resolution adjustment.

[0034] The following steps are then performed: OCR text recognition, page layout analysis, target object detection, seal, signature and logo recognition, image text and region coordinate binding, and visual semantic feature extraction. Each recognition region records the page number, bounding box coordinates, region type, recognition content, and recognition confidence level.

[0035] Spreadsheet data processing: For spreadsheets or tables embedded in documents, the system identifies worksheets, headers, row and column boundaries, merged cells, and data ranges.

[0036] Establish the relationships between table headers, field values, and business objects, and perform data type determination and format standardization for amounts, dates, ratios, numbers, names, and enumeration items.

[0037] Table fields can be represented as:

[0038] TableField={sheet id ,table id ,row index column index header name ,raw value normalized value value type confidence}

[0039] Audio data processing: For audio data, the system performs noise reduction, speech activity detection, speech segmentation, speaker separation and automatic speech recognition.

[0040] Each speech segment records: start time; end time; speaker identifier; transcribed content: emotional or tone characteristics; transcription confidence level.

[0041] Video data processing: For video data, the system performs shot segmentation, keyframe extraction, subtitle extraction, audio track transcription, object detection, motion recognition, and scene analysis. Based on timestamps, video frames, subtitles, audio transcription, and motion events are aligned to form temporal video content segments.

[0042] In step 104, during the specific process of generating a unified multimodal content fragment, the preprocessing results of each modality are converted into a unified format multimodal content fragment.

[0043] Each multimodal content segment includes:

[0044] ContentFragment={fragment id ,task id ,source id modality type ,raw content_reference ,processed content ,position information ,semantic vector parsing confidence business object_candidate ,relation links[] ,processing version}

[0045] The location description method is determined based on the modality type: text uses page numbers, paragraph numbers, and character ranges; images use page numbers and bounding box coordinates; tables use worksheet, row, column, and cell coordinates; audio uses start and end times; and video uses time segments, frame numbers, and screen areas. By standardizing the content fragment format, data from different modalities can enter the same subsequent processing flow.

[0046] In step 2:

[0047] In step 201: Establishing cross-modal content fragment associations; cross-modal associations are established between unified content fragments. Association methods include: spatial location association; temporal location association; document structure association; business entity association; semantic similarity association; contextual co-occurrence association; and data source association.

[0048] For example, in scanned contracts, the area of ​​the seal image is spatially associated with the names of the signatories and the signing date nearby; in video data, a keyframe is temporally associated with the speech-to-text and subtitles within the same time period; and in the main body of the contract and the attached tables, the same contract number, subject name, or amount fields are associated with entities.

[0049] For any two content fragments F i and F j The system calculates the correlation score:

[0050] A ij =λ1A position +λ 2Atime+ λ 3Aentity +λ 4Asemantic +λ 5Acontext Where λ1-λ5 are weighting coefficients.

[0051] When the association score exceeds a preset association threshold, the system establishes an association edge between the two content segments. Multiple content segments and their association edges constitute a multimodal content association graph, which is used for subsequent joint semantic understanding.

[0052] Step 202: Determine the semantic understanding objective and construct the model input context. Specifically, the content understanding objective is determined based on the task type, the rule domain to be matched, and the business object. The understanding objective may include: identifying the business entity, extracting the amount, time, or quantity, judging business behavior, identifying risk events, judging the completeness of materials, extracting compliance elements, and determining the facts required for rule conditions.

[0053] Using the understanding of the target or the problem to be judged as query information, relevant content fragments are selected from the multimodal content association graph.

[0054] For long documents, long audio, or long video, a hierarchical context management strategy is adopted: first, a local segment window is built; then, similar segments are aggregated according to business objects; chapter-level or time-time-level summaries are generated; and finally, a global business context is formed.

[0055] When a fragment contains high-risk labels, key business entities, low-confidence information, or potential rule conditions, the system marks the fragment as a reserved fragment and continues to carry it in subsequent inference windows.

[0056] In step 203, during the process of invoking the multimodal big model for joint semantic understanding, the understanding objective, multimodal content fragments, fragment relationships, business context, and output format constraints are input into the multimodal big model. The multimodal big model uses a cross-modal attention mechanism to jointly model text, images, tables, audio, and video content, outputting structured semantic understanding results. The output includes: business entities, entity attributes, business events, entity relationships, behavioral intentions, risk labels, compliance elements, temporal and spatial constraints, uncertain information, and corresponding evidence fragments.

[0057] The structured semantic understanding result is represented as follows:

[0058] SemanticResult={business objects[] ,entities[],attributes[],events[],relations[],intentions[],risk labels[] compliance elements[] ,contextual constraints[] evidence references[] ,uncertainties[]}

[0059] To reduce errors caused by the free generation of large models, the system requires that each key conclusion in the model output must be associated with at least one original content fragment identifier.

[0060] In step 204: generating structured business evidence units, the semantic understanding results are converted into structured business evidence units based on the business semantic model, business field dictionary, and rule evidence template.

[0061] Each business evidence unit includes:

[0062] EvidenceUnit={evidence id ,task id business object evidence category,subject,predicate,object,normalized value ,value type measurement unit ,sourcef ragment_ids[] ,source positions[] model confidence ,source reliabilit y,fused confidence support relations[] ,conflict relations[] evidence status}

[0063] For example, if the system identifies "payment cycle is 45 days" from the main text of the contract and "payment period is 45 days" from the attached table, it will normalize the two items into payment cycle evidence with a standard value of 45 days and retain the two original sources.

[0064] The system standardizes values ​​such as amount, date, term, percentage, name, and business category. For example:

[0065] Convert “five million yuan” to the numerical value 5,000,000; convert “within one month” to the standard period of 30 days or the business-defined period; convert different date formats to a unified date format; map synonymous business terms to unified field names.

[0066] In step 3:

[0067] In the process of performing multi-source evidence fusion without being subject to 301 restrictions, when multiple content fragments support the same business fact, evidence fusion is performed. If multiple evidence values ​​are consistent, the fusion confidence score is calculated based on the identification confidence score, source reliability, and modal weight of each piece of evidence. ; where: C i Let Q be the confidence level for identifying the i-th piece of evidence; i For the reliability of the source of evidence; M i The weights for the corresponding modes; C f The confidence level of the fused evidence.

[0068] If the evidence values ​​from multiple sources are inconsistent, instead of directly selecting any value, an evidence conflict relationship is established, and the conflict field, conflict value, source location, and confidence level are recorded before proceeding to the subsequent secondary verification process.

[0069] In step 302: the specific process of performing evidence quality, integrity, and consistency verification.

[0070] Perform the following checks on business evidence: whether the data type is valid, whether the field format is correct, whether the value is within a reasonable range, whether the units are consistent, whether the original content can support the evidence conclusion, whether the multimodal evidence is consistent, whether the information of the same business object matches, whether the necessary evidence required by the rule is complete, and whether the confidence level of the evidence meets the usage requirements.

[0071] For evidence that fails verification, the system performs different processing based on the anomaly type: when the OCR confidence is low, image enhancement and OCR recognition are performed again; when table fields are misaligned, the relationship between the table header and cells is re-identified; when the audio transcription confidence is low, noise reduction and transcription are performed again; when the context is insufficient, the adjacent content window is expanded; when there is conflict among multiple sources of evidence, a large model is called to perform local secondary reasoning; when key materials are missing, a request for supplementary materials is generated; evidence that still cannot be confirmed after secondary processing is marked as "pending confirmation" or "invalid".

[0072] In step 303: During the process of obtaining and parsing the applicable business rules, the applicable rule scope is determined from the rule base based on the business type, rule domain, business object, region, organization, time, and authority scope in the task. Business rules can originate from corporate policies, approval standards, risk control requirements, compliance standards, or operational procedures.

[0073] Represent the rules as structured rule objects:

[0074] RuleObject={rule id ,rule name ,rule type business scope ,applicable objects[] ,conditions[],condition logic ,required evidence[] exception conditions[] priority, dependency rules[] exclusion rules[] ,contained rules[] execution action effective time ,expiry time version, semantic vector ,status}

[0075] The system first excludes rules that are invalid, inapplicable to the business scope, or have mismatched permission scope.

[0076] In step 304: the specific process of constructing a rule matching request, a rule matching request is generated based on the verified business evidence.

[0077] Rule matching requests include:

[0078] RuleMatchRequest={task id business type business object normalized evidence[] risk labels[] ,contextual constraints[] ,rule domain ,expected action_type minimum confidence}

[0079] It also generates simplified semantic summaries based on business evidence for use in rule-based semantic vector recall.

[0080] In step 4:

[0081] In step 401: During the specific process of performing the first-level rule pre-screening, rapid pre-screening is carried out based on business type, business object, rule label, risk label, entity category, and key fields.

[0082] For example, in contract review tasks, the system first retains rules related to contract approval, payment terms, signing status, and performance risks, while excluding rules related to customer service quality inspection or invoice reimbursement.

[0083] The first-level pre-screening is completed using an inverted index, a tag index, or a structured query, outputting a set of first-candidate rules. If the set of first-candidate rules is empty, but high-risk tags exist in the business evidence, the system triggers a rule missing detection process.

[0084] In step 402: During the specific process of performing the second-level rule semantic recall, the business evidence summary is converted into a business semantic vector, and similarity is calculated between it and the rule semantic vector.

[0085] Semantic similarity is calculated using cosine similarity: Among them, V e V is the semantic vector of business evidence. r This is the semantic vector of the rule. Rules with similarity higher than a preset recall threshold are selected, or the top K rules are selected based on similarity ranking to form the second candidate rule set.

[0086] In step 403: During the specific process of performing the third-level conditional fine matching, the conditions in the candidate rules are parsed one by one, and evidence corresponding to each rule condition is searched from the business evidence.

[0087] For each rule condition, determine: whether the field exists, whether the data type is consistent, whether the operation conditions are met, whether the context constraints are met, whether the confidence level of the evidence meets the standard, whether there are any exception clauses, and whether there is any conflicting evidence.

[0088] The status of rule conditions can be divided into: satisfied, not satisfied, insufficient evidence, conflicting evidence, inapplicable, and pending confirmation.

[0089] If all the necessary conditions of a rule are met, it is included in the high-level candidate set; if some necessary conditions are missing, it is marked as a weak match; if there are necessary conditions that are not explicitly met, it is marked as a miss.

[0090] In step 404, during the detailed reordering of candidate rules, the candidate rule text, rule conditions, business evidence summary, and evidence source are combined into rule-evidence input pairs, which are then input into the cross-coding model for reordering. The cross-coding model outputs rule relevance scores, used to identify rules with similar themes but inapplicable conditions, as well as rules with different keywords but the same business meaning. After reordering, the system retains a preset number of highly relevant candidate rules for the deterministic execution and inference verification stages.

[0091] In step 405: the specific process of performing rule condition calculation and large model auxiliary verification; for deterministic conditions such as numerical comparison, date comparison, enumeration judgment, and logical combination, the rule engine is called to perform calculation.

[0092] For example: Contract amount > 3 million yuan AND payment period > 30 days;

[0093] For conditions involving implicit natural language conditions, exception clauses, context dependencies, or cross-modal joint judgments, the large model is invoked for verification. The large model verification must output: the rule condition being verified, the judgment result, and the evidence identifiers cited.

[0094] Reasoning for the judgment; explanation of uncertainties; whether manual review is recommended. Large model validation results serve as an auxiliary factor in rule condition judgment and do not directly override the deterministic calculation results in high-risk rules.

[0095] In step 5:

[0096] In step 501: the specific process of calculating the comprehensive matching score of the rules, a comprehensive matching score is calculated for each candidate rule:

[0097] S=w1S semantic +w2S field +w3S condition +w4S coverage +w5S evidence +w6S scope+w7Shistory;

[0098] Wherein: S semantic S represents semantic similarity. field For field consistency; S condition S represents the degree of condition satisfaction. coverage Necessary evidence coverage; S evidence For the credibility of evidence; S scope For consistency of application scope; S history The historical performance is scored; w1 to w7 are weighting parameters.

[0099] The required evidence coverage ratio is expressed as:

[0100] Among them, I j Indicates the first j Does the necessary evidence exist? If it exists, assign a value of 1; if it does not exist, assign a value of 0. j This indicates the importance weight of the evidence.

[0101] In step 502: the specific process of determining the rule matching status, the rule matching status is determined based on the comprehensive matching score, the fulfillment of necessary conditions, and the evidence status.

[0102] In one implementation: if the overall score is not lower than the first threshold, all necessary conditions are met and there is no conflict, it is determined as "hit"; if the overall score is lower than the first threshold but not lower than the second threshold, and there are conditions to be confirmed, it is determined as "high probability hit".

[0103] If some necessary conditions are met but the evidence is incomplete, it is identified as a "weak match"; if necessary evidence is missing, it is identified as "evidence to be supplemented"; if there is a clear failure to meet necessary conditions, it is identified as a "missed match". The matching threshold is configured according to the business risk level and rule type, with higher-risk rules using stricter matching thresholds.

[0104] In step 503: During the multi-rule conflict detection process, when multiple rules are simultaneously hit or have a high probability of being hit, the system analyzes the relationships between the rules. Conflict detection includes at least: execution action conflicts, conditional logic conflicts, priority conflicts, scope of application conflicts, rule version conflicts, prerequisite dependency conflicts, inclusion or duplication conflicts, and exception clause conflicts.

[0105] For example, if one rule requires "automatic approval" and another rule requires "transfer to manual review", this constitutes an action conflict; when both general rules and special rules are matched, the system determines the execution result based on the priority of special rules; when both new and old version rules are matched, only the currently valid version is retained.

[0106] In the specific process of resolving rule conflicts in the "No 504" rule, rule conflicts are handled in the following order: excluding invalid or expired rules; verifying the scope of application of the rule; verifying whether the preceding rules are valid; special rules take precedence over general rules; high-priority rules take precedence over low-priority rules; new version rules take precedence over the replaced old version rules; rules with more sufficient evidence take precedence; high-risk actions override low-risk actions. When a conflict cannot be automatically resolved according to preset logic, a conflict description is generated, and secondary reasoning of the large model or manual review is triggered.

[0107] In step 505: During the specific process of performing reverse verification, based on the final execution action to be output, reverse check whether the action has sufficient rule basis and evidence support.

[0108] For example, if "automatic rejection" is to be executed, the system performs reverse verification: whether there is a valid rejection rule; whether all the necessary conditions of the rejection rule are met; whether there is original evidence supporting the rejection; whether there is contrary evidence; whether there is an exception rule with higher priority; and whether the rule matching confidence level reaches the automatic rejection threshold.

[0109] If the reverse verification fails, the system will cancel the automatic execution and proceed to supplementary verification or manual review.

[0110] In step 6,

[0111] In step 601, during the process of determining the specific actions to be executed in the tiered business operations, the final execution action is determined by comprehensively considering the rule matching status, business risk level, evidence completeness, conflict status, and overall confidence level. The execution strategy includes:

[0112] High confidence level, complete evidence, no conflict: automatic approval, automatic rejection, or direct execution;

[0113] Medium to high confidence level, non-critical evidence missing: risk warning or condition met; Medium confidence level, necessary evidence missing: request for supplementary materials;

[0114] Low confidence level, conflicting evidence: re-analyze or transfer to manual review;

[0115] High risk and significant conflict: Suspend business processing and trigger risk alert.

[0116] In step 602: the specific process of generating interpretable rule matching results; generating interpretable results for each hit rule, including at least:

[0117] Rule number, rule name, rule version, rule hit status, overall matching score, sub-matching score, met conditions, unmet conditions, conditions to be confirmed, list of associated evidence, evidence modality type, location of original evidence, rule conflict status, conflict resolution process, final execution action, and overall confidence level.

[0118] Precise location information is provided based on the original evidence modality: text evidence is located to page number, paragraph, and character range; image evidence is located to page number and bounding box; table evidence is located to worksheet and cell; audio evidence is located to timestamp and speaker.

[0119] Video evidence was located down to time period, key frame, and area coordinates.

[0120] In step 603: During the specific process of generating the decision reasoning path, the rule decision path is recorded in the processing order: original multimodal data → multimodal content fragments → cross-modal association → structured semantic results → business evidence units → candidate rules → condition matching → comprehensive scoring → conflict detection → reverse verification → final business action;

[0121] Each processing node records its corresponding input, output, model version, rule version, processing time, and confidence level. This enables complete playback and auditing of any rule's hit result.

[0122] In step 7:

[0123] In step 701: the specific process of receiving feedback from manual review and business execution; for tasks transferred to manual review, requiring supplementary materials, or after business execution, collect: manual confirmation results; modified evidence values; final execution actions; false alarm or omission flags; rule conflict handling results; user feedback; and subsequent business results. Bind the feedback information with the original task, business evidence, and rule matching results to form a feedback sample.

[0124] In step 702: During the specific process of implementing optimization based on the feedback results, different processing objects are optimized according to the feedback type.

[0125] If the problem originates from multimodal parsing, adjust the image processing parameters, speech recognition parameters, or table parsing strategy.

[0126] If the problem originates from cross-modal association, then adjust the weights of location, time, entity, and semantic association.

[0127] If the problem originates from rule recall, then adjust the semantic recall threshold, the number of candidate rules, or the rule semantic vector.

[0128] If the problem stems from a mismatch of rules, adjust the scoring weights, hit thresholds, prompt word templates, or rule conditions.

[0129] If a rule is found to be missing, a suggestion to add a new rule will be generated and submitted to the rule administrator for review.

[0130] Optimization can be triggered when the cumulative sample size reaches a preset number, the system performance is below a threshold, the false alarm rate or false negative rate exceeds the limit, the rule version is updated, or a preset period is reached.

[0131] In step 703: During the specific process of performing offline verification and version release, after the model parameters, matching threshold, rule weights or prompt words are updated, the system uses historical datasets to perform offline evaluation and link replay.

[0132] Evaluation metrics include: multimodal parsing accuracy, business evidence extraction accuracy, rule recall rate, rule matching accuracy, false positive rate, false negative rate, rule conflict identification rate, explanatory evidence coverage, average response latency, and manual review rate.

[0133] When the updated solution meets the preset performance requirements, release a new version; otherwise, retain the original version and continue to optimize it.

[0134] In the specific process of saving the full-link log, the business task saves the full-link log from data access to execution completion, including: input data summary, multimodal parsing results, content fragment information, cross-modal correlation, large model input and output summary, business evidence and evidence verification results, candidate rule set, matching score, conflict detection results, reverse verification results, final execution action, manual review and feedback results, model, rule and prompt word version.

[0135] By using the task identifier, you can query and replay the complete processing of any business matter.

[0136] The technical effects achieved by this invention are as follows:

[0137] Through the above process, this invention gradually transforms multimodal business data such as original text, images, tables, audio, and video into structured business evidence with clear business meaning, original source, location coordinates, and confidence level. Based on the evidence, this invention completes multi-level recall, condition judgment, comprehensive scoring, conflict detection, and hierarchical execution of business rules.

[0138] This invention forms the following complete technical closed loop:

[0139] Multimodal business data acquisition—modal preprocessing—unified content fragment generation—cross-modal association—joint understanding of large models—structured business evidence generation—evidence verification—multi-level rule matching—rule conflict resolution—reverse verification—hierarchical decision execution—interpretable output—feedback optimization.

[0140] This process avoids directly using the natural language results of large models as input for rule execution, ensuring that each rule hit result has computable structured conditions, localizable multimodal evidence, and replayable processing paths, thereby improving the accuracy, reliability, and auditability of business rule matching and automated decision-making. Attached Figure Description

[0141] Figure 1 This is an overall flowchart of a method for understanding large model content and matching business rules for multimodal data according to the present invention. Detailed Implementation

[0142] To make the objectives and advantages of this invention clearer, the invention will be specifically described below with reference to embodiments. It should be understood that the following text is merely used to describe one or more specific embodiments of the invention and does not strictly limit the scope of protection specifically claimed by the invention.

[0143] like Figure 1 As shown, a method for understanding large model content and matching business rules for multimodal data is presented.

[0144] First, it receives business data in at least two modalities from text, images, tables, audio, and video, establishes business processing tasks, and identifies each data modality; it performs entity recognition and segmentation on text, optical character recognition and layout analysis on images, structural parsing and numerical standardization on tables, speech recognition and speaker separation on audio, and keyframe extraction and audio track transcription on video, generating multimodal content fragments in a unified format. Each fragment records the modality type, processing content, location coordinates, and confidence level.

[0145] Based on spatial location, timestamp, document structure, business entities, and semantic similarity, establish the association relationship between cross-modal content fragments to form a multimodal content association graph;

[0146] The understanding target is determined according to the task type and the applicable domain of the rules. Relevant fragments are selected from the association graph to construct the context. A multimodal big model is called to perform joint semantic understanding and output structured semantic understanding results. The results include entities, attributes, events, relationships, intentions, risk tags and at least one original content fragment identifier on which they are based.

[0147] Based on the business semantic model and evidence template, the semantic understanding results are converted into structured business evidence units. Each business evidence unit records the business object, evidence category, standardized value, value type, source fragment identifier, identification confidence, and source reliability. Consistent fusion is performed on multi-source evidence supporting the same fact, conflicting evidence is marked, and a business evidence set is formed after data format, value range, and integrity verification.

[0148] Obtain applicable business rules and parse them into structured rule objects containing conditions, required evidence, priorities, dependencies, and execution actions;

[0149] Based on the aforementioned business evidence set, a rule matching request is constructed, and the following steps are executed sequentially: a first-level rule pre-screening is performed using business type and key fields; a second-level semantic recall is performed using the similarity between the evidence semantic vector and the rule semantic vector; a third-level fine-grained matching is performed to determine the state in which the rule conditions are met; and candidate rules are cross-coded and reordered to obtain highly relevant candidate rules.

[0150] The conditions of highly relevant candidate rules are calculated. Numerical comparison and logical combination conditions are completed by the rule engine, while natural language implicit conditions or cross-modal joint judgment conditions are verified by the large model. The comprehensive matching score is calculated based on semantic similarity, field consistency, condition satisfaction, evidence coverage and evidence credibility to determine the rule hit status.

[0151] For multiple matching rules, conflict detection is performed on actions, conditions, and priorities. Conflicts are resolved according to strategies such as rule validity, priority of special rules, priority of high priority, and sufficiency of evidence. The proposed actions are then reverse-verified to determine whether their rule basis and supporting evidence are sufficient.

[0152] Based on the rule hit status, evidence integrity, and conflict resolution results, the system performs tiered business actions such as automatically approving, rejecting, requesting supplementary materials, or transferring the request to manual review, generating interpretable matching results. These results include the hit rules, matching scores, associated evidence, and the location of the evidence in the original modality.

[0153] Record the complete decision-making path from data input to action output, obtain human feedback and business execution results, use the feedback to optimize multimodal preprocessing parameters, cross-modal association weights, rule recall thresholds and matching score weights, and update and release after offline evaluation;

[0154] The structured business evidence unit constitutes an intermediate evidence layer between multimodal understanding output and rule execution, enabling rule matching to be based on computable and traceable structured evidence, rather than directly relying on the natural language output of a large model.

[0155] A method for content understanding and business rule matching of large models for multimodal data specifically includes the following steps: Step 1: 101: Receive multimodal business data and establish business processing tasks; 102: Identify data modalities and select corresponding preprocessing processes; 103: Perform submodal standardized preprocessing; 104: Generate unified multimodal content fragments; Step 2: 201: Establish cross-modal content fragment associations; 202: Determine semantic understanding objectives and construct model input context; 203: Call the multimodal large model for joint semantic understanding; 204: Generate structured business evidence units; Step 3: 301: Perform multi-source evidence fusion; 302: Perform evidence quality, integrity, and consistency verification; 303: Extract and parse applicable business rules; 304: Construct rule matching requests; Step 4: Step 401: Perform first-level rule pre-screening; Step 402: Perform second-level rule semantic recall; Step 403: Perform third-level conditional fine-grained matching; Step 404: Perform candidate rule fine-grained re-ranking; Step 405: Perform rule condition calculation and large model-assisted verification; Step 5: Calculate the rule comprehensive matching score; Step 502: Determine the rule matching status; Step 503: Perform multi-rule conflict detection; Step 504: Perform rule conflict resolution; Step 505: Perform reverse verification; Step 6: Determine the hierarchical business execution actions; Step 602: Generate interpretable rule matching results; Step 603: Generate decision reasoning paths; Step 7: Receive manual review and business execution feedback; Step 702: Implement optimization based on feedback results; Step 703: Perform offline verification and version release; Step 704: Save the entire link log.

[0156] Example 1: Multimodal content understanding and business rule matching in contract review scenarios

[0157] Step 1: Business data acquisition; The enterprise contract review system receives the contract tasks to be reviewed.

[0158] The tasks include: PDF file of the contract text; scanned images of contract attachments; Excel spreadsheet of contract amount list; audio recording of the contract signing process; video footage of contract performance; and a database of enterprise business rules.

[0159] The main body of the contract includes: the two parties to the contract; the contract amount; the payment method; the payment cycle; and the liability for breach of contract.

[0160] Terms and conditions, including the term of service.

[0161] The scanned attachment images contain: a stamped page; a business license image; bank account verification; and handwritten signature information.

[0162] The Excel spreadsheet contains: project amount; payment schedule; supplier information; and acceptance plan.

[0163] The audio recording contains: the content of the communication between the two parties; explanations of contract changes; and information on special agreements.

[0164] The video file contains: on-site delivery status; product acceptance process; personnel operation behavior.

[0165] First, create a contract review task:

[0166] Task_ID:HT_202607230001; Business_Type: Contract Risk Review; Rule_Domain: Contract Approval Rule Base; and all related data will be grouped under the same business processing task.

[0167] Step 2: Multimodal data preprocessing; automatic identification based on the type of input data.

[0168] For the contract text PDF file: The process includes: PDF parsing; page splitting; text extraction; clause segmentation; entity recognition; and keyword extraction. For example, the identified information includes: Contract amount: 5 million yuan; Payment period: 45 days; Contract parties: Company A and Company B.

[0169] For scanned attachment images: Perform: Image enhancement; OCR recognition; Seal detection; Signature area detection; Layout analysis. For example: OCR recognition result: Party A: A Technology Co., Ltd.; Party B: B Supply Chain Co., Ltd.; Signing date: [Date]; Simultaneously saved: Source: Page 3 of Contract Attachment; Location: Bounding Box (120, 260, 560, 350); Recognition confidence: 0.96;

[0170] For the Excel spreadsheet: Analysis: Project Name: Smart Device Procurement Project; Contract Amount: 5,000,000; Payment Schedule: Payment 30 days after acceptance;

[0171] For audio files: Perform: speech-to-text transcription; speaker separation; timestamp. Result: 00:03:21; Supplier: "Payment period adjusted to 45 days".

[0172] For video files: Extract: keyframes; behavior events; time segments. Obtain: Video time: 15:20-16:10; Events:

[0173] Step 3: On-site equipment acceptance completed; generate unified multimodal content fragments; convert the information from the different sources mentioned above into unified content fragments.

[0174] For example: Contract amount content fragment: Fragment_ID:F001; Modal type: Text; Content: Contract amount 5 million yuan; Source: Page 5 of the contract text; Confidence: 0.98; Image seal fragment: Fragment_ID:F002; Modal type: Image; Content: Detected company seal; Source: Page 3 of the attachment; Region: (x1,y1,x2,y2); Confidence: 0.95; Table amount fragment: Fragment_ID:F003; Modal type: Table; Field: Project amount value: 5 million yuan; Source: Sheet1-A5; Confidence: 0.97. Establish the relationship between the three fragments: F001 || Entity consistency | F003 | F002 || Business object association | Contract signing subject forms a contract business object association graph.

[0175] Step 4: Joint Understanding of Multimodal Large Models

[0176] This includes: contract text; OCR results; table fields; audio transcription; video events; and business objectives.

[0177] The common input to the multimodal large model is "to determine whether the contract meets the approval rules." The model then performs cross-modal joint reasoning.

[0178] Output: 20; Business Target: Procurement Contract Key Facts: 1. Contract Amount: 5 million RMB; Source: Text + Table; 2. Payment Cycle: 45 days; Source: Text + Audio Recording; 3. Contract Signed: Source: Image Seal Detection; 4. Equipment Accepted: Source: Video Keyframes; Simultaneously Generated: Risk Tag: Abnormal Payment Cycle Risk;

[0179] Step 5: Generate structured business evidence:

[0180] Transform the model understanding results into rule-computable evidence.

[0181] Generation: Evidence 1: Evidence_ID: E001; Field: contract_amount; Value: 5000000; Source: F001, F003

[0182] Credibility: 0.99; Evidence 2: Evidence_ID: E002; Field: payment_period; Value: 45 days; Source: F001, F004

[0183] Confidence: 0.96; Evidence 3: Evidence_ID: E003; Field: contract_signed; Value: true; Source: F002; Confidence: 0.95; All the above evidence retains: original source; data location; model inference result; confidence level.

[0184] Step 6: Loading Business Rules and Structured Modeling. Load contract approval rules from the rule base.

[0185] For example: Rule: Rule Number: R001; Rule Name: Large Contract Approval Rule; Conditions: Contract Amount > 3 million RMB; and Payment Cycle > 30 days; Action: Submit for Advanced Approval; Converted to: Rule_Object:

[0186] {rule_id:R001,conditions:amount>3000000payment_period>30,action:Advanced Approval}

[0187] Simultaneously, a rule semantic vector is generated: "large contract, high amount, long payment cycle, advanced approval" for subsequent semantic matching.

[0188] Step 7: Intelligent Rule Matching; First, perform rule pre-screening. Based on: Business type: Contract review, Risk label: Abnormal payment cycle; Recall: Candidate rules: R001, R005, R012; Then, perform semantic matching. Calculate: Contract amount evidence + Payment cycle evidence + Contract type; Match degree with rule R001: 0.97; Enter the fine-grained verification stage.

[0189] Step 8: Rule condition verification; Call the rule engine to perform deterministic judgment: Condition 1: 5 million yuan > 3 million yuan; satisfied. Condition 2: 45 days > 30 days; satisfied. Condition 3: Contract type is a procurement contract; satisfied. Therefore: Rule R001: Hit.

[0190] Step 9: Rule Conflict Detection; Continue checking other matching rules. Found: Rule R008: Payment cycle exceeds 60 days; automatic risk interception; due to: 45 days < 60 days; this rule is not satisfied. Judgment: No execution conflict exists.

[0191] Step 10: Generate decision results; Output: Review results: Rule matched: R001; Rule name: Large contract approval rule; Match confidence: 97.5%; Action: Enter advanced approval process; Simultaneously generate explanatory information: Judgment basis: 1. Contract amount: 5 million yuan; Source: Page 5 of the contract text; 2. Payment cycle: 45 days; Source: Page 8 of the contract text; 3. Payment cycle exceeds 30 days; Rule conditions are met.

[0192] Step 11: Human Feedback and Model Optimization

[0193] After senior approvers complete the review: Input feedback: Review result: Correct according to the rules; Record: Input data; Model output, rule matching process, manual result.

[0194] If long-term statistics reveal a large number of false positives for a certain type of contract, the system will automatically adjust: rule weights; matching thresholds; and semantic recall parameters.

[0195] Example 2: Bill Risk Control Review Scenario; In the bill review scenario:

[0196] Input: Invoice image, transaction contract, bank statement, and customer service recording.

[0197] Invoice identified via OCR: Amount: 2 million yuan; Contract text identified: Contract amount: 1.5 million yuan; Bank statement identified: Actual payment: 2 million yuan; The system found inconsistencies in the amounts across the three modalities.

[0198] Therefore, the following is generated: Evidence conflict: Invoice amount ≠ Contract amount; Trigger: Risk rule: Abnormal invoice amount; Execution action: Transfer to manual review; Simultaneously output: Conflict basis: Amount area on page 1 of the invoice; Amount clause on page 3 of the contract; 5th transaction in the bank statement; The risk assessment process is thus interpretable.

[0199] Through the above implementation methods, the present invention achieves the following: unified conversion of text, images, tables, audio, and video into business-computable evidence; establishment of a data bridge between multimodal content and business rules; avoidance of the traditional simple combination of "large model output + rule field matching"; support for joint judgment of cross-modal evidence; support for rule conflict detection and automatic resolution; output of interpretable decision results containing the location of original evidence; and continuous optimization of matching effect based on human feedback.

[0200] Compared to traditional manual review methods, this invention can reduce the workload of business personnel making repetitive judgments, improve the consistency of rule execution, and enhance the accuracy, reliability, and auditability of enterprise intelligent decision-making systems.

[0201] The above description is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained in this invention are implemented according to conventional methods in the art unless otherwise specified or limited.

Claims

1. A method for understanding the content of large models and matching business rules for multimodal data, characterized in that: It receives business data in at least two modalities from text, images, tables, audio, and video; establishes business processing tasks; identifies each data modality; performs entity recognition and segmentation on text; performs optical character recognition and layout analysis on images; performs structure parsing and numerical standardization on tables; performs speech recognition and speaker separation on audio; performs keyframe extraction and audio track transcription on video; and generates multimodal content fragments in a unified format. Each fragment records the modality type, processing content, location coordinates, and confidence level. Based on spatial location, timestamp, document structure, business entities, and semantic similarity, establish the association relationship between cross-modal content fragments to form a multimodal content association graph; The understanding target is determined according to the task type and the applicable domain of the rules. Relevant fragments are selected from the association graph to construct the context. A multimodal big model is called to perform joint semantic understanding and output structured semantic understanding results. The results include entities, attributes, events, relationships, intentions, risk tags and at least one original content fragment identifier on which they are based. Based on the business semantic model and evidence template, the semantic understanding results are converted into structured business evidence units. Each business evidence unit records the business object, evidence category, standardized value, value type, source fragment identifier, identification confidence, and source reliability. Consistent fusion is performed on multi-source evidence supporting the same fact, conflicting evidence is marked, and a business evidence set is formed after data format, value range, and integrity verification. Obtain applicable business rules and parse them into structured rule objects containing conditions, required evidence, priorities, dependencies, and execution actions; Based on the aforementioned set of business evidence, a rule matching request is constructed, and the following steps are executed sequentially: first-level rule pre-screening is performed using business type and key fields; Second-level semantic recall is performed using the similarity between evidence semantic vectors and rule semantic vectors; third-level fine matching is performed by judging the state of rule condition satisfaction on a condition-by-condition basis; candidate rules are cross-coded and reordered to obtain highly relevant candidate rules. The conditions of highly relevant candidate rules are calculated. Numerical comparison and logical combination conditions are completed by the rule engine, while natural language implicit conditions or cross-modal joint judgment conditions are verified by the large model. The comprehensive matching score is calculated based on semantic similarity, field consistency, condition satisfaction, evidence coverage and evidence credibility to determine the rule hit status. For multiple matching rules, conflict detection is performed on actions, conditions, and priorities. Conflicts are resolved according to strategies such as rule validity, priority of special rules, priority of high priority, and sufficiency of evidence. The proposed actions are then reverse-verified to determine whether their rule basis and supporting evidence are sufficient. Based on the rule hit status, evidence integrity, and conflict resolution results, the system performs tiered business actions such as automatically approving, rejecting, requesting supplementary materials, or transferring the request to manual review, generating interpretable matching results. These results include the hit rules, matching scores, associated evidence, and the location of the evidence in the original modality. Record the complete decision-making path from data input to action output, obtain human feedback and business execution results, use the feedback to optimize multimodal preprocessing parameters, cross-modal association weights, rule recall thresholds and matching score weights, and update and release after offline evaluation; The structured business evidence unit constitutes an intermediate evidence layer between multimodal understanding output and rule execution, enabling rule matching to be based on computable and traceable structured evidence, rather than directly relying on the natural language output of a large model.

2. The method for understanding large model content and matching business rules for multimodal data as described in claim 1. Its features are: Specifically, the following steps are included: Step 1: 101: Receive multimodal service data and establish service processing tasks; 102: Identify the data modality and select the corresponding preprocessing procedure; 103: Perform modal normalization preprocessing; 104: Generate unified multimodal content fragments; Step 2: 201: Establish cross-modal content fragment associations; 202: Define the semantic understanding objective and construct the model input context; 203: Call upon a multimodal large model for joint semantic understanding; 204: Generate structured business evidence units; Step 3: 301: Perform multi-source evidence fusion; 302: Perform verification of the quality, completeness, and consistency of evidence; 303: Obtain and parse applicable business rules; 304: Request to construct a rule to match; Step 4: 401: Perform pre-screening using the first-level rules; 402: Execute the second-level rule semantic recall; 403: Perform a level 3 conditional fine-grained match; 404: Perform fine-grained reordering of candidate rules; 405: Calculation of execution rule conditions and large-scale model-assisted verification; Step 5: 501: Calculation rule for comprehensive matching score; 502: Determines the rule matching status; 503: Perform multi-rule conflict detection; 504: Execution rule conflict resolution; 505: Perform reverse verification; Step 6: 601: Determine the tiered business execution actions; 602: Generate interpretable rule matching results; 603: Generate decision reasoning paths; Step 7: 701: Receive manual review and business execution feedback; 702: Implement optimizations based on feedback results; 703: Perform offline verification and version release; 704: Save full-link logs.

3. The method for understanding large model content and matching business rules for multimodal data according to claim 1, characterized in that: In step 1: In step 101: Receiving multimodal service data and establishing a specific service processing task, The multimodal business data includes one or more of the following: text files, scanned images, spreadsheets, audio files, video files, and structured business fields. The multimodal business data originates from contract systems, invoice systems, approval systems, work order systems, customer service systems, knowledge bases, multimedia acquisition devices, or external business interfaces. Upon receiving the business data, a unique task identifier is generated for this processing step, and the following task attributes are obtained: business type, business object identifier, data source, file or data source identifier, applicable rule area, user permission scope, task creation time, and business context information. When multiple files or multiple modal data belong to the same business matter, they are merged into the same business processing task based on business number, contract number, work order number, personnel identifier, organization identifier, time range, or file association. In step 102: Identifying data modes and selecting the corresponding preprocessing steps: The modality type of the input data is identified based on the file extension, file header information, media encoding, data structure, and content characteristics. When a single file contains multiple modal contents, the system identifies it as a composite multimodal file and splits it into text areas, image areas, table areas, audio tracks, video frames, and subtitle information. The corresponding preprocessing components are called according to the modality type: text data enters the text parsing process, images or scanned documents enter the image parsing process, table data enters the table structure parsing process, audio data enters the speech processing process, and video data enters the video time series analysis process. For unrecognized data types, mark them as unknown modalities and call the general parsing component or transfer them to manual confirmation based on the file content; In step 103: the preprocessing for modal normalization includes text data processing, image data processing, tabular data processing, audio data processing, and video data processing; In step 104: the specific process of generating a unified multimodal content fragment, the preprocessing results of each modality are converted into a unified format multimodal content fragment; The location description method is determined according to the modality type: text uses page numbers, paragraph numbers, and character ranges; images use page numbers and bounding box coordinates; tables use worksheet, row, column, and cell coordinates; audio uses start and end times; and video uses time periods, frame numbers, and screen areas. By unifying the content fragment format, data from different modalities can enter the same subsequent processing flow.

4. The method for understanding large model content and matching business rules for multimodal data according to claim 1, characterized in that: In step 2: In step 201: Establishing cross-modal content fragment associations; cross-modal associations are established between unified content fragments; association methods include: spatial location association; temporal location association; document structure association; business entity association; semantic similarity association; contextual co-occurrence association; data source association; When the association score is greater than the preset association threshold, the system establishes an association edge between two content segments; multiple content segments and their association edges constitute a multimodal content association graph, which is used for subsequent joint semantic understanding. In step 202: Determine the semantic understanding objective and construct the model input context. Specifically, the content understanding objective is determined based on the task type, the rule domain to be matched, and the business object. The understanding objective may include: identifying the business entity, extracting the amount, time, or quantity, judging the business behavior, identifying risk events, judging the completeness of materials, extracting compliance elements, and determining the facts required for the rule conditions. Using the understanding of the target or the problem to be judged as query information, relevant content fragments are selected from the multimodal content association graph; For long documents, long audio, or long video, a hierarchical context management strategy is adopted: first, a local segment window is built; then, similar segments are aggregated according to business objects; chapter-level or time-time-level summaries are generated; finally, a global business context is formed. In step 203: the specific process of calling the multimodal big model for joint semantic understanding; the understanding objective, multimodal content fragments, fragment relationships, business context, and output format constraints are input into the multimodal big model; the multimodal big model uses a cross-modal attention mechanism to jointly model text, images, tables, audio, and video content, and outputs structured semantic understanding results; the output content includes: business entities, entity attributes, business events, entity relationships, behavioral intentions, risk labels, compliance elements, time and space constraints, uncertain information, and corresponding evidence fragments; In step 204: the specific process of generating structured business evidence units, the semantic understanding results are converted into structured business evidence units based on the business semantic model, business field dictionary and rule evidence template; The system standardizes values ​​such as amount, date, term, percentage, name, and business category.

5. The method for understanding large model content and matching business rules for multimodal data according to claim 1, characterized in that: In step 3: In step 301: During the multi-source evidence fusion process; when multiple content fragments support the same business fact, evidence fusion is performed on them; if multiple evidence values ​​are consistent, the fusion confidence score is calculated based on the identification confidence score, source reliability, and modal weight of each piece of evidence. ; where: C i Let Q be the confidence level for identifying the i-th piece of evidence; i For the reliability of the source of evidence; M i The weights for the corresponding modes; C f The confidence level of the fused evidence; If the evidence values ​​from multiple sources are inconsistent, instead of directly selecting any value, an evidence conflict relationship is established, and the conflict field, conflict value, source location, and confidence level are recorded before proceeding to the subsequent secondary verification process. Step 302: In the specific process of performing evidence quality, integrity, and consistency verification, Perform the following checks on business evidence: whether the data type is valid, whether the field format is correct, whether the value is within a reasonable range, whether the units are consistent, whether the original content can support the evidence conclusion, whether the multimodal evidence is consistent, whether the information of the same business object matches, whether the necessary evidence required by the rule is complete, and whether the confidence level of the evidence meets the usage requirements. For evidence that fails verification, the system performs different processing based on the anomaly type: when the OCR confidence is low, image enhancement and OCR recognition are performed again; when table fields are misaligned, the relationship between the table header and cells is re-identified; when the audio transcription confidence is low, noise reduction and transcription are performed again; when the context is insufficient, the adjacent content window is expanded; when there is conflict among multiple sources of evidence, a large model is called to perform local secondary inference; when key materials are missing, a request for supplementary materials is generated; evidence that still cannot be confirmed after secondary processing is marked as "pending confirmation" or "invalid". In step 303: the specific process of obtaining and parsing applicable business rules; based on the business type, rule domain, business object, region, organization, time and permission scope in the task, the applicable rule scope is determined from the rule base; business rules may come from corporate policies, approval specifications, risk control requirements, compliance standards or operating procedures; In step 304: the specific process of constructing a rule matching request, a rule matching request is generated based on the verified business evidence; It can also generate simplified semantic summaries based on business evidence for use in rule-based semantic vector retrieval.

6. The method for understanding large model content and matching business rules for multimodal data according to claim 1, characterized in that: In step 4: In step 401: During the specific process of performing the first-level rule pre-screening, rapid pre-screening is carried out based on business type, business object, rule label, risk label, entity category, and key fields. The first-level pre-screening is completed using an inverted index, a tag index, or a structured query, and outputs the first candidate rule set. If the first candidate rule set is empty, but there are high-risk tags in the business evidence, the system triggers the rule missing detection process. In step 402: During the specific process of executing the second-level rule semantic recall, the business evidence summary is converted into a business semantic vector, and similarity is calculated between it and the rule semantic vector; Semantic similarity is calculated using cosine similarity: ; Among them, V e V is the semantic vector of business evidence. r The rule semantic vector is used; rules with similarity higher than the preset recall threshold are selected, or the top K rules are selected according to similarity to form the second candidate rule set; In step 403: During the specific process of performing the third-level conditional fine matching, the conditions in the candidate rules are parsed one by one, and the evidence corresponding to each rule condition is searched from the business evidence; For each rule condition, determine: whether the field exists, whether the data type is consistent, whether the operation conditions are met, whether the context constraints are met, whether the confidence level of the evidence meets the standard, whether there are any exception clauses, and whether there is any conflicting evidence. The status of rule conditions can be divided into: satisfied, not satisfied, insufficient evidence, conflicting evidence, inapplicable, and pending confirmation; If all the necessary conditions of a rule are met, it is included in the high-level candidate set; if some necessary conditions are missing, it is marked as a weak match; if there are necessary conditions that are not explicitly met, it is marked as a miss. In step 404: the specific process of performing fine reordering of candidate rules, the candidate rule text, rule conditions, business evidence summary and evidence source are combined into a rule-evidence input pair, which is then input into the cross-coding model for reordering; the cross-coding model outputs a rule relevance score, which is used to identify rules with similar themes but inapplicable conditions, as well as rules with different keywords but the same business meaning; after reordering, the system retains a preset number of highly relevant candidate rules to enter the deterministic execution and inference verification stage; In step 405: the specific process of performing rule condition calculation and large model auxiliary verification; for deterministic conditions such as numerical comparison, date comparison, enumeration judgment, and logical combination, the rule engine is called to perform calculation; For conditions involving implicit conditions in natural language, exception clauses, context dependencies, or cross-modal joint judgments, the large model is invoked for verification. When the large model is verified, it must output: the rule condition being verified, the judgment result, and the evidence identifier cited. Reasons for judgment; explanation of uncertainty; whether manual review is recommended; large model validation results serve as auxiliary factors for rule condition judgment and do not directly cover the deterministic calculation results in high-risk rules.

7. The method for understanding large model content and matching business rules for multimodal data according to claim 1, characterized in that: In step 5: In step 501: the specific process of calculating the comprehensive matching score of the rules, a comprehensive matching score is calculated for each candidate rule: In step 501: the specific process of calculating the comprehensive matching score of the rules, a comprehensive matching score is calculated for each candidate rule: S=w1S semantic +w2S field +w3S condition +w4S coverage +w5S evidence +w6S scope +w7Shistory; Wherein: S semantic S represents semantic similarity. field For field consistency; S condition S represents the degree of condition satisfaction. coverage Necessary evidence coverage; S evidence For the credibility of evidence; S scope For consistency of application scope; S history The historical performance is scored; w1 to w7 are the weighting parameters; The required evidence coverage ratio is expressed as: Among them, I j Indicates the first j Does the necessary evidence exist? If it exists, assign a value of 1; if it does not exist, assign a value of 0. j This indicates the importance weight of the evidence; In step 502: the specific process of determining the rule matching status, the rule matching status is determined based on the comprehensive matching score, the fulfillment of necessary conditions, and the evidence status; In step 503: During the multi-rule conflict detection process; when multiple rules are hit simultaneously or with a high probability of being hit, the system analyzes the relationship between the rules; conflict detection includes at least: conflict of execution actions, conflict of conditional logic, conflict of priority, conflict of applicable scope, conflict of rule versions, conflict of prerequisite dependencies, conflict of inclusion or duplication, and conflict of exception clauses. In the specific process of resolving rule conflicts in the "No Thick 504" rule, rule conflicts are handled in the following order: exclude invalid or ineffective rules; verify the scope of application of the rules; verify whether the preceding rules are valid; special rules take precedence over general rules; high-priority rules take precedence over low-priority rules; new version rules take precedence over the replaced old version rules; rules with more sufficient evidence take precedence; high-risk actions cover low-risk actions; when a conflict cannot be automatically resolved according to the preset logic, a conflict description is generated, and a second reasoning of the large model or manual review is triggered. In step 505: During the specific process of performing reverse verification, based on the final execution action to be output, reverse check whether the action has sufficient rule basis and evidence support.

8. The method for understanding large model content and matching business rules for multimodal data according to claim 1, characterized in that: In step 6, In step 601: During the process of determining the specific execution actions for tiered business operations; considering the rule matching status, business risk level, evidence completeness, conflict status, and overall confidence level, the final execution action is determined; the execution strategy includes: High confidence level, complete evidence, no conflict: automatic approval, automatic rejection, or direct execution; Medium to high confidence level, non-critical evidence missing: risk warning or condition met; Medium confidence level, necessary evidence missing: request for supplementary materials; Low confidence level, conflicting evidence: re-analyze or transfer to manual review; High risk and significant conflict: Suspend business processing and trigger risk alert; In step 602: the specific process of generating interpretable rule matching results; generating interpretable results for each hit rule, including at least: Rule number, rule name, rule version, rule hit status, overall matching score, sub-item matching score, met conditions, unmet conditions, conditions to be confirmed, list of associated evidence, evidence modality type, location of original evidence, rule conflict status, conflict resolution process, final execution action, overall confidence level; Precise location information is provided based on the original evidence modality: text evidence is located to page number, paragraph, and character range; image evidence is located to page number and bounding box; table evidence is located to worksheet and cell; audio evidence is located to timestamp and speaker. Video evidence was located down to time period, key frame, and region coordinates; In step 603: During the specific process of generating the decision reasoning path, the rule decision path is recorded in the processing order: original multimodal data → multimodal content fragments → cross-modal association → structured semantic results → business evidence units → candidate rules → condition matching → comprehensive scoring → conflict detection → reverse verification → final business action; Each processing node records its corresponding input, output, model version, rule version, processing time, and confidence level.

9. The method for understanding large model content and matching business rules for multimodal data according to claim 1, characterized in that: In step 7: In step 701: the specific process of receiving feedback from manual review and business execution; for tasks transferred to manual review, supplementary materials, or after business execution, collect: manual confirmation results; modified evidence values; final execution actions; false alarm or omission flags; and rule conflict handling results; User feedback; subsequent business results; binding feedback information with original tasks, business evidence, and rule matching results to form feedback samples; In step 702: During the specific process of implementing optimization based on the feedback results, different processing objects are optimized according to the feedback type; Optimization can be triggered when the cumulative sample size reaches a preset number, the system performance is below a threshold, the false alarm rate or false negative rate exceeds the limit, the rule version is updated, or a preset period is reached. In step 703: During the specific process of performing offline verification and version release, after the model parameters, matching threshold, rule weight or prompt words are updated, the system uses historical datasets to perform offline evaluation and link replay. Evaluation metrics include: multimodal parsing accuracy, business evidence extraction accuracy, rule recall rate, rule matching accuracy, false positive rate, false negative rate, rule conflict identification rate, explanatory evidence coverage, average response latency, and manual review rate. When the updated solution meets the preset performance requirements, release a new version; otherwise, retain the original version and continue to optimize it. In the specific process of saving the full-link log, the business task saves the full-link log from data access to execution completion, including: input data summary, multimodal parsing results, content fragment information, cross-modal correlation, large model input and output summary, business evidence and evidence verification results, candidate rule set, matching score, conflict detection results, reverse verification results, final execution action, manual review and feedback results, model, rule and prompt word version; By using the task identifier, you can query and replay the complete processing of any business matter.