Intelligent collaborative execution method and system of multi-modal rule fusion dataset and medium

Abstract

This invention discloses an intelligent collaborative execution method, system, and medium for multimodal rule fusion datasets. The method includes: obtaining an execution plan containing rule structure hashes and quality fingerprints; obtaining the current structure hash of the target rule from a knowledge base and performing version verification with the hash in the execution plan; if inconsistent, loading the latest rule and triggering dynamic dependency reconstruction; calculating the data source priority score based on the quality fingerprint and employing a dynamic weight allocation mechanism; selecting the highest-scoring data source as the primary data source and initiating a four-level degradation mechanism; when rule conclusions conflict, calculating a score by comprehensively considering priority, quality score, and visual confidence, and outputting the highest-scoring rule. This invention achieves rule version consistency assurance, quality-aware scheduling, and multimodal rule fusion reasoning, improving execution accuracy and system robustness.

Description

Technical Field

[0001] This invention relates to the fields of data processing and artificial intelligence technology, and in particular to an intelligent collaborative execution method, system and medium for multimodal rule fusion datasets. Background Technology

[0002] In highly regulated fields such as financial risk control, medical quality control, industrial quality inspection, government approval, and customs supervision, business decisions often require combining domain rules with data queries. However, existing technologies have the following shortcomings when handling such complex requirements: Text-to-SQL solutions (such as TaBERT and BRIDGE) can only generate simple database queries and cannot handle multiple system calls and rule dependencies; general agent solutions (such as ReAct and AutoGPT) rely on static configuration for tool calls and lack dynamic awareness of data source quality; traditional rule engines (such as Drools) only support IF-THEN rules and cannot handle visual rules and time-series rules; large language model solutions (such as GPT-4) suffer from illusion problems in specialized domains and cannot guarantee rule version consistency.

[0003] In summary, existing solutions fail to achieve deep integration of rule execution and data acquisition, version consistency assurance, and multimodal input support. Therefore, an intelligent collaborative execution method is needed that can fully utilize deep information in rule fusion datasets (such as rule structure hashes, quality fingerprints, and relationships between rules). Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by providing an intelligent collaborative execution method for multimodal rule fusion datasets. This method achieves an end-to-end intelligent closed loop from user natural language input to multi-system collaborative invocation, and can be enhanced in conjunction with large language models to improve decision-making credibility and deep analysis capabilities. The core steps include:

[0005] (1) Obtain the execution plan, which includes the rule identifier and the corresponding rule structure hash, as well as the quality fingerprint of the data source; when the user input includes an image or video, the execution plan also includes dynamically generated visual processing parameters.

[0006] (2) Obtain the current structure hash of the target rule from the rule knowledge base and perform dynamic version verification with the structure hash in the execution plan (real-time comparison in online mode, and verification based on local cache timeliness mark in offline mode).

[0007] (3) If the structure hash is inconsistent, the latest rule content is loaded from the knowledge base to replace the rule content in the execution plan, and inference is performed based on the latest rule; when a rule is marked as obsolete, dynamic reconstruction of the dependency relationship is triggered: recursively retrieve candidate rules that depend on the rule, and search for alternative rules through semantic matching algorithm (weighted cosine similarity of rule text embedding and syntax tree similarity).

[0008] (4) When performing data query, the priority scores of multiple candidate data sources are calculated based on the quality fingerprint. A dynamic weight allocation mechanism is adopted to automatically adjust the weights of quality score, source credibility, timeliness, etc. according to the business scenario. For visually dependent scenarios, a visual quality decay coefficient is introduced.

[0009] (5) Select the data source with the highest priority score as the primary data source for invocation, and activate the four-level data source degradation mechanism to automatically switch to the secondary data source. At the same time, manage the blacklist dynamically based on historical failure rate and real-time load. In addition, when multiple rule conclusions conflict, calculate the comprehensive score by combining rule priority, quality score, and visual detection confidence, and select the rule with the highest score as the final output.

[0010] The present invention also provides an intelligent collaborative system and a computer-readable storage medium for implementing the above method. Beneficial effects

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0012] 1. Rule version consistency guarantee

[0013] By using a full-process verification mechanism based on rule structure hashes, the execution plan is compared with the structure hash in the rule knowledge base before rule inference is executed. If there is a discrepancy, the latest rule content is automatically loaded, ensuring that inference is always based on the latest valid version of the rule. This solves the problems of cache invalidation and version inconsistency after rule updates in traditional solutions.

[0014] 2. Dynamic Dependency Restructuring and Fault Tolerance

[0015] When a rule is marked as invalid, dependency reconstruction is automatically triggered. Alternative rules are searched based on semantic matching, enabling adaptive updates of the execution plan after rule changes. This avoids interruption of the overall inference due to the invalidation of a single rule, thus improving the robustness and maintainability of the system.

[0016] 3. Quality-aware data source scheduling

[0017] The call priority is dynamically calculated based on the quality fingerprint of the data source. When the main data source call fails, a multi-level degradation mechanism is activated to automatically switch to the best available data, ensuring the accuracy of the inference results and the stability of the system even when the data source quality is uneven or some data sources are faulty.

[0018] 4. Multimodal rule fusion reasoning

[0019] It supports multimodal inputs such as text, images, and videos. By dynamically generating visual processing parameters, it integrates visual detection results and text rules into the reasoning process, solving the problem that traditional rule engines cannot handle visual rules and achieving end-to-end fusion of physical world visual states to rule decisions.

[0020] 5. Multidimensional Conflict Resolution and Self-Optimization

[0021] When multiple rules are hit simultaneously and their conclusions conflict, the optimal output is dynamically determined by combining rule priority, quality fingerprint, and visual detection confidence. The conflict resolution strategy can be continuously optimized based on user feedback, enabling conflict resolution to adapt to business changes and improve the accuracy of multi-rule collaborative reasoning.

[0022] 6. Incremental learning and continuous evolution

[0023] The execution strategy model is incrementally updated based on user feedback data, and the rule confidence and execution parameters are adjusted according to the feedback. This enables the system to continuously learn and optimize from actual operation, and gradually improve the accuracy of execution plan generation and inference effect.

[0024] 7. Differentiated resource scheduling

[0025] Based on the difficulty level information of the execution plan, differentiated computing resources or scheduling priorities are allocated to different tasks to achieve flexible scheduling that enables rapid response for simple tasks and ensures accuracy for complex tasks, thereby optimizing the overall resource utilization. Attached Figure Description

[0026] Figure 1 This is a system architecture diagram of the present invention;

[0027] Figure 2 This is a flowchart illustrating the execution plan generation and execution process of the present invention;

[0028] Figure 3 This is a schematic diagram of the version consistency guarantee mechanism of the present invention;

[0029] Figure 4 This is a graph showing the data source quality-aware invocation and dependency analysis of the present invention.

[0030] Figure 5 This is a flowchart of the rule-based reasoning and data fusion process of the present invention;

[0031] Figure 6 This is a flowchart of the incremental learning and self-evolution process of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] S1. Definition and Construction of Multimodal Fusion Rule Dataset (QIR-KG)

[0034] The core unit of QIR-KG is as follows:

[0035] Natural Language Request Unit (Q Unit): The core content is the original user input and query type; the multimodal extension is to add multimodal input references (such as image URLs and video keyframe indexes).

[0036] Intent and Instruction Unit (I Unit): The core content includes intent types and parameters, data acquisition instructions (such as SQL instructions); multimodal extensions include adding multimodal task instructions (such as visual inspection and visual measurement).

[0037] Rule logic expression unit (R unit): The core content is a three-layer abstract JSON structure, including rule source, rule structure hash, relationship between rules, and association type; the multimodal extension is a standardized encapsulation of newly added visual rules, which shares the same three-layer abstract structure as text rules.

[0038] Knowledge Graph Embedding Unit (KG Unit): The core content is text entity ID, type, and vector representation; the multimodal extension is to add a fusion vector representation of visual entities, which is a weighted concatenation of visual entity features and text entity features.

[0039] Output results (output_result): The core content is structured output, including results, confidence scores, reasoning, and rule references; multimodal expansion adds visual evidence fields (such as keyframe URLs and detection box coordinates).

[0040] Metadata: The core content includes domain, difficulty, quality fingerprint, data source, association rule identifier, and update log; multimodal expansion adds visual quality dimensions (clarity, illumination, occlusion, etc.).

[0041] Construction Notes: Each QIR-KG data record can be generated in any way. Entities and attributes fill the kg_layer; the three-layer rule structure and rule structure hash fill the r_layer; the user's original request and multimodal input references fill the q_layer; intent recognition results and data acquisition instructions fill the i_layer; output_result is used for the output of model training input-output pairs or knowledge base knowledge, and data is extracted from other layers and concatenated to fill it; based on the record content, content_hash, record_structure_hash, and quality score are calculated to generate metadata.quality_fingerprint.

[0042] In plain text scenarios, only the core text fields of each unit are enabled; in multimodal scenarios, multimodal extended fields of each unit are enabled. Each record includes fields such as record_id, version, dataset_id, and dataset_version to support version control. The following data record structure is a simplified example (JSON serialization):

[0043] {"record_id": "WATER_001", "version": 2, "dataset_id": "DS-CUSTOMS-001",

[0044] "q_layer": {"query": "Measure the draft of the Yuanyang", "query_type": "visual_measurement", "multimodal_input": {"type": "video", "url": "s3: / / vessel.mp4","key_frames":

[320] }},

[0045] "i_layer": {"intent": {"type": "visual_measurement", "params": {"vessel": "Ocean Number"}}, "data_fetch_instruction": "SELECT length FROM vesselWHERE name='Ocean Number';", "instruction": {"task_type": "visual_measurement", "command": "detect_waterline"}},

[0046] "r_layer": {"rule_id": "R004", "rule_type": "threshold_calculation","rule_structure_hash": "a3f5c8e1d9b2f4a6", "priority": 8, "effective_time": "2025-01-01T00:00:00Z", "rule_relations": [{"target_rule_id": "R001", "relation_type": "input_dependency"}], "association_type": "sequential", "trigger_condition": {"operator": "AND", "conditions": [{"field": "waterline_y", "exists": true}, {"field": "loadline_y", "exists": true}]}, "execution_logic": {"type": "conditional", "condition": "waterline_detected", "then": {"steps": ["compute_distance", "interpolate_depth"]}, "else": {"action": "error_handler"}}, "output_conclusion": "Calculate the draft", "source": {"filename":"Regulations.pdf", "clause": "Article 5.3", "confidence": 0.95}},

[0047] "kg_layer": {"entities": [{"id": "e1", "type": "waterline", "attributes": {"y": 350}, "embedding": [0.12, -0.34]}, {"id": "e2", "type": "loadline", "attributes": {"y": 280}, "embedding": [0.08, -0.29]}], "cross_modal_links": [{"visual_entity_id": "e1", "text_entity_id": "vessel_001", "similarity": 0.85}]},

[0048] "output_result": {"output": {"result": 8.65, "confidence": 0.94}, "inference_reasoning": "The draft line Y = 350, the load line Y = 280, and the depth of 8.65m is obtained by interpolation", "rule_reference": ["R001", "R004"], "visual_evidence": {"frame": 320, "bbox": [200,350,800,5], "url": "s3: / / vessel.mp4#frame=320"}},

[0049] "metadata": {"domain": "customs", "difficulty": 0.72, "quality_fingerprint": {"content_hash": "7d4f1a2e8b3c9d0e", "record_structure_hash": "9e8f7d6c5b4a3c2d", "quality_score": 0.93, "source_credibility": 0.9, "timeliness": 0.98, "domain_weight": 0.85, "diversity_contribution": 0.76, "visual_quality": {"clarity": 0.92, "illumination": 0.85, "occlusion": 0.98, "frame_stability": 0.90, "aggregate_score": 0.91}, "fusion_score": 0.91}, "update_log": [{"time": "2025-04-01T14:20:00Z", "version": 2, "content": "Rule structure hash change"}]}}

[0050] The above JSON serialization format is for illustrative purposes only and does not constitute a limitation on the scope of protection of this invention.

[0051] The dataset serves as the training data source and the foundation of the runtime knowledge base for this system. The following information is extracted and transformed from the dataset to construct the runtime rule knowledge base. The knowledge base is stored using a relational database (such as PostgreSQL) or a graph database (such as Neo4j), and includes at least the following tables / collections:

[0052] The rule table extracts the rule_id, rule content (merging trigger_condition and execution_logic into JSON format), rule structure hash, version number, and deprecated flag (initially false) from the R layer, and extracts key metadata (such as source file identifier, clause number, and rule priority) from the source field of the R layer. The structured representation of the rule content must support condition parsing during subsequent inference. Specifically, the rule structure hash generation involves: first, standardizing and serializing the rule's abstract syntax tree (AST); then concatenating the standardized AST string with the external variables and key metadata that the rule depends on; and finally, using a cryptographic hash algorithm to generate a fixed-length hash value to uniquely identify the logical version of the rule.

[0053] Rule Relationship Table: Dependencies between rules are extracted from the `rule_relations` field in the R layer or through rule content analysis, including `rule_id`, `related_rule_id`, `relation_type` (e.g., "input dependency" or "output mapping"), and `execution_order`. If dependencies are not explicitly provided in the dataset, they are inferred from field references in the rule content, and manual review and correction are supported.

[0054] The Quality Fingerprint table extracts quality metrics from the `quality_fingerprint` field in the metadata, including quality score, source credibility, timeliness, domain weight, diversity contribution, and visual clarity, lighting conditions, and occlusion level in multimodal scenarios. Each record is associated with a data source ID (obtainable from the Q or R layer) and records the `last_updated` timestamp. The quality fingerprint is a set of multidimensional quality metrics for QIR-KG data records, including at least quality score, source credibility, timeliness, domain weight, and visual quality dimensions. The quality score is calculated based on data integrity, consistency, and accuracy; source credibility is preset according to the data source type (e.g., 1.0 for official databases, 0.8 for third-party APIs); visual quality dimensions include at least one or more of image clarity, lighting conditions, occlusion level, and frame rate stability, which can be calculated through weighted summation. The quality fingerprint update mechanism follows these principles: dynamic metrics are updated in real-time after each successful data source call; static metrics are updated weekly; and all metrics are fully synchronized daily.

[0055] Visual Entity Table (Optional): Extracts visual entity information from the entities field of the KG layer, including entity ID, type, bounding box coordinates (x, y, width, height), frame index, detection confidence, etc., to support fast retrieval of visual rules and spatial relationship determination.

[0056] The knowledge base supports both incremental updates and full synchronization. Furthermore, to support multimodal semantic computation, the knowledge base needs to integrate visual feature embedding and knowledge graph embedding.

[0057] Visual feature embeddings can be extracted using a vision-language pre-trained model (such as an image-text alignment model based on contrastive learning). The vector dimension can be preset (e.g., 512 dimensions). It can be directly obtained from the multi_modal_input.visual_features field of the Q layer of the dataset, or recalculated and stored as needed.

[0058] Knowledge graph embedding can employ conventional embedding model training methods for subsequent calculation of data source relevance.

[0059] S2. Training Dataset Construction and Execution Strategy Model Training

[0060] The training dataset is constructed by extracting samples from the dataset. Each QIR-KG record constitutes a training sample, constructed as follows:

[0061] Input: Extract the user query (q_layer.query) and multimodal input reference (q_layer.multi_modal_input, if any) from the Q layer, and extract the intent and instruction (i_layer.instruction) from the I layer. Concatenate these to form the model input. The standardized format of the input sequence is as follows: .in If multimodal input exists, obtain the visual feature embedding (e.g., a 512-dimensional vector) from q_layer.multi_modal_input.visual_features. This vector can be used directly as the embedding input or converted into a text representation. If there is no visual input, this part is empty. : Consists of q_layer.query and i_layer.instruction, in the format: "query: {query} instruction: {instruction}". SEP: Separator marker.

[0062] Output: Extract the structured results from the `output_result.output` field as the expected output. The target sequence is a structured JSON string, including three top-level fields: `metadata`, `data_queries`, and `rule_inference`. This format is completely consistent with the execution plan defined in this invention.

[0063] The model can adopt a sequence-to-sequence architecture, with encoder and decoder lengths preset as needed. Visual feature embeddings can be fused through projection layers. A conventional optimizer and cross-entropy loss function are used during training. On the validation set, the accuracy of the generated execution plan (complete matching of structured fields) must reach a preset accuracy threshold. After training, the model should have the following capabilities: generate structured execution plans based on user input (including multimodal input). The rule instructions in the plan must include rule identifiers, execution order, dependencies, associated query IDs, complete rule content, rule structure hash, and rule version number, and support the embedding of visual processing instructions.

[0064] This system supports a dual-mode operation mechanism: the default mode is online, where the rule reasoning module retrieves the latest rule content from the rule knowledge base during execution and compares it with the rule structure hash carried in the execution plan. If they are inconsistent, the knowledge base takes precedence and is updated. When the knowledge base access is abnormal, the system automatically switches to offline mode and directly uses the rule content output by the model to ensure system robustness.

[0065] S3, Dynamic optimization within the domain

[0066] After the system has been running continuously for a stable period (e.g., a preset number of stable days, and both the execution plan generation accuracy and inference accuracy have reached the corresponding preset thresholds), the latest sample data is used to analyze domain features, including rule type distribution, data source response time, and visual data quality distribution. Normalization and analysis of variance are used to screen effective features, which serve as the basis for parameter tuning. Dynamically tuned parameters include: dependency analysis threshold (e.g., default 0.7, adjustable range 0.6-0.8), conflict resolution weights (weights for each dimension can be adjusted within ±0.1), and visual detection confidence thresholds (e.g., default: 0.7 for recognition, 0.8 for measurement, 0.75 for tracking, adjustable range ±0.1). When optimizing, use the A / B testing method to randomly and evenly distribute a preset percentage (e.g., 5%) of traffic to the new parameter combination. Compare the daily average values ​​of three indicators: accuracy, response time, and user complaint rate. If the new parameter combination improves the accuracy by no less than a preset percentage (e.g., 2%), reduces the response time by no less than a preset percentage (e.g., 10%), and reduces the complaint rate by no less than a preset percentage (e.g., 15%) for a preset number of consecutive preset days (e.g., 7 days), then it will be fully effective.

[0067] S4. Execution Plan Generation

[0068] It receives user input (which may include image / video attachments) and calls the strategy model to generate a structured execution plan. Visual processing instructions use a standardized format (see the input parameter definition in the "Visual Service Interface Specification" of S6 for details). Multimodal keyframes are generated using a uniform sampling algorithm by default; basic scenes (difficulty level 1-2) require at least 3 frames, and complex scenes (difficulty level 3-5) require at least 5 frames. Optional keyframe detection algorithms (such as those based on inter-frame similarity) can be further optimized.

[0069] If the rule identifier in the execution plan is marked as deprecated (deprecated=true) in the knowledge base, then the following dependency refactoring process will be executed:

[0070] ① Recursively search for all rules that directly or indirectly depend on this rule, and generate a list of rules to be deleted;

[0071] ② For each rule in the list to be deleted, search the knowledge base to see if there is a functionally equivalent alternative rule: first, search for explicit alternative rules by mapping the rule identifier; if there is no mapping, search for implicit alternative rules by matching the rule function semantics (e.g., the cosine similarity of the rule text embedding is greater than a preset similarity threshold, which can be dynamically adjusted according to the domain or rule complexity, such as between 0.7 and 0.9).

[0072] ③ If alternative rules exist, add them to the execution plan and rebuild the dependencies based on the rule_relations in the knowledge base;

[0073] ④ If no alternative rule exists, mark the rule as "no valid dependency" and remove it from the plan;

[0074] ⑤ Check whether the dependencies of the remaining rules form a directed acyclic graph (DAG). If a circular dependency is detected, interrupt execution and trigger manual review. After manual review, experts modify the dependencies between rules and resubmit to the execution plan generation module to generate a new acyclic execution plan.

[0075] The plan includes: metadata (difficulty level, data source quality), a list of data query instructions (including query_id, system, interface, method, parameters, dependencies), and a list of rule reasoning instructions (including rule_id, execution_order, dependencies, associated_query_id, association_type, complete rule content, rule structure hash, and rule version number). For visual rules, the rule content includes visual operators (such as above and below). After generating the plan, the rule dependencies are compared with the latest rule_relations in the knowledge base; if they are inconsistent, the execution plan is updated.

[0076] like Figure 2 As shown, the system receives user input (which may include image / video attachments), calls the strategy model to generate a structured execution plan, executes rule inference after rule version verification, and finally merges and encapsulates the results into a structured JSON output.

[0077] S5. Dependency Analysis and Optimization

[0078] Analyze the data dependencies between queries, combine knowledge graph embedding to calculate the data source correlation, and generate an optimized call sequence graph. For example... Figure 4As shown, the system first calculates the data source relevance (integrating entity semantic similarity and field matching, and further integrating visual similarity and knowledge graph similarity in multimodal scenarios), and calculates the priority score by combining the quality fingerprint (quality score, source credibility, timeliness, domain weight, and visual quality dimension). Based on this, it selects the parallel or serial calling strategy and executes a four-level degradation mechanism when the data source call fails.

[0079] The relevance is calculated as follows: Relevance = α × Entity semantic similarity + (1-α) × Field matching degree, where α is a preset weight coefficient that can be dynamically adjusted according to the business scenario (e.g., between 0.5 and 0.7). Field matching degree is the exact matching rate between the query field and the data source field. Entity semantic similarity is obtained by calculating the cosine similarity of the vectors of the entities involved in the user query and the predefined related entities of each data source in the knowledge graph embedding layer; all embedded vectors need to be normalized before calculating the similarity so that the magnitude of each vector is 1. For visual entities, the similarity between the user query entity and the related entities of the data source in the visual feature space and the knowledge graph space are calculated separately, and then weighted and fused to obtain the entity semantic similarity: Entity semantic similarity = β × Visual similarity + (1-β) × Knowledge graph similarity, where the visual similarity is calculated based on the visual feature embedding extracted by the visual-language pre-trained model, and the knowledge graph similarity is calculated based on the embedding calculated by the knowledge graph embedding model; the fused similarity also needs to be normalized so that the magnitude is 1.

[0080] The correlation degree of each data source is calculated, and data sources with a correlation degree greater than a preset correlation degree threshold (this threshold can be dynamically optimized by the system based on historical call success rates, such as between 0.6 and 0.8) are included in the candidate set. If there are two or more data sources in the candidate set, they are called in parallel; if there is only one data source, they are called serially. During parallel calls, the maximum concurrency of a single batch can be dynamically adjusted according to the system load: monitoring three indicators—CPU utilization, memory utilization, and network I / O—and adjusting the concurrency based on the comparison results of each indicator with preset thresholds (e.g., when any indicator is higher than the first threshold, the concurrency decreases; when all indicators are lower than the second threshold, the concurrency increases). The above thresholds are examples and can be preset according to hardware resources and business needs. When a call fails, it automatically switches to the second-best data source; data sources with consecutive failures are temporarily blacklisted, and subsequent requests are automatically downgraded.

[0081] Computational resources are allocated based on difficulty levels, with higher-difficulty tasks receiving higher priority and longer timeouts. The difficulty level is determined by a weighted evaluation of factors such as the number of rules, visual complexity, and data query scale. These factors are normalized and then summed with weights, mapped to a predetermined range to represent the difficulty level. The weights and threshold values ​​for each factor can be configured according to the business scenario.

[0082] S6, Dynamic Invocation of Business Systems

[0083] The API gateway dynamically loads converters based on the interface protocol and calculates data source priority scores based on quality fingerprints. The general formula for priority scores is:

[0084] Score = w1 × Quality Score + w2 × Source Credibility + w3 × Timeliness + w4 × Domain Weight + w5 × Visual Clarity + w6 × Lighting Conditions + w7 × Occlusion Degree

[0085] The weights satisfy w1+w2+w3+w4+w5+w6+w7=1, and different weights can be configured for different scenarios. For example, in a general scenario, the default weights (for plain text) are: w1, w2, w3, and w4 take preset values ​​(e.g., 0.3, 0.25, 0.2, 0.25), and the visual dimension weight is 0. In visually dependent scenarios, the visual dimension weight can be set to a non-zero value, and the weights of other dimensions can be adjusted accordingly (the above weights are just examples and can be preset according to business needs). Diversity contribution is used as an auxiliary screening indicator. Only when the diversity contribution of the data source is lower than a preset threshold (e.g., 0.5) will the score be multiplied by a preset decay coefficient (e.g., 0.8).

[0086] Visual service interface specification: Input parameters include video_url (or image_url), key_frames (array), task_type (detection / measurement / tracking), and target_entities (array); output format is JSON, including frame_index and a detections array (each detection includes entity_type, bounding box [x,y,w,h], confidence, and attributes). Visual detection result fusion rules:

[0087] Detection task: Non-maximum suppression (NMS) is used to merge overlapping boxes. The IoU threshold can be preset according to the detection task (e.g., 0.5). The highest confidence value is taken from the detection results of multiple frames for the same entity.

[0088] Measurement task: For the detection results of multiple frames of the same entity, key frames (key_frames specified in the execution plan) are assigned a higher weight than non-key frames (e.g., 2 times), and non-key frames are assigned a weight of 1. First, calculate the weighted average. If the sample size is sufficient and the dispersion exceeds the preset threshold (e.g., standard deviation > 0.1), then remove the maximum and minimum values ​​and take the weighted median. If the sample size is insufficient, directly take the weighted average.

[0089] Tracking task: Use Kalman filtering to smooth the trajectory of multi-frame detection results to obtain continuous entity positions.

[0090] Low confidence handling: Different task types have different confidence thresholds (which can be preset according to the scenario, such as 0.7 for recognition, 0.8 for measurement, and 0.75 for tracking). If the detection confidence is higher than the threshold, normal execution proceeds; if the confidence is between 0.5 and the threshold, alternative rule inference is triggered, and the current detection result is multiplied by a preset attenuation coefficient (such as 0.8) to participate in auxiliary inference; if the confidence is ≤0.5, it is directly discarded. If a preset number of consecutive keyframes are discarded, manual review is triggered. In online mode, the latest quality fingerprint from the knowledge base is used first; in offline mode, the quality fingerprint in the execution plan is used directly. Data source invocation adopts a four-level data source invocation strategy, with the following steps:

[0091] ① Sort by priority score in descending order and select the one with the highest score as the primary data source;

[0092] ② If the primary data source fails, the second-best data source (the second one) will be automatically called, and the failure log will be recorded.

[0093] ③ If the suboptimal data source fails, call the third data source;

[0094] ④ If the third data source still fails, trigger alternative rule reasoning and send an emergency alert to the administrator.

[0095] Failed data source handling rules: Failed data sources are added to a temporary blacklist. The blacklist duration is dynamically adjusted based on the failure rate (e.g., 1 hour for a 24-hour failure rate > 20%, 6 hours for a failure rate > 50%). After the blacklist expires, the data source is automatically restored as a candidate data source and enters a preset validity observation period (e.g., 72 hours). If the cumulative number of calls reaches a preset number (e.g., 10) and the success rate is higher than a preset threshold (e.g., 80%) during the observation period, the data source is restored to normal priority; otherwise, it is re-added to the blacklist. The above values ​​are examples and can be preset according to the operation and maintenance strategy.

[0096] S7, Rule-based Reasoning and Data Fusion

[0097] like Figure 5 As shown, the rule reasoning and data fusion process includes: cache verification (rule structure hash comparison), visual rule processing (spatial relationship judgment, video temporal analysis, detection result fusion), dependent sequential execution (parallel / serial scheduling), conflict resolution (comprehensive score calculation), and output encapsulation (additional source information and visual evidence).

[0098] S7.1 Cache Management Strategy

[0099] The current content of the rule is retrieved from the local cache, compared with the rule structure hash, and if they are inconsistent, the execution plan is updated based on the knowledge base. For example... Figure 3As shown, the system ensures version consistency through rule structure hash comparison: when the rule structure hash in the execution plan is inconsistent with the current rule structure hash in the knowledge base, the latest rule is loaded from the knowledge base and the cache is updated; when a rule is marked as deprecated, the dependency reconstruction process is triggered.

[0100] In online mode, high-frequency rules employ longer cache validity periods and asynchronous verification; low-frequency rules undergo real-time verification supplemented by periodic fallback verification. When the deprecated flag of a rule in the knowledge base changes, a status change notification is immediately pushed to all cache nodes; if the push fails, the cache is immediately updated upon detecting inconsistencies during periodic verification, and the relevant inference results are marked as pending review.

[0101] In offline mode, the maximum cache capacity is preset based on the device type (e.g., unlimited for cloud devices, 10,000 for edge devices, and 500 for mobile devices). Caches exceeding the capacity are discarded based on a weighted score, with the weighting coefficient preset according to the scenario. The cache uses encrypted local file persistence (AES-256) and supports incremental synchronization to the cloud.

[0102] S7.2 Visual Rule Processing

[0103] Visual entity information is extracted from the query results to determine spatial relationships. Visual spatial relationships are defined based on the coordinate features of entity bounding boxes, including the following three core types of relationships:

[0104] above: The y-coordinate of entity A is less than the y-coordinate of entity B, and the overlap of the x-coordinates is greater than a preset threshold (e.g., 0.5).

[0105] below: The y-coordinate of entity A is greater than the y-coordinate of entity B, and the overlap of the x-coordinates is greater than a preset threshold.

[0106] inside: The bounding box of entity A is completely inside the bounding box of entity B.

[0107] Video timing rule processing: Timing events obtained from the video analytics service must include timestamps. Rules such as "tilt exceeding 5 degrees within 3 seconds" require checking the tilt angle changes of consecutive frames. First, obtain the actual video frame rate (fps), convert the time condition into frame count (frame count = time × fps), and use a sliding window size equal to the converted frame count with a step size of 1 frame.

[0108] S7.3 Dependency Sequential Execution

[0109] Rules are executed in the order of dependencies, rules without dependencies are executed in parallel, and the output of preceding rules is passed to subsequent rules.

[0110] S7.4 Conflict Resolution

[0111] When rule conclusions conflict, a comprehensive score is calculated by considering rule priority, domain weight, quality score, and visual detection confidence, and the output with the highest score is selected. The priority (1-10 points) is normalized to the 0-1 range: norm_priority = priority / 10. The initial comprehensive score formula is: score = first weight × norm_priority + second weight × domain_weight + third weight × quality_score + fourth weight × vision_confidence, where vision_confidence is the visual detection confidence (taken from the visual detection results; if there is no visual rule, it is set to 0). The above weights are examples and can be preset according to business needs.

[0112] When visual rules conflict with textual rules, the `vision_confidence` weight can be automatically increased to a higher value while the weights of other dimensions are correspondingly reduced. The adjusted weight vector is already normalized and can be directly used for calculation. The conflict resolution results are logged for subsequent reinforcement learning training.

[0113] S7.5 Source Information Attached

[0114] Extract source information from the rule content and attach it to the reasoning conclusion. Visual rules are accompanied by visual evidence (keyframe URLs, detection box coordinates), in the format: {"frame":320,"bbox":[x,y,w,h],"url":"s3: / / bucket / video.mp4#frame=320"}. If the visual evidence includes sensitive information such as faces or license plates, it must be automatically blurred before storage.

[0115] S7.6 adds dynamic rule adaptation

[0116] New rules are retrieved from the knowledge base based on real-time data. If the triggering conditions are met, they are added to the execution queue and executed. Conflict resolution is triggered when conflicts occur.

[0117] S7.7 Rule-based reasoning error rollback

[0118] If an error occurs during inference (such as failure to parse rule content or missing dependency data), roll back to the execution node before the error occurred, load the latest rule content, re-execute the inference, and record the reason for the error in the log.

[0119] S8. Result Fusion and Output

[0120] The business data and rule conclusions are encapsulated in a structured manner, and the output JSON format strictly follows the output_result.output specification, including at least result (reasoning conclusion), confidence (0-1), reason (natural language reason), rule_references (rule identifier array), and visual_evidence (optional). The output content must include rule source information (file name, document number, clause) to ensure traceability.

[0121] S9, Large Language Model Collaborative Enhancement

[0122] After anonymizing the structured output, input it into a large language model to generate a natural language interpretation report. Example prompt template: "Please generate a natural language interpretation report based on the following structured data. The language should be concise and logically clear, including the reasoning basis and confidence level explanation (structured JSON)." For large models that support multimodal processing, the `visual_evidence` field is converted to a standard image URL input.

[0123] The large language model interface is set with a preset timeout (e.g., 3 seconds). If the call times out or the large language model service is unavailable, the structured inference result will be output directly, and the reason for the service degradation and the timestamp will be recorded in the system log.

[0124] S10, User Feedback Collection

[0125] Collect user feedback (acceptance, rejection, correction) and record the reason tags. Feedback tags include: data error, rule inapplicability, logical error, model misunderstanding, rule version outdated, inaccurate visual detection, and others. Partial acceptance is supported: users can individually mark a rule conclusion among multiple rule conclusions as accepted / rejected, such as accepting R001 but rejecting R003. In the case of partial acceptance, the weight of the accepted rule sample is 1, the weight of the rejected rule sample is 0, and the weight of unlabeled samples (samples corresponding to other rules that are not labeled in the same request) is a preset value (such as 0.5), which serves as the basis for model fine-tuning. If the user selects "correction", they need to provide the corrected rule content or sample. The system will update the corrected rule to the knowledge base (generating a new rule structure hash), and the corrected sample is labeled as "human-verified sample", with a weight set higher than that of regular samples (such as 1.2), and will be given priority for inclusion in the incremental learning training set. Feedback data is stored in the database, including timestamp, user_id, request_id, rule_id, action (acceptance / rejection / correction), and reason_tag.

[0126] S11, Background Incremental Learning and Self-Evolution

[0127] like Figure 6As shown, after collecting user feedback (acceptance / rejection / correction and reason labels), the system continuously evolves through version-aware sample management (weight decay of old version samples), semi-automatic sample re-labeling, strategy model fine-tuning, rule confidence update and parameter optimization (dependency analysis threshold, conflict resolution weight, visual detection confidence threshold), and after verification by A / B testing.

[0128] Regularly trigger optimization tasks:

[0129] Model fine-tuning: The strategy model is fine-tuned using feedback data, and the weights of old version rule samples are decayed. The old version sample weight = original weight × preset decay base (e.g., 0.8) ^ version difference. When the version difference exceeds a preset threshold (e.g., 5), the weight is set to 0; if the version difference < 0 (the sample training version is higher than the current rule version), the sample weight is directly set to 0. A stratified sampling strategy is used during fine-tuning (stratified by rule version, business domain, and task difficulty level, with random sampling from each stratum to ensure a balanced sample proportion across strata).

[0130] Rule and Parameter Optimization: Adjust rule confidence. When a new rule is first added to the database, the confidence level can be set to a preset default value (e.g., 0.9), or it can be linked to the rule priority (e.g., priority 1-3 → 0.7, 4-7 → 0.8, 8-10 → 0.9). Update formula: New confidence level = λ × old confidence level + (1-λ) × (number of adoptions / total number of triggers), where λ is a preset smoothing factor (e.g., default 0.7, value range 0.5-0.9); if the total number of triggers = 0, the new confidence level remains the same as the old confidence level. Parameter Optimization: The step size for adjusting the dependency analysis threshold, the step size for adjusting the conflict resolution weight, and the step size for adjusting the visual detection confidence threshold can be preset (e.g., 0.05, 0.02, 0.01). After adjustment, it needs to be verified through the test set (accuracy decrease does not exceed the preset percentage, e.g., 2%) before it takes effect.

[0131] In the weight optimization for conflict resolution, the DQN (Deep Q-Network) reinforcement learning algorithm can be used. Specifically, the state space is the dimensional feature vector of the conflict rules, including rule priority, domain weight, quality score, and visual detection confidence; the action space is the adjustment step size of each dimension weight (e.g., ±0.02 or ±0.05); the reward function is the user feedback adoption rate (+1 for adoption, -1 for rejection, and 0 for partial adoption). The network structure can be a two-layer fully connected layer (e.g., 128-dimensional), and the experience replay buffer capacity can be set to 10000. The exploration rate ε can start from a preset initial value (e.g., 0.3) and linearly decrease by a preset step size (e.g., 0.01) daily until it drops to a preset lower limit (e.g., 0.05). Through this mechanism, the system can dynamically optimize the conflict resolution strategy based on user feedback. Version-aware sample management: Sample relabeling is triggered when the rule structure hash changes. A semi-automatic annotation method is adopted: candidate annotations are first pre-generated based on a large language model / visual model. The accuracy of the pre-generated annotations is required to be no less than a preset threshold (such as 80%). If it is lower than this value, the annotations are regenerated. Then, domain experts review and confirm the annotations. Once the review is passed, the annotations are included in the training set.

[0132] Offline operation integration: Offline feedback data is automatically synchronized after recovery. Local database table structure: feedback table (id, request_id, rule_id, action, reason_tag, timestamp, synced_flag). Feedback data generated during offline operation has a default synced_flag value of 0. After network recovery, a background thread synchronizes data in batches, with a preset number of records per batch (e.g., 100). Upon successful synchronization, synced_flag is updated to 1. If synchronization fails, an error log is recorded and the data is added to a retry queue, with a maximum preset number of retries (e.g., 3 times).

[0133] Explanation of the synergistic effect of the algorithm:

[0134] In the technical solution of this invention, multiple algorithms work together to form an overall technical effect.

[0135] The combination of BERT, YOLO, and K-means: BERT analyzes rule text, YOLO extracts visual features, and K-means clusters request intent. The combination of these three can fully characterize the features of a new domain and achieve deployment of new domains within hours.

[0136] The combination of cross-modal semantic model, GraphSAGE, and exponential moving average: the cross-modal semantic model establishes the initial association, GraphSAGE learns the node representation, and the exponential moving average dynamically updates the edge weights, together realizing the complete lifecycle of the rule-data-request association graph, enabling rule changes to be synchronized within minutes.

[0137] Macro-weight configuration + Q-learning combination: Macro-weight configuration initializes a conflict resolution strategy that conforms to business experience, and Q-learning continuously optimizes the weights based on user feedback, so that the conflict resolution strategy not only conforms to business experience but can also be continuously improved.

[0138] The combination of knowledge graph embedding, visual feature embedding, and weighted fusion: knowledge graph embedding provides semantic relationships, visual feature embedding provides visual semantics, and weighted fusion enables cross-modal association, making the calculation of association degree of multimodal data sources more accurate and effectively reducing system response time.

[0139] The above algorithm combination is seamlessly integrated in the data flow and synergistically enhances the overall effect.

[0140] System initialization instructions (applicable to the following embodiments)

[0141] Construct a rule fusion dataset, including multimodal data items, visual rules, and visual entity annotations. Train a policy model to learn the mapping from natural language to structured execution plans (training hyperparameters are the same as S2). Load the rule knowledge base, supporting incremental updates based on version numbers.

[0142] An Application Programming Interface (API) description library is built to uniformly manage business system interfaces. An API adaptation layer is added to standardize request / response formats (requests include request_id, timestamp, and parameters; responses include code, data, and msg; code=0 indicates success, code≠0 indicates an exception, and different exception codes correspond to different error types). Supported protocol types include REST, SOAP, Message Queuing Telemetry Transport (MQTT), and GraphQL. The converter adopts a plug-in architecture.

[0143] Initialize the feedback database to store user feedback and tags.

[0144] Example 1: Customs Vessel Risk Inquiry Scenario (Basic Text Process)

[0145] Step 1: User input: Customs officer inputs "Check the risk level of the ship Yuanyang".

[0146] Step 2: Execution Plan Generation: The strategy model generates a plan, including metadata (difficulty level: Level 2, Advanced), data query instruction q1 (ship monitoring system, parameter {"vessel_name":"Oceanship"}), rule reasoning instructions R001 (rule identifier R001, execution order 1, no dependencies, associated with q1), and R003 (execution order 2, dependent on [R001], associated with q1). After generation, the plan is compared with the knowledge base to confirm the validity of the rules and the consistency of dependencies.

[0147] Step 3: Call Dependency Analysis and Optimization: Single data source, serial calls.

[0148] Step 4: Business System Call: The API gateway calls the ship monitoring system to obtain ship data: origin_region=high-risk region A, value_anomaly_flag=1, violation_count=1.

[0149] Step 5: Rule Reasoning: Retrieve R001 and R003 from the cache; the rule structures have consistent hashes. Execute R001: Condition met, output risk_level=high. Execute R003: Depends on R001 and violation_count≥1, output risk_level=severe. Retrieve the newly added rule R005 (key tracking vessel); condition met, merge the outputs.

[0150] Step 6: Result Fusion: Output structured JSON, including reasoning conclusions and rule sources.

[0151] Step 7: Large Model Collaboration: After anonymization, input the data into a large language model to generate a natural language interpretation report.

[0152] Step 8: Feedback Collection: User feedback is adopted and stored in the feedback database.

[0153] Step 9: Incremental learning: Triggered daily, adjust rule confidence.

[0154] Example 2: Customs draft survey multimodal measurement scenario (multimodal core example)

[0155] Step 1: User input: Customs officer uploads video (30s, 25fps) and inputs "Measure the current draft of the Yuanyang ship".

[0156] Step 2: Execution Plan Generation: The strategy model generates a plan, based on the difficulty level (complex scenario) requirement of key_frames ≥ 5 frames. By default, uniform sampling is used to generate key_frames of [100, 200, 300, 400, 500]; including q1 (ship monitoring system) and q2 (video analysis service, parameters {"video_url":"s3: / / vessel.mp4","key_frames":[100, 200, 300, 400, 500]}), and rule reasoning instructions R001-R006.

[0157] Step 3: Invoke dependency analysis: q1 and q2 have a correlation degree of 0.75, with two data sources in the candidate set, and are invoked in parallel.

[0158] Step 4: Business system call: The ship monitoring system returns a calibration coefficient of 0.01 and a load line depth of 8.5m; the video analysis service returns the detection results of each frame, uses NMS to merge overlapping boxes, and takes the average value of the measurement task by weighting the key frames (key frame weight 2 times).

[0159] Step 5: Rule-based reasoning: R001-R003 Extract visual information; R004 Interpolate to calculate depth; R005 Wave compensation: Take the depth of a single frame from 5 frames, and when the sample size is ≥3 and the standard deviation is >0.1, remove the maximum and minimum values ​​and take the weighted median; R006 Assess the confidence level of 0.94 based on visual quality. Attach visual evidence (keyframe URLs, detection box coordinates).

[0160] Steps 6-9: Same as in Example 1, and will not be repeated here.

[0161] Example 3: Customs laboratory pesticide residue test result verification scenario (multi-data source serial call example)

[0162] Step 1: User input: The inspector inputs "Check the pesticide residue test results of report PR-20250415-001, check whether each item exceeds the standard, and verify whether the testing operation complies with GB2763-2021 and laboratory SOP".

[0163] Step 2: Execution Plan Generation: The strategy model generates an execution plan, including metadata difficulty level 4 and the domain "Customs-Laboratory". Data query commands: q1: LIMS system, interface / report, parameter report_id=PR-20250415-001. q2: Quality control system, interface / records, parameter report_id=PR-20250415-001. Rule reasoning commands: R201: Value exceeding the standard judgment (depends on q1). R202: Operating procedure compliance (depends on q2). R203: Comprehensive judgment (depends on R201 and R202).

[0164] Step 3: Call dependency analysis: The correlation between q1 and q2 is calculated to be 0.55 (lower than the preset threshold of 0.7), so the calls are made sequentially, calling q1 first and then q2.

[0165] Step 4: Business System Call: LIMS returns test data: Chlorpyrifos 0.8 mg / kg, Cypermethrin 0.3 mg / kg, Carbendazim 2.5 mg / kg. The quality control system returns operation records: sample weighing, extraction, purification steps and quality control data (recovery rate, blank sample, etc.).

[0166] Step 5: Rule-based reasoning and data fusion: R201: Searching GB2763-2021, the maximum residue limit (MRL) for carbendazim is 2 mg / kg, and the detected value of 2.5 mg / kg exceeds the limit; the rest are acceptable. R202: All quality control indicators are acceptable; the similarity between the purification procedure description and the SOP is 0.80 (<0.85), therefore it is judged as "non-standard". R203: Due to the existence of items exceeding the limits, the overall judgment is "unacceptable".

[0167] Steps 6-9: Same as in Example 1, and will not be repeated here.

[0168] Example 4: Financial Credit Approval Scenario (Multi-rule Conflict Resolution)

[0169] Step 1: User input: The account manager inputs "Assess the credit risk of client Wang Wu".

[0170] Step 2: Execution Plan Generation: The strategy model generates a plan, including data query instructions q1 (credit API), q2 (anti-fraud system), and q3 (customer database), as well as rule inference instructions R101 (recommend approval) and R102 (recommend rejection). All rules have the same priority, 7.

[0171] Step 3: Invoke Dependency Analysis: Anti-fraud results affect credit scoring strategies. After the correlation is calculated, q1, q2, and q3 are called in parallel.

[0172] Step 4: Business system calls: The credit scoring API returns a credit score of 720, the anti-fraud system returns "at risk", and the customer database returns "income verification is missing".

[0173] Step 5: Rule Reasoning and Conflict Resolution: R101 and R102 are both satisfied, resulting in a conflict. A weighted scoring mechanism is used to calculate the overall score: score = 0.3 × norm_priority + 0.2 × domain_weight + 0.2 × quality_score + 0.3 × vision_confidence (no vision in this scenario, vision_confidence = 0). R101 score is 0.80, R102 score is 0.76, so R101 is selected (recommended, additional verification of income proof is required). The conflict resolution results are logged. Optionally, reinforcement learning (such as DQN) can be used to optimize the weight coefficients based on user feedback.

[0174] Steps 6-9: Same as in Example 1, and will not be repeated here.

[0175] Example 5: Edge mobile scenario (customs on-site inspection, offline inference and feedback synchronization)

[0176] Step 1: User input: Take a photo of the container with the mobile terminal and enter "Inspect container C001".

[0177] Step 2: Execution plan generation: Offline mode, generating plans based on local cache rules, including visual rules (identifying container numbers).

[0178] Steps 3-5: Local lightweight visual model detection (e.g., YOLOv8n, model size <50MB, input image resolution 640×640, deployed using ONNX+TensorRT, inference speed ≥30fps, detection latency ≤50ms), confidence threshold 0.7, discard if below 0.5. Feedback is temporarily stored locally.

[0179] Step 6: After network recovery, a background thread synchronizes feedback data in batches of 100 records each. Upon successful synchronization, the `synced_flag` is updated to 1. The maximum number of retries is 3. This process of collecting and synchronizing feedback data provides an offline data foundation for incremental learning and accumulates training samples for reinforcement learning optimization mechanisms. In actual deployment, the system can periodically trigger model fine-tuning and conflict resolution weight optimization (such as using the DQN algorithm) based on synchronized user feedback, achieving policy self-evolution.

[0180] The above embodiments respectively demonstrate: the basic plain text process (Embodiment 1), multimodal visual processing and parallel invocation (Embodiment 2), serial invocation of multiple data sources and correlation judgment (Embodiment 3), multi-rule conflict resolution (Embodiment 4), and offline inference and feedback synchronization at the edge (Embodiment 5), covering all the core steps of the present invention. Deployment in other fields (such as medical, industrial, and government) can be implemented similarly with reference to the above embodiments, and will not be described in detail here.

[0181] Note: The above embodiments are merely examples and do not constitute a limitation of the present invention.

[0182] System Architecture

[0183] like Figure 1 As shown, the system includes the following modules, which communicate asynchronously via message queues, can be deployed independently, and support horizontal scaling:

[0184] Dataset building module 810: Building rule fusion datasets (see S1).

[0185] Model training module 820: trains the execution policy model and supports incremental fine-tuning (see S2, S11).

[0186] Execution plan generation module 830: Generates a structured execution plan based on user input (see S4).

[0187] Call the dependency analysis module 840: analyze data dependencies, calculate data source correlation, and optimize the call order (see S5).

[0188] API Gateway Module 850: Dynamically loads protocol converters, selects the optimal data source based on quality fingerprints, and integrates degradation and blacklist management (see S6).

[0189] Rule reasoning module 860: Performs rule reasoning, including version verification, visual rule processing, conflict resolution, etc. (see S7).

[0190] Result fusion module 870: Structured encapsulation of output results (see S8).

[0191] Large Language Model Interface Module 880: De-identifies and calls the large language model to generate natural language explanations, and supports timeout degradation (see S9).

[0192] Incremental Learning Module 890: Collects user feedback and periodically triggers model fine-tuning and rule optimization (see S10, S11).

[0193] Global Metadata Management Module 895: Stores and manages metadata at the dataset level.

Claims

1. An intelligent collaborative execution method for multimodal rule fusion datasets, characterized by: Includes the following steps: (1) Obtain the execution plan, which includes a rule identifier and a corresponding rule structure hash, as well as a quality fingerprint of the data source; wherein, the rule structure hash is a unique identifier generated based on the rule logic content; when the user input includes visual data, the execution plan also includes dynamically generated visual processing parameters. (2) Obtain the current rule structure hash of the target rule from the rule knowledge base and perform version verification with the rule structure hash in the execution plan; (3) In response to the inconsistency between the rule structure hash in the execution plan and the current rule structure hash, the latest rule content is loaded from the rule knowledge base to update the execution plan, and inference is performed based on the latest rule content; wherein, when a rule is marked as invalid, dependency reconstruction is triggered, and alternative rules are searched based on semantic matching; (4) When performing a data query, the calling priority of multiple candidate data sources is determined based on the quality fingerprint, and the data source with the highest priority is selected as the main data source for calling; (5) When the primary data source fails to be called, a multi-level degradation mechanism is started to automatically switch to the secondary data source; Among them, the quality fingerprint is a set of multi-dimensional quality indicators recorded in QIR-KG data; When the execution plan includes multiple data query instructions, the data dependencies between the data query instructions are analyzed, and the data source correlation is calculated by combining knowledge graph embedding to optimize the calling order; wherein, for visual entities, the calculation of the data source correlation integrates visual feature similarity and semantic similarity; The execution plan is generated based on user input, which includes at least one of text, images, or videos. For user input containing images or videos, the execution plan includes visual processing instructions, which include visual task type, target entity information, and keyframe parameters.

2. The intelligent collaborative execution method for multimodal rule fusion datasets according to claim 1, characterized in that: The execution plan also includes dependencies between rules and query-rule association types; during inference, the execution order of rules is determined based on dependencies, and rules without dependencies are executed in parallel.

3. The intelligent collaborative execution method for multimodal rule fusion datasets according to claim 1, characterized in that: The execution plan also includes difficulty grading information, which is determined by a comprehensive evaluation based on at least one of the following factors: number of rules, visual complexity, and data query scale. This information is used to allocate differentiated computing resources or scheduling priorities to different tasks.

4. The intelligent collaborative execution method for multimodal rule fusion datasets according to claim 1, characterized in that: It also includes an incremental learning step, which updates the execution strategy model based on user feedback data and adjusts the rule confidence or execution parameters according to the feedback; among which, when the rule version changes, the training samples corresponding to the old version rule are subjected to weight decay processing.

5. The intelligent collaborative execution method for multimodal rule fusion datasets according to claim 1, characterized in that: During reasoning, for visual rules, conditional judgments are made based on the visual detection results; the triggering conditions of the visual rules include visual spatial relationship judgments, which are based on the positional information of visual entities.

6. The intelligent collaborative execution method for multimodal rule fusion datasets according to claim 1, characterized in that: It also includes a conflict resolution step: when multiple rules match simultaneously and the conclusions conflict, the final output is determined by combining rule priority, quality fingerprint and visual detection confidence.

7. An intelligent collaborative system for multimodal rule fusion datasets, characterized in that: The system is used to execute the intelligent collaborative execution method for the multimodal rule fusion dataset as described in any one of claims 1 to 6, comprising: An execution plan generation module is used to obtain an execution plan, which includes a rule identifier and a corresponding rule structure hash, as well as a quality fingerprint of the data source; when the user input includes visual data, the execution plan also includes dynamically generated visual processing parameters. The rule reasoning module is used to obtain the current rule structure hash of the target rule from the rule knowledge base and perform version verification with the rule structure hash in the execution plan; in response to inconsistencies, it loads the latest rule content from the rule knowledge base to update the execution plan and performs reasoning based on the latest rule content; when a rule is marked as invalid, it triggers dependency reconstruction and searches for alternative rules based on semantic matching. The data source invocation module is used to determine the invocation priority of multiple candidate data sources based on the quality fingerprint when performing data queries, and select the data source with the highest priority as the primary data source for invocation; when the primary data source invocation fails, a multi-level degradation mechanism is activated to automatically switch to the secondary data source.

8. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the intelligent collaborative execution method for the multimodal rule fusion dataset as described in any one of claims 1 to 6.

Images